Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding caching decorator to cache and reuse heavy data #26

Merged
merged 6 commits into from
Dec 11, 2020
Merged

Conversation

VincGargasson
Copy link
Collaborator

@VincGargasson VincGargasson commented Dec 6, 2020

GOAL OF THIS PR

Storing in background heaviest data in a cache function and being able to reuse it within 60 sec.

Function used : @cache from flask-caching

How it works? :

  1. Defining the configuration of the cache function at the beginning of the script main.py
  2. Integrating the cache.memoize() decorator between the callback and function we aim
  3. The config used here store the data into temporary file in app/.cache

@VincGargasson VincGargasson requested review from frgfm, Akilditu, pechouc, abdi-adel and nrslt and removed request for frgfm December 6, 2020 13:44
@frgfm frgfm added this to the 0.1.1 milestone Dec 6, 2020
@frgfm
Copy link
Member

frgfm commented Dec 6, 2020

@VincGargasson I think you need to add your cache-data-dir to the .gitignore ;)

Copy link
Member

@frgfm frgfm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! The .gitignore needs to be updated, as well as the requirements.txt. The rest seems fine to me, could you post some references on how you used flask_caching so that I can take a closer look please?

@VincGargasson
Copy link
Collaborator Author

Thanks for the PR! The .gitignore needs to be updated, as well as the requirements.txt. The rest seems fine to me, could you post some references on how you used flask_caching so that I can take a closer look please?

@frgfm I'll change this right away.
The way I used it is as follow:

  1. I set up the cache function at the begining of main.py and used filesystem config so the data are stored into a directory of the project easily (we can use redit database but it's more complex).
  2. I spoted function that collect some potential heavy data and put cache.memoize() function between callback and function
  3. The function keep in memory the data for 60sec so that if the function is called within this timeframe with the same args, it gives back the ouput immediately.

As you can see, it's quite simple and I have discovered it last week on several forums.
Please let me know if you still have some questions.

@frgfm
Copy link
Member

frgfm commented Dec 8, 2020

@VincGargasson Nice! Thanks for the explanation!
Although, regarding the PR, you still need to remove the byte content files from the PR (you put the folder in .gitignore, but you need to remove those that were staged 🙃 )

@frgfm
Copy link
Member

frgfm commented Dec 10, 2020

Let me know if you need help with that @VincGargasson 👌

Copy link
Member

@frgfm frgfm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all good! I just removed the bin files, and renamed the cache dir to .cache for consistency

@frgfm frgfm merged commit f8ce3a7 into master Dec 11, 2020
@frgfm frgfm deleted the cached-data branch December 11, 2020 00:14
@frgfm frgfm linked an issue Dec 11, 2020 that may be closed by this pull request
2 tasks
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Cache] Shared data between dashboards should be loaded only once
2 participants