Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improved caching decorator #263

Merged
merged 1 commit into from
Jan 5, 2021
Merged

Improved caching decorator #263

merged 1 commit into from
Jan 5, 2021

Conversation

jakobj
Copy link
Member

@jakobj jakobj commented Dec 30, 2020

This PR improves the disk cache decorator:

  • the cache file consists of a single large dict containing (key: fitness) pairs; no need for a linear search through keys to find a match
  • function arguments are now only stored for a single function call to support consistency checks; this avoid unnecessarily large cache files by storing arguments for each(!) result
  • for using multiple processes, a lock is introduced to make sure they are reading from/writing to the cache file without interference

@jakobj jakobj added the enhancement New feature or request label Dec 30, 2020
@jakobj jakobj added this to the 0.3.0 milestone Dec 30, 2020
@coveralls
Copy link
Collaborator

coveralls commented Dec 30, 2020 •
edited
Loading

Coverage Status

Coverage decreased (-0.09%) to 95.259% when pulling 2f50e50 on jakobj:enh/faster-caching into 8a159fc on Happy-Algorithms-League:master.

HenrikMettler
HenrikMettler approved these changes Jan 4, 2021
View reviewed changes
Copy link
Contributor

@HenrikMettler HenrikMettler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me :)

cgp/utils.py Show resolved Hide resolved
@jakobj
Copy link
Member Author

jakobj commented Jan 4, 2021

thanks @HenrikMettler! squashed commits, now waiting for travis

@jakobj
Improve caching decorator
2f50e50 
- change from appending dicts to a pickle file to a single pickled
dict
- avoid storage of function arguments for each result
- introduce multiprocessing file locks for runs with multiple processes
@jakobj jakobj merged commit 20ee3cb into Happy-Algorithms-League:master Jan 5, 2021
@jakobj jakobj deleted the enh/faster-caching branch January 5, 2021 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HenrikMettler HenrikMettler HenrikMettler approved these changes

@mschmidt87 mschmidt87 Awaiting requested review from mschmidt87

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
0.3.0
Development

Successfully merging this pull request may close these issues.
3 participants
@jakobj @coveralls @HenrikMettler