Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(artifacts): extend cache cleanup to allow specifying a target fraction #6152

Merged
merged 6 commits into from
Aug 24, 2023

Conversation

moredatarequired
Copy link
Contributor

@moredatarequired moredatarequired commented Aug 23, 2023

Fixes

Description

Add an optional target_fraction argument to ArtifactsCache.cleanup so that cleanup(target_fraction=0.25) will remove at least 75% of the current bytes and leave the most recently accessed 25% (or less) used.

馃 Generated by Copilot at 8b5a1b8

Improved the cleanup function of the ArtifactsCache class to support custom cache size limits and ratios. Modified wandb/sdk/artifacts/artifacts_cache.py to implement the new functionality.

Testing

This is a dependency of #6149, which adds tests for full coverage at a higher level.

Checklist

  • Include reference to internal ticket "Fixes WB-NNNNN" and/or GitHub issue "Fixes #NNNN" (if applicable)
  • Ensure PR title compliance with the conventional commits standards

馃 Generated by Copilot at 8b5a1b8

To reduce the cache size with ease
We added some optional keys
To the cleanup function
That controls the reduction
And calculates the target with these

@moredatarequired moredatarequired changed the title feat(artifacts): extend cache cleanup to allow specifying a target fraction to clear feat(artifacts): extend cache cleanup to allow specifying a target fraction Aug 23, 2023
@github-actions github-actions bot added cc-feat and removed cc-feat labels Aug 23, 2023
@moredatarequired moredatarequired marked this pull request as ready for review August 23, 2023 21:11
@github-actions github-actions bot added cc-feat and removed cc-feat labels Aug 23, 2023
@codecov
Copy link

codecov bot commented Aug 23, 2023

Codecov Report

Merging #6152 (ce947ee) into main (cee7fdf) will decrease coverage by 0.13%.
The diff coverage is 66.66%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #6152      +/-   ##
==========================================
- Coverage   77.96%   77.84%   -0.13%     
==========================================
  Files         377      377              
  Lines       43613    43627      +14     
==========================================
- Hits        34004    33960      -44     
- Misses       9557     9615      +58     
  Partials       52       52              
Flag Coverage 螖
unittest 81.44% <66.66%> (-0.14%) 猬囷笍

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage 螖
wandb/sdk/artifacts/artifacts_cache.py 90.40% <66.66%> (-3.30%) 猬囷笍

... and 17 files with indirect coverage changes

@moredatarequired moredatarequired merged commit ffb751b into main Aug 24, 2023
73 of 74 checks passed
@moredatarequired moredatarequired deleted the cleanup-cache-by-fraction branch August 24, 2023 03:29
def cleanup(
self,
target_size: Optional[int] = None,
target_fraction: Optional[float] = None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks existing callsites, e.g.

reclaimed_bytes = cache.cleanup(target_size, remove_temp)

New arguments can be safely added at the end, but not in the middle.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃う

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants