Strategy: Temporarily remove response caching#10073
Merged
samford merged 1 commit intoHomebrew:masterfrom Dec 21, 2020
samford:strategy-remove-response-caching
Merged
Strategy: Temporarily remove response caching#10073samford merged 1 commit intoHomebrew:masterfrom samford:strategy-remove-response-caching
samford merged 1 commit intoHomebrew:masterfrom
samford:strategy-remove-response-caching
Conversation
The simple approach here caches all header or body content from responses, so memory usage continually grows with each fetch. This becomes more of a notable issue with long livecheck runs (e.g., `--tap homebrew/core`). Instead, we should only cache the header/body for URLs that we know will be fetched more than once in a given run. Being able to determine which URLs will be fetched more than once requires structural changes within livecheck strategies, so this will take a bit of work to implement. I've been working on this off and on and I'll introduce a more sophisticated method of livecheck-wide caching in a later PR. In the interim time, it's best to remove this caching behavior until I've finished working on an approach that provides benefits (reducing duplicate fetches) while minimizing detriments (increased memory usage).
Contributor
|
Review period ended. |
BrewTestBot
approved these changes
Dec 20, 2020
dawidd6
approved these changes
Dec 20, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
brew stylewith your changes locally?brew typecheckwith your changes locally?brew testswith your changes locally?brew manlocally and committed any changes?This is a follow-up to #9529. I meant to address the code where header and body content for responses were being cached in
@headersand@page_contenthashes (using the URL as the key) but forgot along the way.The issue as I see it is that the simple approach here caches all header or body content from responses, so memory usage continually grows over long livecheck runs (e.g.,
--tap homebrew/core).Instead, we should only cache the header/body data for URLs that we know will be fetched more than once in a given run. The naive approach here has been fine for the
Xorgstrategy, only because the number of pages being cached is very small. A run across homebrew/core could potentially cache 2000+ pages, which is significant.Being able to determine which URLs will be fetched more than once requires structural changes within livecheck strategies, so this will take a bit of work to implement. I've been implementing this off and on (as time permits) and I'll introduce a more sophisticated method of livecheck-wide caching in a later PR. The required changes to strategies also coincide with some refactoring that I already had planned to be able to improve the testability of
livecheckcode, so that work will be used regardless.In the interim time, it's best to remove this naive caching behavior until I've finished working on an approach that provides the benefits we want (reducing duplicate fetches) while minimizing detriments (increased memory usage).
Labeling this as critical, so we can get this in ASAP.