EXPERIMENT: Simplify mirror sync #3844
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Just start with a clean repository version for a mirror sync. This way, there is no need to keep e record of all existing content in memory at any time. This should reduce the memory pressure on the task worker (yet to be confirmed). [noissue]
mdellweg
force-pushed
the
refactor_mirror
branch
from
d4d22b8
to
d87c260
Compare
|
Somehow it decreases the memory usage a bit (but I suspect, what we see is that the garbage collection did not even think of starting to do any work...), but at the expense of taking almost twice the time as the original sync. This is clearly not the best idea in it's current form... Tested with 100 000 content units and stream policy.
|
|
Also, I think when you scale this up to something like PyPI or dockerhub mirroring, it's be much much worse |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Just start with a clean repository version for a mirror sync. This way, there is no need to keep e record of all existing content in memory at any time. This should reduce the memory pressure on the task worker (yet to be confirmed).
[noissue]