Restore previous performance of private RubyGems servers #5826
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
deivid-rodriguez
force-pushed
the
fast-custom-servers-again
branch
2 times, most recently
from
44e817f
to
88dfd62
Compare
deivid-rodriguez
force-pushed
the
fast-custom-servers-again
branch
from
88dfd62
to
deeaac5
Compare
It's explicitly loaded when monkeypatching RubyGems, which we do very early. So neither autoloading it, nor explicitly loading it anywhere else is necessary.
It's already included by the parent.
The compact index should not request any marshaled gemspecs whatsoever.
And use RubyGems helpers consistently.
Don't fetch remote metadata eagerly. Instead, resolve optimistically without metadata, and then check whether the result is valid after resolution. If not, re-resolve without the invalid candidates. The second re-resolve could potentially fail again, in that case we'll accept the error, because the original issue was very rare in any case, so I imagine needing a second re-resolve should be even more edgy. And this code should go anyways once custom servers migrate to modern APIs that provide metadata information.
deivid-rodriguez
force-pushed
the
fast-custom-servers-again
branch
from
deeaac5
to
8bc9405
Compare
deivid-rodriguez
added a commit
that referenced
this pull request
Aug 24, 2022
Restore previous performance of private RubyGems servers (cherry picked from commit f8c76ea)
|
@deivid-rodriguez Thank you so much! This fixed some problems we were tearing our hair out about! Really appreciate all your hard work!! |
|
No problem, I realized it was a painful thing and there are two many private servers on old APIs. Glad it got fixed! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was the end-user or developer problem that led to this PR?
Since #5256, resolution for private gem servers not implementing the compact index API need to fetch marshaled gemspecs for ALL gems considered by resolution, in order to fetch their metadata requirements. This is painfully slow.
What is your fix for the problem, implemented in this PR?
My fix is to restore the old optimistic approach of resolving without metadata, and then only if the result is not valid, leverage the work done in #5840 to retry resolution excluding the specs with conflicting metadata requirements.
I added a spec to make sure that we're no longer downloading all gemspecs and also tested this in real life on a private Github server.
Closes #5385.
I leave this as WIP for now until I merge #5840, since this one builds on that.#5840 merged.Make sure the following tasks are checked