Skip to content

Conversation

@mattbrictson
Copy link
Owner

@mattbrictson mattbrictson commented Dec 27, 2024

Inspired by the suggestions in #66, this PR incorporates the concurrent-ruby gem to fetch changelogs concurrently. Doing so can yield a significant speed boost.

For example, for a project with 80 outdated gems, bundle ui in this PR now takes just 7 seconds to run from startup to rendering the complete list of changelogs:

real	0m7.025s
user	0m2.131s
sys	0m0.521s

Whereas bundle ui on the main branch in the same scenario takes 36 seconds:

real	0m36.357s
user	0m2.531s
sys	0m0.684s

For now I'm considering this a proof-of-concept. It may need testing and more error handling (e.g. to deal with HTTP 429 errors) prior to merging.

Edit: To prevent overloading github.com and rubygems.org when fetching changelogs, I limited the concurrency to 25 threads. I tested with a project with over 100 outdated gems and did not encounter any 429 errors. Increasing the limit beyond 25 threads did not seem to yield any additional speed improvement.

@mattbrictson mattbrictson added the ✨ Feature Adds a new feature label Dec 27, 2024
@vfonic
Copy link

vfonic commented Dec 27, 2024

Wow! Super clean solution! 👏

429 Too Many Requests might be an issue, but test it out on a huge project.¹ Maybe GitHub rate limit is high enough, I don't know. I remember that CocoaPods (Obj-C/Swift package manager) used GitHub as their repository directly and it was working fine.

Also, maybe you can somehow limit the amount of parallel requests that are made and/or retry failed requests that fail due to 429 Too Many Requests.

I've never used concurrent-ruby, but the implementation looks so simple, I'm impressed! 👏

¹ - test it before optimizing something that turns out it doesn't have to be optimized.

@mattbrictson
Copy link
Owner Author

mattbrictson commented Dec 27, 2024

Thanks for reviewing! My understanding is that future from concurrent-ruby uses an unbounded number of threads by default, so for a very large number of requests, we could potentially encounter 429 errors.

That said, it is fairly trivial to set a fixed upper bound for the size of the future thread pool. I think I'll set an upper bound just to be safe, like 50 threads max, and then test against a large project to confirm the 429s aren't a problem.

@mattbrictson mattbrictson marked this pull request as ready for review December 27, 2024 23:38
@mattbrictson mattbrictson linked an issue Dec 27, 2024 that may be closed by this pull request
@mattbrictson mattbrictson merged commit 1dd54a4 into main Dec 27, 2024
8 checks passed
@mattbrictson mattbrictson deleted the concurrent branch December 27, 2024 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

✨ Feature Adds a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ability to skip finding changelogs

3 participants