Progress Bar During Downloads #2
Conversation
|
I like it, however I'm not sure of including ProgressBar globally. Saying this because there is a progressbar gem: https://rubygems.org/gems/progressbar and it might clash. Rubinius guys did a simple implementation: http://github.com/evanphx/rubinius/blob/master/configure#L241-271 That did not depend on the the additional class. We also need to check how this could affect the automated output, specially for projects like Bundler and such. Guys? |
|
I added the progressbar class to the gem namespace, so it shouldn't conflict with anything now. |
|
Hello, I think the good idea will be remove ProgressBar at all. Also, all the remote operations are been handled by this, so fetching of specs and everything is streamed. There is also another problem, it doesn't work with 1.8.7: Tested with See my patch to only generate the progress report when a gem is being downloaded: http://gist.github.com/600349 |
|
I updated it to remove the progressbar class and to use a single line that gets redrawn as suggested in your gist. Also updated to work with 1.8.7. |
|
Thank you Ryan Posted about this to rubygems-devel: http://rubyforge.org/pipermail/rubygems-developers/2010-September/005573.html Waiting for feedback there before decide to integrate it. |
|
Hi Luis, I believe I addressed your issue of using print and flush directly by wrapping them in say-like statements. As no one has commented, can we get this incorporated? |
|
Hello Ryan. I bring this topic to RubyGems about making the progress bar something part of UI component to have better control of it. Also, this print/flush approach do not work nice with remote automations like Chef, so move of this to the UI and have an option to turn it of will be the best. See my thread and further comments there: http://rubyforge.org/pipermail/rubygems-developers/2010-September/005573.html This should be integrated, but to my bad, I'm not 100 percent sure on the approach and to my bad haven't received the feedback from other RubyGems maintainers Will keep you posted. |
|
Hello Ryan, I'm working today on extend UserInteraction to support a configurable Download reporter, in that way, it will make more easy for non-tty interfaces to work and not generate a bunch of output. Thank you again for your contribution. Will post a link to the commits once I'm done for you to test and review. |
|
Been wanting this forever. Any kind of feedback when |
|
I defer to Luis (who needs to stop deferring to the list). |
|
Hello guys, I've posted this to the list for review: http://rubyforge.org/pipermail/rubygems-developers/2010-December/005942.html I've implemented Ryan idea into a more modular and non-tty friendly version under download-reporter branch: https://github.com/rubygems/rubygems/tree/download-reporter Let me know your thoughts guys. |
|
Hello Guys, Wanted to let you know this has been merged into master (slightly different code, same idea) into master: Closing this out. Ryan, thank you for your contribution, patience and initial code for this implementation. |
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` This commit fixes the issue by completely overriding rubygems installer and removing all the unnecessary stuff. In particular, the `Gem::Specification.reset` call that was constantly invalidating gem specification caches and thus mutating shared state. As a results, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` This commit fixes the issue by completely overriding rubygems installer and removing all the unnecessary stuff. In particular, the `Gem::Specification.reset` call that was constantly invalidating gem specification caches and thus mutating shared state. As a results, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` This commit fixes the issue by completely overriding rubygems installer and removing all the unnecessary stuff. In particular, the `Gem::Specification.reset` call that was constantly invalidating gem specification caches and thus mutating shared state has been moved to the main thread, after after installation of all gems is finished. As a results, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` The fix is to completely overriding rubygems installer to get rid of the places where it accessed and modified shared gem specification cache. In particular: * Stop calling `Gem::Specification.reset` after each gem installation and instead move it the main thread, after after installation of all gems has finished. * Stop using rubygems implementation of `Gem::Specification.latest_spec_for(name)` in favor of `Gem::Specification.stubs_for(name).first`, which does the same thing but without traversing and potentially loading all specifications. * Remove all the other code that's not actually necessary for bundler. Technically only the first point would be necessary to fix the race condition but I figured since I needed to overwrite the method I would only include what's needed, and make it faster. As a result, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Hello all,
This pull request brings in a progress bar that will be displayed while downloading gems. This is particularly useful when downloading large gems such as qtbindings on Windows. If the extra STDOUT output is undesirable, it would be easy to modify it to only output for large files (maybe > 1 Mb?). I like the feedback for all file sizes personally. Let me know what you think.
Thanks,
Ryan Melton