-
Notifications
You must be signed in to change notification settings - Fork 317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance improvement #230
Conversation
Test failed but I don't know what is wrong with this. Maybe, not related with this PR. |
instead of https://bundler.rubygems.org/api/v1/dependencies.json ```ruby require 'geminabox' gems = 'rake,flexmock,parallel_tests,simplecov,rr,timecop,test-unit,test-unit-rr,oj,pry,pry-nav,fluent-plugin-s3,fluent-plugin-forest,fluent-plugin-record-reformer,msgpack,json,yajl-ruby,cool.io,http_parser.rb,sigdump,tzinfo,tzinfo-data,strptime' json_url = "https://bundler.rubygems.org/api/v1/dependencies.json?gems=#{gems}" marshal_url = "https://bundler.rubygems.org/api/v1/dependencies?gems=#{gems}" def elapsed_time(label) started = Time.now yield elapsed = Time.now - started puts "#{label}: #{elapsed.to_f} sec" end elapsed_time(:json_url) do Geminabox.http_adapter.get_content(json_url) end elapsed_time(:marshal_url) do Geminabox.http_adapter.get_content(marshal_url) end ``` ``` json_url: 2.01807 sec marshal_url: 1.261804 sec ```
BEFORE: 1.10931879 sec AFTER: 0.047122157 to resolve dependencies for bones,rcov,rspec,rubyforge,ZenTest,addressable,multipart-post,faraday,faraday-middleware,httpauth,jwt,multi_xml,jar-dependencies,ffi,rbnacl,curses,launchy,bones-rcov,bones-rspec,bones-rubyforge,bones-zentest,git,functional-ruby,ref,gherkin,gherkin3,trollop,term-ansicolor,hashie,oauth2,nokogiri,termios,json_pure,RubyInline,metaclass,needle,echoe,jruby-pageant,bcrypt_pbkdf,rbnacl-libsodium,minitest,psych,tins,facets,polyglot,unf_ext,sexp_processor,concurrent-ruby,method_source,jamespath ```ruby require 'geminabox' local_gem_list = Marshal.load(File.read('bench/local_gem_list_bones.marshal')) remote_gem_list = Marshal.load(File.read('bench/remote_gem_list_bones.marshal')) started = Time.now combined_gem_list = Geminabox::GemListMerge.from(local_gem_list, remote_gem_list) elaspsed = Time.now - started puts "#{elaspsed.to_f} sec" expected = Marshal.load(File.read('bench/combined_gem_list_bones.marshal')) puts combined_gem_list == expected ```
BEFORE: 1.454 sec AFTER: 1.116 sec to resolve dependencies for `unicorn,rake,server-starter,oneline_log_formatter,sinatra,builder,httpclient,nesty,faraday`
@reggieb ping |
These look like a great set of commits. I'm no longer using geminabox, so not in a position to work on it. I've also been keeping quiet so others can takeover, but hanging on incase things get lost. I am however in a position to give you commit rights ;) I'll leave it in your hands, and feel free to get in touch if you need any more access (rubygems.org etc…) |
@tomlea thanks, can you add me to gem owner so that I can publish a gem? EDIT: Thanks, I checked I got an owner bit. |
This PR includes 3 improvements for performance.
1. Use /api/v1/dependencies
api/v1/dependencies
returns marshal dump.api/v1/dependencies.json
returns JSON contents. The former is faster. We should use the former.as an example to resolve dependencies for
rake,flexmock,parallel_tests,simplecov,rr,timecop,test-unit,test-unit-rr,oj,pry,pry-nav,fluent-plugin-s3,fluent-plugin-forest,fluent-plugin-record-reformer,msgpack,json,yajl-ruby,cool.io,http_parser.rb,sigdump,tzinfo,tzinfo-data,strptime
. You can see detail with this commit 41e06f72. Speed up gem_list_merge
to resolve dependencies for
activemodel,arel,tzinfo,activerecord-deprecated_finders,i18n,multi_json,minitest,thread_safe,concurrent-ruby,method_source,term-ansicolor,treetop,gherkin,polyglot,json_pure,multi_test,cucumber-core,gherkin3,cucumber-wire,event-bus,bones,net-scp,metaclass,rspec-support
. You can see a detail with this commit e53e468.EDIT: I could improve more using sort_by! instead of sort! 39f5b2f
to resolve dependencies for
bones,rcov,rspec,rubyforge,ZenTest,addressable,multipart-post,faraday,faraday-middleware,httpauth,jwt,multi_xml,jar-dependencies,ffi,rbnacl,curses,launchy,bones-rcov,bones-rspec,bones-rubyforge,bones-zentest,git,functional-ruby,ref,gherkin,gherkin3,trollop,term-ansicolor,hashie,oauth2,nokogiri,termios,json_pure,RubyInline,metaclass,needle,echoe,jruby-pageant,bcrypt_pbkdf,rbnacl-libsodium,minitest,psych,tins,facets,polyglot,unf_ext,sexp_processor,concurrent-ruby,method_source,jamespath
3. Set Accept-Encoding: gzip
2ffcafe
https://bundler.rubygems.org/api/v1/dependencies
accepts gzip encoding. Using gzip encoding was faster based on experiments.to resolve dependencies for
unicorn,rake,server-starter,oneline_log_formatter,sinatra,builder,httpclient,nesty,faraday