Bundler should not try to force install StubSpecifications

[fzakaria]

I am using bundler with Nixpkgs/NixOS where I pre-install some gems but also rely on bundler to install the remaining gems.
When I try to use the --force / --redownload option, bundler fails.

I have done some investigation into this @deivid-rodriguez and I would like your feedback.

Nix has the ability to setup GEM_PATHS that are prebaked with some gems.
For instance

[nix-shell:~/code/nix/playground/jruby-bundler-rake]$ echo $GEM_NIX_ENV
/nix/store/b7hsis6amsl5k2phixrqp7m34yjppxg9-gem-env/lib/jruby/gems/2.5.0

[nix-shell:~/code/nix/playground/jruby-bundler-rake]$ tree /nix/store/b7hsis6amsl5k2phixrqp7m34yjppxg9-gem-env/lib/jruby/gems/2.5.0
/nix/store/b7hsis6amsl5k2phixrqp7m34yjppxg9-gem-env/lib/jruby/gems/2.5.0
├── bin
│   ├── bundle -> /nix/store/1722j9vvjslpk8lhyzs7898ji90n9qn6-bundler-2.1.4/lib/jruby/gems/2.5.0/bin/bundle
│   ├── bundler -> /nix/store/1722j9vvjslpk8lhyzs7898ji90n9qn6-bundler-2.1.4/lib/jruby/gems/2.5.0/bin/bundler
│   └── hello-world -> /nix/store/g1vqkp7vg0cn92mjzsi4j9hd03d28a2s-hello-world-1.2.0/lib/jruby/gems/2.5.0/bin/hello-world
├── build_info
├── doc
├── extensions
├── gems
│   ├── bundler-2.1.4 -> /nix/store/1722j9vvjslpk8lhyzs7898ji90n9qn6-bundler-2.1.4/lib/jruby/gems/2.5.0/gems/bundler-2.1.4
│   └── hello-world-1.2.0 -> /nix/store/g1vqkp7vg0cn92mjzsi4j9hd03d28a2s-hello-world-1.2.0/lib/jruby/gems/2.5.0/gems/hello-world-1.2.0
└── specifications
    ├── bundler-2.1.4.gemspec -> /nix/store/1722j9vvjslpk8lhyzs7898ji90n9qn6-bundler-2.1.4/lib/jruby/gems/2.5.0/specifications/bundler-2.1.4.gemspec
    └── hello-world-1.2.0.gemspec -> /nix/store/g1vqkp7vg0cn92mjzsi4j9hd03d28a2s-hello-world-1.2.0/lib/jruby/gems/2.5.0/specifications/hello-world-1.2.0.gemspec

The above is an example of a gem environment setup by Nix where I've pre-installed bundler & the hellow-world gem.
I add this environment variable $GEM_NIX_ENV to my $GEM_PATH.

When I execute bundle install, bundler works correctly; and determines that the gem is already installed.
When I execute bundle install --redownload, bundler fails; since it tries to check if the gem is present on the cache.

Bundler enters the following codeblock https://github.com/rubygems/rubygems/blob/master/bundler/lib/bundler/source/rubygems.rb#L137 however the specification for the hello-gem is a StubSpecification since bundler has found that the gem already exists.

StubSpecification specs do not have a remote, so the gem itself is not downloaded.
Unfortunately, the way Nix creates the environment, the gem itself is not in the cache directory.

Suggestion: should bundler skip that code block if it's of type StubSpecification & the cache entry does not exist rather than failing ?

I can look into also fixing the nixpkgs code as well with how it creates the environment however this was a pretty thorny issue to try and diagnose.

Post steps to reproduce the problem

I have included a reproducing minimal example you can find: https://github.com/fzakaria/jruby-bundler-nix-failure
It will require that you have Nix installed (https://nixos.org/download.html) which can be done on Darwin or any Linux distribution.

> [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Using bundler 2.1.4
Using hello-world 1.2.0
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.


> [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --force
[DEPRECATED] The `--force` option has been renamed to `--redownload`
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.1.4
Installing hello-world 1.2.0
Bundler::GemNotFound: Could not find hello-world-1.2.0.gem for installation
An error occurred while installing hello-world (1.2.0), and Bundler
cannot continue.
Make sure that `gem install hello-world -v '1.2.0' --source
'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  hello-world

Environment

Bundler       2.1.4
  Platforms   ruby, universal-java-14
Ruby          2.5.7p0 (2020-08-03 revision 67816) [java]
  Full Path   /nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/bin/jruby
  Config Dir  /nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/etc
RubyGems      3.0.6
  Gem Home    /home/fmzakari/code/nix/playground/jruby-bundler-rake/.gem
  Gem Path    /nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/ruby/gems/shared:/nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/jruby/gems/2.5.0:/nix/store/b7hsis6amsl5k2phixrqp7m34yjppxg9-gem-env/lib/jruby/gems/2.5.0:/home/fmzakari/code/nix/playground/jruby-bundler-rake/.gem
  User Home   /home/fmzakari
  User Path   /home/fmzakari/.gem/jruby/2.5.0
  Bin Dir     /home/fmzakari/code/nix/playground/jruby-bundler-rake/.gem/bin
Tools         
  Git         2.29.0
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2020-01-05
Git SHA           32a4159325
Released Version  true

Bundler settings

gem.test
  Set for the current user (/home/fmzakari/.bundle/config): "minitest"
gem.mit
  Set for the current user (/home/fmzakari/.bundle/config): true
gem.coc
  Set for the current user (/home/fmzakari/.bundle/config): false

Gemfile

Gemfile

source "https://rubygems.org"

gem 'hello-world', '1.2.0'

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    hello-world (1.2.0)

PLATFORMS
  java

DEPENDENCIES
  hello-world (= 1.2.0)

BUNDLED WITH
   2.1.4

[fzakaria]

I have opened a matching issue with Nixpkgs with a fix on that side NixOS/nixpkgs#104977 however I feel there is an opportunity here anyways for bundler to be better.

For instance, should bundler anyways be more tolerant to a cache miss?
Should the failure itself be more descriptive ?

Thanks for looking at this when you get the change 🙏

[deivid-rodriguez]

Hi @fzakaria!

I'll try to find some time to install nix and repro this, but in the mean time these are my thoughts.

I think the problem in this case is not that the specification at hand is a StubSpecification. All of the specifications at this point are stub specifications I believe. The problem is that the underlying specification doesn't have a remote, because it's not created from the Gemfile/lockfile (which contain the source & remote information) but from the installed specifications in the standard rubygems locations.

In my opinion, we should treat this case the same as we treat default gems. If you specify a default gem in your Gemfile, it's no longer treated as a default gem, and it starts being locked in the lockfile and installed from the remote sources and cached normally. If on the other hand you don't specify a default gem in the Gemfile, you can still require it but it will be picked up from the standard location, and you want be able to choose its version, it will be the version that comes with ruby.

For these gems, I think the same strategy could work.

[fzakaria]

Sorry for the late response; I was battling COVID :(
Is that specific to rake @deivid-rodriguez ?

I was able to offset the issue with NixOS/nixpkgs#104977 being accepted; I hope it's idiomatic.
The issue was that the GEM_PATH setup by the Nix system had the gems but does not maintain the cached file.
(Which makes sense since the rationale is this creates a Ruby environment outside of Bundler)

I however ran into a interesting case where I wanted to provide my users a default gemset (i.e. rake) but they still used Bundler for their application development. The reality of not having the cache file present caused the problem.

It was easily remedied in Nix by keeping the cache file however I find the lack of fallback to download it again via bundler the ultimate bug report here. The cache file should be a short-circuit to download but not cause it to fail if missing?

[deivid-rodriguez]

Sorry for the late response; I was battling COVID :(

Hope feel better now 💪.

Is that specific to rake @deivid-rodriguez ?

No, not at all.

I understood the issue, yeah. My preferred fix as I tried to explain would be to treat this "default gemset" locations the same way as we treat default gems: they're not cached, they are not reinstalled, they are only "used" when needed.

Changing that would essentially involve adding a similar trick to this:

rubygems/bundler/lib/bundler/rubygems_integration.rb

Lines 422 to 440 in cf06c72

	# Add default gems not already present in specs, and return them as a hash.
	def add_default_gems_to(specs)
	specs_by_name = specs.reduce({}) do |h, s|
	h[s.name] = s
	h
	end
	Bundler.rubygems.default_stubs.each do |stub|
	default_spec = stub.to_spec
	default_spec_name = default_spec.name
	next if specs_by_name.key?(default_spec_name)
	specs << default_spec
	specs_by_name[default_spec_name] = default_spec
	end
	specs_by_name
	end

but with the gems in these "alternative gem locations".

[fzakaria]

I see -- they become a no-op for the redownload operation then ?

How would Bundler distinguish it as a "default gemset" though? It's only specified via the GEM_PATH argument so it just looks like any gem installation? (curious)

Seems like you have a good handle on the "issue" opened by me.
Feel free to to close this issue then since it sounds like a feature request instead; I also understand if you decide not to address/fix it since my use-case is pretty specific/thorny. I at a minimum wanted to raise the issue, thank you for your responses on the matter.

[deivid-rodriguez]

It would always exit early from the method you pointed out earlier as being already installed with a "Using hello-world" message:

rubygems/bundler/lib/bundler/source/rubygems.rb

Lines 107 to 112 in cf06c72

	if (installed?(spec) || Plugin.installed?(spec.name)) && !force
	print_using_message "Using #{version_message(spec)}"
	return nil # no post-install message
	end

If on the other hand, if you do specify it in the Gemfile, then it will be treated as a regular gem and installed where bundler puts gems and cached normally (even if it's also present in alternative gem locations).