bundle install reports missing gem, but gem is clearly available locally

[ioquatix]

% bundle install --binstubs --path .bundle
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Could not find gem 'utopia (~> 1.6.6)' in any of the gem sources listed in your Gemfile or available on
this machine.
beemo% gem list utopia

*** LOCAL GEMS ***

utopia (1.6.6, 1.6.5, 1.6.2)

Not sure why this is happening, this has worked fine in the past. The gem is only available locally since I'm testing it for release.

Gemfile

source "https://rubygems.org"

gem "utopia", "~> 1.6.6"
# gem "utopia-tags-gallery"
# gem "utopia-tags-google-analytics"

gem "rake"
gem "bundler"

group :development do
    # For `rake server`:
    gem "puma"

    # For `rake console`:
    gem "pry"
    gem "rack-test"
end

[ioquatix]

Just FYI:

beemo% ls -lah /usr/lib/ruby/gems/2.3.0/gems | grep utopia
drwxr-xr-x  9 root root 4.0K May  9 12:33 utopia-1.6.2
drwxr-xr-x  9 root root 4.0K May 10 23:54 utopia-1.6.5
drwxr-xr-x  9 root root 4.0K May 13 00:23 utopia-1.6.6

[ioquatix]

I tried as far back as bundler 1.11.0 and I still had the problem. This makes me think it's not an issue with bundler but a problem with my setup, or a combination of the two. This is a new dev machine I just set up.. it might be different from my older one in some way.

[RochesterinNYC]

What is the output of running bundle env in the directory containing your Gemfile?

[ioquatix]

is disable_shared_gems a problem?

beemo% bundle env      
Environment

    Bundler   1.12.3
    Rubygems  2.5.1
    Ruby      2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
    Git       2.8.2

Bundler settings

    gem.test
      Set for the current user (/home/samuel/.bundle/config): "rspec"
    gem.mit
      Set for the current user (/home/samuel/.bundle/config): "true"
    gem.coc
      Set for the current user (/home/samuel/.bundle/config): "false"
    bin
      Set for your local app (/tmp/ut/.bundle/config): "bin"
      Set for the current user (/home/samuel/.bundle/config): "bin"
    path
      Set for your local app (/tmp/ut/.bundle/config): ".bundle"
    disable_shared_gems
      Set for your local app (/tmp/ut/.bundle/config): "true"
    orig_path
      Set via BUNDLE_ORIG_PATH: "/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl"

Gemfile

    source "https://rubygems.org"

    gem "utopia", "~> 1.6.6"
    # gem "utopia-tags-gallery"
    # gem "utopia-tags-google-analytics"

    gem "rake"
    gem "bundler"

    group :development do
        # For `rake server`:
        gem "puma"

        # For `rake console`:
        gem "pry"
        gem "rack-test"
    end

Gemfile.lock

    <No /tmp/ut/Gemfile.lock found>

[ioquatix]

Hmm. Nothing I do, editing the config, deleting the config, lets me set BUNDLE_DISABLE_SHARED_GEMS: false... That seems odd.

[ioquatix]

Deleting the config and deleting --path avoids the problem, that reverts back to the working behaviour, but then it wants to install into the system gems which, IMHO, is undesirable.

[ioquatix]

Looks like this is affecting other users. Setting BUNDLE_PATH also sets BUNDLE_DISABLE_SHARED_GEMS.

https://bugzilla.redhat.com/show_bug.cgi?id=1225662

Is this the right behaviour?

RedHat's solution was to avoid using bundler:

Our solution was to re-evaluate the use of bundle install in
downstream. Because we install all gems via rpms and we ship a
Gemfile.lock in the gemset rpm, there is no need to run bundle install
on the downstream appliances.

At the very least, the message that bundler prints is wrong, since clearly the gem IS available locally :)

Could not find gem 'utopia (~> 1.6.6)' in any of the gem sources listed in your Gemfile or available on this machine.

[ioquatix]

Relevant source where the setting is forced:

https://github.com/bundler/bundler/blob/e8c962ef2a3215cdc6fd411b6724f091a16793d6/lib/bundler/cli/install.rb#L99

https://github.com/bundler/bundler/blob/e8c962ef2a3215cdc6fd411b6724f091a16793d6/lib/bundler.rb#L421-L422

Perhaps it makes sense for there to be a option which controls this behaviour explicitly? Or, perhaps a different setting, e.g. --vendor-path which sets path and also disables shared gems?

[segiddins]

But it isn't available locally in the path you've told bundler to consider -- bundler literally never sees that gem.

[ioquatix]

@segiddins --path is where to install the gems, no?

[andoq]

Just a note that this was a regression for me after updating bundler - I can reproduce it by breaking a working project by upgrading from bundler 1.10.3 to 1.12.4

my project setup:

I install ruby using rbenv
I install bundler with: gem install bundler
I install gems with: gem install --path vendor/bundle

I run into issues when using Guard, which has some dependency on bundler itself that could not resolve. When guard tries to run bundle exec rspec, it runs into an error:

Could not find 'bundler' (>= 0) among 150 total gems(s)

I've tried regenerating and/or removing all my various binstubs as well.

Ultimately, if I change the

BUNDLE_DISABLE_SHARED_GEMS

setting in my config file to false, then all is well. I'm not sure how this scenario is supposed to work, but it has broken between versions for me.

[andoq]

Just a note, quick testing narrowed my versions. This behavior changed in 1.12.pre.1 with a slightly different error message, and then 1.12.pre.2 exhibits the current behavior.

1.11.2 works for me... not that this narrows things a whole bunch, there's alot of changes in 1.12...

[chrismo]

Setting BUNDLE_PATH also sets BUNDLE_DISABLE_SHARED_GEMS. ... Is this the right behaviour?

I believe so. The latter setting tells it not to use system gems, the former setting tells it where to stash the gems since it's not sharing.

FWIW, I just copied your Gemfile and was able to get it to install fine. You could try rm .bundle dir and trying again?

[coilysiren]

Looks like there is some issue with BUNDLE_DISABLE_SHARED_GEMS and upgrading from 1.X to 1.12 that needs to be resolved still? Although its unclear if thats the issue @ioquatix was experiencing

[andoq]

Follow up to my previous comment: I found my issue like this while using guard-rspec, and for that gem it was a creating a process in with a bundler environment, and there was a fix to using with_original_env instead of with_clean_env, which has a deprecation comment. Here's the issue for guard-rspec:

guard/guard-rspec#369

So my issue my be unreleated to the original issue from ioquatix.

[ioquatix]

I believe so. The latter setting tells it not to use system gems, the former setting tells it where to stash the gems since it's not sharing.

It should be okay to do what I wanted. I don't think setting --path should also change where bundler looks for gems, IMHO.

FWIW, I just copied your Gemfile and was able to get it to install fine. You could try rm .bundle dir and trying again?

Of course, because that gem is now installed to rubygems.org

[ioquatix]

I think if you just changed "Could not find gem 'utopia (> 1.6.6)' in any of the gem sources listed in your Gemfile or available on this machine." to "Could not find gem 'utopia (> 1.6.6)' in any of the gem sources listed in your Gemfile." you'd at least make the current (IMHO broken) behaviour clearer.

[indirect]

Good point, thanks. Made a ticket for it.

[ioquatix]

Thanks for everyone's effort. Just to clarify, this is still an issue w.r.t. the nominal gem behaviour. I don't think this issue is closed just by changing the wording - there is still an underlying problem with the behaviour here.

[indirect]

@ioquatix just to be clear about the behavior: Your original post says that it has "worked in the past", but Bundler has exclusively read gems from and installed gems to the same single location since 1.0, over six years ago. It has been documented for that entire time in the help for install where it talks about the --path and --system flags.

Bundler 2.0 will include a user-wide .gem download cache, in order to save on having to download gems repeatedly, but it's not going to change the way that Bundler treats sources.

[ioquatix]

@indirect I reviewed everything above. The issue here is that setting --path also forces BUNDLE_DISABLE_SHARED_GEMS. This means that simply by adding --path, bundle install fails to work in the case that gems are available locally but not in the remote source, but without --path it does work correctly.

RedHat stopped using bundler in the instance given above due to the behaviour discussed. Adding --path has a side-effect that is non-obvious.

I think the correct solution is that setting --path shouldn't force BUNDLE_DISABLE_SHARED_GEMS. If there isn't already a flag for BUNDLE_DISABLE_SHARED_GEMS perhaps there should be one - so that it's the explicit choice of the user.

[indirect]

@ioquatix yup, that's how it works. We released some pre-1.0 versions of Bundler that did not combine --path and disabling shared gems, and it was a complete disaster from a UX, documentation, usability, and debugging perspective. Not a bug, and we're not planning to change it.

[ioquatix]

@indirect It would be interesting to know what was the problems relating to UX, documentation and usability?

Clearly, the behaviour I see here is bad for UX.

No where can I see this documented:

       --path=<path>
              The location to install the specified gems to. This defaults to  Rubygems'  setting.  Bundler  shares  this
              location with Rubygems, gem install ... will have gem installed there, too. Therefore, gems installed with-
              out a --path ... setting will show up by calling gem list. Accodingly, gems installed  to  other  locations
              will not get listed. This setting is a remembered option.

According to this documentation, adding --path=$DEFAULT_RUBYGEMS_PATH should work the same as not specifying path at all, but it won't, because it will additionally set BUNDLE_DISABLE_SHARED_GEMS. So, that documentation is missing important details.

Finally, from a usability POV, there is no way to change this behaviour. If I am installing local gems for testing, there is no obvious way for me to use these gems and also specify --path for installation.

[ioquatix]

@indirect interesting, two more people run into the same problems, have the same response (that it's confusing).

For the puppet labs issue, see here for discussion: https://tickets.puppetlabs.com/browse/BKR-987

[cbaenziger]

For reference, this issue just helped me understand what is going on for my use case.

I'm using ChefDK which ships a ton of Gems and a Gemfile. To lock references to what they ship, I want to use their Gemfile but I don't want to vendor gems they already provide. (I would think my bundle should add on top of theirs). I too could not understand why I was getting different output with --path versus without; now I understand some of the difference provided by BUNDLE_DISABLE_SHARED_GEMS.

Further, I see my .bundle/config silently resets BUNDLE_DISABLE_SHARED_GEMS to false every run which is really annoying; and that file overrides my ~/.bundle/config so it does not seem I can permanently set BUNDLE_DISABLE_SHARED_GEMS?