Bundler 2.1.0 does not expose executables from gems without "bundle exec", while 2.0.2 does

[jimeh]

Apologies if this is intended and documented behavior in 2.1.0. I've had a quick look at the release and can't find anything that seems directly related to it. Hence I'm reporting it here so it can either be addressed, or documented better if it is intended behavior.

In short, I can no longer run bundle audit when installing the bundler-audit gem via bundle install. I have to run bundle exec bundle audit (or bundle exec bundle-audit). While if I install the gem with gem install bundler-audit, the gem's executables are installed just fine.

Demo

Setup

Dockerfile.bundler202:

FROM ruby:2.6.5-alpine

WORKDIR /app
RUN echo 'source "https://rubygems.org/"'  > Gemfile
RUN echo 'gem "bundler-audit"'            >> Gemfile
RUN echo 'gem "brakeman"'                 >> Gemfile

RUN gem install bundler -v '2.0.2' && bundle install

CMD bundle audit check --update

Dockerfile.bundler210:

FROM ruby:2.6.5-alpine

WORKDIR /app
RUN echo 'source "https://rubygems.org/"'  > Gemfile
RUN echo 'gem "bundler-audit"'            >> Gemfile
RUN echo 'gem "brakeman"'                 >> Gemfile

RUN gem install bundler -v '2.1.0' && bundle install

CMD bundle audit check --update

Running

WIth Bundler 2.0.2 it works fine:

$ docker run --rm "$(docker build -q -f Dockerfile.bundler202 .)"
Updating ruby-advisory-db ...
Skipping update
ruby-advisory-db: 287 advisories
No vulnerabilities found

While with Bundler 2.1.0 it fails:

$ docker run --rm "$(docker build -q -f Dockerfile.bundler210 .)"
Could not find command "audit".

To dig a bit deeper, using which with Bundler 2.0.2:

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler202 .)" sh
/app # which bundle-audit
/usr/local/bundle/bin/bundle-audit
/app # which brakeman
/usr/local/bundle/bin/brakeman
/app # bundle exec which bundle-audit
/usr/local/bundle/bin/bundle-audit
/app # bundle exec which brakeman
/usr/local/bundle/bin/brakeman

And using which with Bundler 2.1.0:

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler210 .)" sh
/app # which bundle-audit
/app # which brakeman
/app # bundle exec which bundle-audit
/usr/local/bundle/ruby/2.6.0/bin/bundle-audit
/app # bundle exec which brakeman
/usr/local/bundle/ruby/2.6.0/bin/brakeman
/app # gem install bundler-audit
Fetching thor-0.20.3.gem
Fetching bundler-audit-0.6.1.gem
Successfully installed thor-0.20.3
Successfully installed bundler-audit-0.6.1
2 gems installed
/app # which bundle-audit
/usr/local/bundle/bin/bundle-audit
/app # bundle exec which bundle-audit
/usr/local/bundle/ruby/2.6.0/bin/bundle-audit

[deivid-rodriguez]

This sounds like a bug to me, I'll have a look.

[deivid-rodriguez]

But, I think it's not about exposing executables from gems (I don't think they were never exposed like that), but about bundler plugins (I assume bundler-audit is a bundler plugin that adds the audit command).

[jimeh]

@deivid-rodriguez bundler-audit is indeed a plugin for Bundler, but my understanding is that sub-commands are added to bundler like they are with git, by creating a executable called bundle-<my_sub_command> and sticking it in your PATH. Which is what bundler-audit seems to do.

However, after digging a bit deeper into this, Bundler 2.1.0 does behave in a new way. Any gems it installs, do not become part of the global list of gems. For example with 2.0.2 brakeman shows up when you do gem list, while in with Bundler 2.1.0, it does not show under gem list.

Basically, it seems that as of 2.1.0, Bundler installs gems into a special bundler-only area, separate from the global set of gems.

Here's some of the additional checks I did:

• With Bundler 2.0.2:

  $ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler202 .)" sh
  

  /app # gem list | grep brakeman
  brakeman (4.7.2)
  

  /app # which brakeman
  /usr/local/bundle/bin/brakeman
  

  /app # gem info brakeman
  
  *** LOCAL GEMS ***
  
  brakeman (4.7.2)
      Author: Justin Collins
      Homepage: https://brakemanscanner.org
      License: Brakeman Public Use License
      Installed at: /usr/local/bundle
  
      Security vulnerability scanner for Ruby on Rails.
  

  /app # bundle info brakeman
    * brakeman (4.7.2)
          Summary: Security vulnerability scanner for Ruby on Rails.
          Homepage: https://brakemanscanner.org
          Path: /usr/local/bundle/gems/brakeman-4.7.2
  

  /app # bundle exec which brakeman
  /usr/local/bundle/bin/brakeman
  

• With Bundler 2.1.0:

  $ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler210 .)" sh
  

  /app # gem list | grep brakeman
  

  /app # which brakeman
  

  /app # gem info brakeman
  
  *** LOCAL GEMS ***
  
  
  

  /app # bundle info brakeman
    * brakeman (4.7.2)
          Summary: Security vulnerability scanner for Ruby on Rails.
          Homepage: https://brakemanscanner.org
          Path: /usr/local/bundle/ruby/2.6.0/gems/brakeman-4.7.2
  

  /app # bundle exec which brakeman
  /usr/local/bundle/ruby/2.6.0/bin/brakeman
  

[deivid-rodriguez]

Interesting, thanks for the thorough testing, that makes sense, yeah. I'll get to this as soon as I can, butt can you check which version of rubygems is this happening against (gem --version) and whether it also happens using the latest (gem update --system 3.1.1)?

[jimeh]

So both containers were using rubygems 3.0.3 (and Ruby 2.6.5):

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler202 .)" gem --version
3.0.3

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler210 .)" gem --version
3.0.3

I've updated both Dockerfiles to rubygems 3.1.1, however 3.1.1 includes Bundler 2.1.0 as a default gem, which seems to prevent the 2.0.2 container from using Bundler 2.0.2:

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler202 .)" bundle version
Bundler version 2.1.0 (2019-12-16 commit 683fe9799)

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler210 .)" bundle version
Bundler version 2.1.0 (2019-12-15 commit 683fe9799e)

And trying to uninstall Bundler 2.1.0 yields an error because it's a default gem:

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler202 .)" sh
/app # gem uninstall bundler -v 2.1.0
Gem bundler-2.1.0 cannot be uninstalled because it is a default gem

And needless to say, this means the 2.0.2 container has the same issues as the 2.1.0 container, cause they're both now running Bundler 2.1.0 :)

Dockerfiles

Dockerfile.bundler202:

FROM ruby:2.6.5-alpine

WORKDIR /app
RUN echo 'source "https://rubygems.org/"'  > Gemfile
RUN echo 'gem "bundler-audit"'            >> Gemfile
RUN echo 'gem "brakeman"'                 >> Gemfile

RUN gem update --system 3.1.1
RUN gem install bundler -v '2.0.2' && bundle install

CMD bundle audit

Dockerfile.bundler210:

FROM ruby:2.6.5-alpine

WORKDIR /app
RUN echo 'source "https://rubygems.org/"'  > Gemfile
RUN echo 'gem "bundler-audit"'            >> Gemfile
RUN echo 'gem "brakeman"'                 >> Gemfile

RUN gem update --system 3.1.1
RUN gem install bundler -v '2.1.0' && bundle install

CMD bundle audit

P.S. I've changed the base image from ruby:2.6-alpine to ruby:2.6.5-alpine so these examples are more explicit and not affected by any future releases of Ruby.

[deivid-rodriguez]

Thanks so much, I'm taking this next since you made it very easy to reproduce :)

[alexishida]

+1

[deivid-rodriguez]

Ok. So:

• This was introduced by Remove global_path_appends_ruby_scope setting #7163. That was introduced to fix a bug but has created some problems with ruby docker images.

• This issue is specific to the official docker ruby images, because they set $BUNDLE_PATH, which triggers the behaviour of scoping gem installations and their executables to a folder named ruby/<RUBY_ABI_VERSION>. Regular ruby installations don't set $BUNDLE_PATH so they install normally, and globally, to $GEM_HOME.

I think the better fix here would be to change the official ruby images like this:

diff --git a/Dockerfile-alpine.template b/Dockerfile-alpine.template
index dd9f0d8..dd9ee4a 100644
--- a/Dockerfile-alpine.template
+++ b/Dockerfile-alpine.template
@@ -128,7 +128,7 @@ ENV BUNDLE_PATH="$GEM_HOME" \
        BUNDLE_SILENCE_ROOT_WARNING=1 \
        BUNDLE_APP_CONFIG="$GEM_HOME"
 # path recommendation: https://github.com/bundler/bundler/pull/6469#issuecomment-383235438
-ENV PATH $GEM_HOME/bin:$BUNDLE_PATH/gems/bin:$PATH
+ENV PATH $GEM_HOME/bin:$BUNDLE_PATH/ruby/${RUBY_MAJOR}.0/bin:$PATH
 # adjust permissions of a few directories for running "gem install" as an arbitrary user
 RUN mkdir -p "$GEM_HOME" && chmod 777 "$GEM_HOME"
 # (BUNDLE_PATH = GEM_HOME, no need to mkdir/chown both)

As a matter of fact, I think the previous $BUNDLE_PATH/gems/bin entry was added to solve precisely this kind of problem, but it used the wrong path.

I tried this myself and it seems to work, but if you can confirm, it'd be great.

I'll bring this now to https://github.com/docker-library/ruby. They're usually very helpful and quick with these issues, so I think we'll be able to get a fix out soon.

[deivid-rodriguez]

Well, I gave it another thought and since I want to fix this ASAP, I'm proposing to revert the offending PR for the time being. Please see #7501 and let me know if it works for you.

Will ask other maintainers too for their opinion.

[jimeh]

@deivid-rodriguez reverting the offending changes temporarily sounds reasonable to me :)

In my case your suggested fix to the BUNDLE_PATH env var in the official docker images doesn't seem to work. I've probably just done something stupid though since it worked for you. But for me the executable is available in the path, but doesn't show under gem list, yielding a error when trying to run the executable.

In case it's of any help to you, here's some more detail:

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler210 .)" sh

/app # gem list | grep brakeman

/app # which brakeman
/usr/local/bundle/ruby/2.6.0/bin/brakeman

/app # bundle exec which brakeman
/usr/local/bundle/ruby/2.6.0/bin/brakeman

/app # brakeman help
Traceback (most recent call last):
        2: from /usr/local/bundle/ruby/2.6.0/bin/brakeman:23:in `<main>'
        1: from /usr/local/lib/ruby/2.6.0/rubygems.rb:302:in `activate_bin_path'
/usr/local/lib/ruby/2.6.0/rubygems.rb:283:in `find_spec_for_exe': can't find gem brakeman (>= 0.a) with executable brakeman (Gem::GemNotFoundException)

/app # env | sort
BUNDLE_APP_CONFIG=/usr/local/bundle
BUNDLE_PATH=/usr/local/bundle
BUNDLE_SILENCE_ROOT_WARNING=1
GEM_HOME=/usr/local/bundle
HOME=/root
HOSTNAME=c4470f49f384
PATH=/usr/local/bundle/bin:/usr/local/bundle/ruby/2.6.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/app
RUBY_DOWNLOAD_SHA256=d5d6da717fd48524596f9b78ac5a2eeb9691753da5c06923a6c31190abe01a62
RUBY_MAJOR=2.6
RUBY_VERSION=2.6.5
SHLVL=1
TERM=xterm

[deivid-rodriguez]

Yeah, I don't know what I did initially but I later realized it was not so simple, indeed, and got the same you got. Changing the path makes the binstub findable, but rubygems still fails to find the gem (because it's installed to /usr/local/bundle/ruby/2.6.0 and not to $GEM_HOME at /usr/local/bundle).

Anyways, this is why I decided to revert for now, because I need to think of a proper fix to propose upstream.

If you can try the revert though, that'd be great.

[jimeh]

Sure thing, I'll give the revert PR a try :)

And I actually got your suggestion to work by also setting GEM_PATH:

ENV GEM_PATH $GEM_HOME:$BUNDLE_PATH/ruby/${RUBY_MAJOR}.0

$ docker run --rm -ti "$(docker build -q -f Dockerfile.bundler210 .)" sh

/app # gem list | grep brakeman
brakeman (4.7.2)

/app # which brakeman
/usr/local/bundle/ruby/2.6.0/bin/brakeman

/app # bundle exec which brakeman
/usr/local/bundle/ruby/2.6.0/bin/brakeman

/app # brakeman
Loading scanner...
Please supply the path to a Rails application (looking in /app).

/app # env | sort
BUNDLE_APP_CONFIG=/usr/local/bundle
BUNDLE_PATH=/usr/local/bundle
BUNDLE_SILENCE_ROOT_WARNING=1
GEM_HOME=/usr/local/bundle
GEM_PATH=/usr/local/bundle:/usr/local/bundle/ruby/2.6.0
HOME=/root
HOSTNAME=f95802774796
PATH=/usr/local/bundle/bin:/usr/local/bundle/ruby/2.6.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/app
RUBY_DOWNLOAD_SHA256=d5d6da717fd48524596f9b78ac5a2eeb9691753da5c06923a6c31190abe01a62
RUBY_MAJOR=2.6
RUBY_VERSION=2.6.5
SHLVL=1
TERM=xterm

[deivid-rodriguez]

ENV GEM_PATH $GEM_HOME:$BUNDLE_PATH/ruby/${RUBY_MAJOR}.0

Oh yeah, great idea! That, plus properly modifying PATH, sounds pretty solid to me, because since it only adds locations where binstubs and gems can be found (does not change the defaults), it should work consistently across all bundler versions I think.

@indirect What do you think? We might want to upstream these changes to the docker images instead? The bug fix that originated this problem came up a few times, since it makes bundler's behavior very inconsistent. It was not the right choice to ship it like this, but at this point I think we could try upstreaming instead of reverting? (docker images usually get updated quite quickly).

[jimeh]

@deivid-rodriguez FYI, the revert PR also works :)

$ docker run --rm "$(docker build -q -f Dockerfile.bundler-pr7501 .)"
Updating ruby-advisory-db ...
Skipping update
ruby-advisory-db: 287 advisories
No vulnerabilities found

$ docker run --rm "$(docker build -q -f Dockerfile.bundler-pr7501 .)" bundle version
Bundler version 2.1.0.pre.3 (2019-12-17 commit unknown)

Dockerfile.bundler-pr7501:

FROM ruby:2.6.5-alpine as builder

RUN apk add --no-cache build-base git libffi-dev

WORKDIR /src
RUN set -e && \
    git clone --depth=1 \
        --single-branch --branch delay_appending_ruby_scope \
        https://github.com/bundler/bundler.git bundler && \
    cd bundler && \
    gem build *.gemspec


FROM ruby:2.6.5-alpine

WORKDIR /app
RUN echo 'source "https://rubygems.org/"'  > Gemfile
RUN echo 'gem "bundler-audit"'            >> Gemfile
RUN echo 'gem "brakeman"'                 >> Gemfile

COPY --from=builder /src/bundler/*.gem /src/
RUN gem install /src/*.gem && bundle install

CMD bundle audit check --update

[indirect]

If this impacts basically only docker, it seems fine to upstream the fix since existing images won’t be broken by our change :)

[jimeh]

@deivid-rodriguez @indirect I'm not gonna pretend my opinion holds much weight here, but if I've understood things correctly, I believe reverting the changines is probably right way to go, rather than updating the official Ruby docker images.

I believe so beacuse it seems like the behavior in question is not backwards compatible for anyone using Bundler and setting the BUNDLE_PATH environment variable. Of course the determination of if it breaks backwards compatibility or not depends on how much of an "official" use case the BUNDLE_PATH environment variable is. If it was only ever meant for internal purposes, then it's not breaking any intedended public use cases and/or behaviors.

However if it is deemed as breaking backwards compatibility, and if Bundler follows Semantic Versioning (I don't know if it does), then 2.1.0 should probably have been released as 3.0.0.

Obviously please feel to ignore me and/or tell me why I'm wrong if that's the case. Just thought I'd throw my two cents into the mix here :)

[indirect]

That’s a really good point, this is technically breaking. I’d be happiest to revert this and fix the gem update —system thing right away, and then ship this change in 3.0 in a couple of weeks when we also drop old rubies.

[deivid-rodriguez]

bundler has several ways to configure where gems live:

• For the current user with bundle config set path <path> which writes BUNDLE_PATH: <path> to `~/.bundle/config'.
• For the current app with bundle config set --local path <path> which writes BUNDLE_PATH: <path> to ./.bundle/config.
• For the current runtime with ENV["BUNDLE_PATH"].

All of these should be equivalent and set gems to be installed to <path>/ruby/<ruby_abi_version> and the fact that the configuration through an ENV variable didn't was considered a bug and fixed.

Sadly, every bugfix is a backwards incompatible change for everybody relying on the bug... 😞. You just have to make a compromise sometimes. In this case I thought nobody would be relying on the bug, but I missed the case of docker images and how they are setup.

This is just to try and explain the situation, I'm good with reverting.

I just hope nobody had opt-in into the bugfix, and now removed the configuration to opt-in (because of not being need anymore). If someone did that, 2.1.1 will reintroduce the bug for them.

[jimeh]

@deivid-rodriguez Ah, I was not aware that BUNDLE_PATH being set through the environment variable behaved differently from the other methods.

In that case, as it’s less of a breaking change and more of fixing a long-standing bug and inconsistency, my vote falls on leaving the fix in. And fixing the docker images by modifying the PATH and adding GEM_PATH environment variables within them.

But I would say that it needs an explanation in the changelog with some clear info on how you might be affected if you are manually setting BUNDLE_PATH as an environment variable.

[deivid-rodriguez]

Hei, I didn't forget about this :)

For starters I'm adding some tests to the docker official images repo, so we can catch these things in the future. See docker-library/official-images#7166.

[jimeh]

@deivid-rodriguez awesome :)

May I suggest a additional test that ensures that gems installed via bundler also appear under gem list? Technically running bundle audit should verify it's working, but might be good to be double sure, and extra granular so it's more obvious why things are broken when they break.

Also, if you want/need any additional help on this aside from my rants here, I should have a few hours of free time this weekend.

[mltsy]

Eep! This bug is currently breaking several test and staging environments for me and would also be breaking our production environments except that we haven't deployed them without using cached layers in the last few days. Thank you so much for identifying and looking at this, @jimeh and @deivid-rodriguez! It sounds like a fix may be on the way? Or can someone clarify what the best solution would be? (Should we be preparing this use case for a bundler 3.0 update in some way rather than waiting for a fix?)

[deivid-rodriguez]

Hei!

The current workaround is to set

GEM_PATH=$GEM_HOME:$GEM_HOME/ruby/$RUBY_MAJOR.0
PATH=$GEM_HOME/bin:$GEM_HOME/ruby/$RUBY_MAJOR.0/bin:$PATH

Still haven't decided whether to fix this by upstreaming these environment variables changes to the official images, or by reverting the bug fix that caused this issue.

Will come to a decision in the next few days, once I also get input from the official images maintainers.

[deivid-rodriguez]

Ok, so after giving this some more thought, I think the right choice at the moment is to proceed with the revert.

Bundler already has an option to do what the docker images are trying to do: always install gems to the GEM_HOME location. It's BUNDLE_PATH__SYSTEM=true. I did try to introduce this option into the docker images, after noticing that this specific change to how BUNDLE_PATH works would cause issues: docker-library/ruby#285.

However, the change had to be reverted because it conflicts with the --deployment flag, since that automatically sets BUNDLE_PATH=vendor/bundle.

So, my conclusion is that the long term fix for this is to merge BUNDLE_PATH and BUNDLE_PATH__SYSTEM into a single setting. For example, with a special value for BUNDLE_PATH like BUNDLE_PATH=:system. This way, the settings won't conflict and the --deployment flag would simply override the default BUNDLE_PATH=:system configuration.

Since this fix might take time and also require a migration path from BUNDLE_PATH__SYSTEM, I'll revert the offending PR in bundler for now.

I also added docker-library/official-images#7188 to check for any issues with the --deployment flag once the right fix is implemented.

[deivid-rodriguez]

I'm sorry for going in circles here, but after some more thought, I think I changed my mind again. For two reasons:

• Ruby 2.7 will be released today, so we don't have time to fix this in bundler and include the fix with ruby 2.7. That means the default ruby installation will have this bug unless we change the images.

• I just noticed that it's not only ENV-based configuration that work differently to "local application configuration", but also "user configuration". So, correcting my previous statement in Bundler 2.1.0 does not expose executables from gems without "bundle exec", while 2.0.2 does #7494 (comment), unless we keep this bug fix:

  • bundle install --path <path> installs gems to <path>/ruby/<ruby_abi_version>.
  • bundle config set path <path> configures bundler to install gems to <path>.
  • bundle config set --local path <path> configures bundler to install gems to <path>/ruby/<ruby_abi_version>.
  • export BUNDLE_PATH=path configures bundler to install gems to <path>.

• I also noticed that all of our deprecation messages for CLI flags like bundle install --path <path> recommend bundle config path <path> as an alternative. But unless we keep this bug fix and resolve the inconsistency, these are not equivalent alternatives, and can lead to issues! Again, we should correct the deprecation messages to recommend bundle config set --local path <path>, but we don't have time to ship those changes with ruby 2.7.

So, maybe we want to change the images after all. 😳

[jimeh]

@deivid-rodriguez With all the cards on the table, I agree with you that fixing the docker images is probably the approach to take.

And on that topic, I was wondering if there's a specific reason that the docker images use the BUNDLE_PATH environment variable. I could be wrong, but simply removing the env var will also fix things, rather than making it more complicated by adding more to it, and also setting GEM_PATH. That said, removing it might break things for people ¯_(ツ)_/¯

[deivid-rodriguez]

Hei @jimeh, sorry for the delay getting back to this, I took a week off for Christmas. I'll start moving things forward again now.

Regarding your question about why the docker images set BUNDLE_PATH, I think that was a choice made when the images were first started, and I think it was made to try to be consistent with the "one process per container" philosophy. Deviating from bundler's defaults in this way has caused some issues along the way, but I think we've managed to reasonably keep things working and as you point out, removing all that seems risky at this point.

[deivid-rodriguez]

And on that topic, I was wondering if there's a specific reason that the docker images use the BUNDLE_PATH environment variable. I could be wrong, but simply removing the env var will also fix things, rather than making it more complicated by adding more to it, and also setting GEM_PATH. That said, removing it might break things for people ¯_(ツ)_/¯

@jimeh After all, I think your approach is the best thing to do. I digged into the history of the images repo and I see no issues. so I created docker-library/ruby#306. Hopefully we can fix this for good.

[jimeh]

@deivid-rodriguez awesome, it does indeed sound like BUNDLE_PATH is safe to remove :)

I’ve been off work and AFK most of this week, but I should be around again as of today.

[deivid-rodriguez]

Fixed images were released at docker-library/official-images#7239, so closing this issue and fingers crossed 🤞.

[jimbali]

All working for @jimeh and I now :)

[jimeh]

@deivid-rodriguez thanks for all your effort and help with this :)

[mltsy]

Working great for me as well! Thank you very much 🙏