Calling bundle install with BUNDLE_WITH runtime env var persists it in .bundle/config

[mokagio]

I checked all the following documents and couldn't find a solution for my issue:

• Troubleshooting common issues
• Bundler documentation site
• Bundler man pages
• Bundler command line reference

1. Do you have a reproduction script?

https://gist.github.com/mokagio/7dfcd75ffc2fac278c2bde4862c3164b

2. What are you trying to accomplish?

We use some Ruby tooling across the Automattic iOS apps. We want to achieve two things:

• Vendor our gems in a folder within the repository, vendor/bundle
• Have an optional group in the Gemfile, which should be used only by platforms devs and CI.

This setup results in an unexpected change to the .bundle/config when installing the optional group, more on it below.

The need for the optional group is because we use rmagick ~> 3.2.0, which requires an older version of ImageMagick. We don't want the product devs to have to worry about setting up their machine correctly to support it.

3. What command did you run?

In order to make sure gems are vendored, I tried to track .bundle/config in the repo.

---
BUNDLE_PATH: "vendor/bundle"

In order to install the optional group, I run

BUNDLE_WITH=optional_group bundle install

4. What were you expecting to happen?

I was expecting the installation to be successful, including the gem in the optional group, and no change to occur in .bundle/config.

5. What actually happened?

The installation was successful, including the gem in the optional group, but .bundle/config changed, like this:

 ---
 BUNDLE_PATH: "vendor/bundle"
+BUNDLE_WITH: "optional_group"

I wasn't expecting that. My understanding of environment variables is that they can be used to override configurations at runtime. I don't understand why bundle install updated its config file with the environment value I passed it.

Using bundle install --with optional_group, which is a deprecated approach, has the same result.

The docs say:

Flags passed to bundle install or the Bundler runtime, such as --path foo or --without production, are not remembered between commands. If these options must be remembered, they must be set using bundle config (e.g., bundle config path foo).
(Emphasis mine)

The practical implication of this is the -admittedly minor- issue that if a dev needs to do work locally with the gems in the optional group, they need to remember not to commit the changes to .bundle/config.

In the case of our setup, if a dev ends up committing the change to .bundle/config, the CI job running the unit tests will try to install rmagick and fail because the machine it runs on is not provisioned with ImageMagick. Again, this is a minor issue, but I'm still unsure whether the resulting change in the .bundle/config is a feature or a bug.

6. Is there an exception backtrace?

No.

7. bundle env

Details

Environment

Bundler       2.1.4
  Platforms   ruby, x86_64-darwin-19
Ruby          2.7.1p83 (2020-03-31 revision a0c7c23c9cec0d0ffcba012279cd652d28ad5bf3) [x86_64-darwin19]
  Full Path   /Users/gio/.rbenv/versions/2.7.1/bin/ruby
  Config Dir  /Users/gio/.rbenv/versions/2.7.1/etc
RubyGems      3.1.2
  Gem Home    /Users/gio/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0
  Gem Path    /Users/gio/.gem/ruby/2.7.0:/Users/gio/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0
  User Home   /Users/gio
  User Path   /Users/gio/.gem/ruby/2.7.0
  Bin Dir     /Users/gio/.rbenv/versions/2.7.1/bin
Tools         
  Git         2.26.2
  RVM         not installed
  rbenv       rbenv 1.1.2
  chruby      not installed

Bundler Build Metadata

Built At          2020-06-11
Git SHA           unknown
Released Version  false

Bundler settings

path
  Set for your local app (/Users/gio/Developer/a8c/bundler_playground/.bundle/config): "vendor/bundle"
with
  Set for your local app (/Users/gio/Developer/a8c/bundler_playground/.bundle/config): [:screenshots]

Gemfile

Gemfile

source 'https://rubygems.org'

# Just a gem here to verify what's installed and what isn't
gem 'dotenv'

group :screenshots, optional: true do
  gem 'rmagick', '~> 3.2.0'
end

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    dotenv (2.7.5)
    rmagick (3.2.0)

PLATFORMS
  ruby

DEPENDENCIES
  dotenv
  rmagick (~> 3.2.0)

BUNDLED WITH
   2.1.4

---

Thank you for taking the time to read my report (and hopefully respond 😅).

I've been using Bundler for 10 years, almost on a daily basis. This is the first time I find myself opening an issue. I think this proves how much of a great and robust tool this is. Thank you to all the team and contributors for the great effort you put in it. 💖

[deivid-rodriguez]

Hi @mokagio! I just read this and it seems like a bug to me, since all my expectations here match yours. I'll try to find some time to repro and hopefully fix it.

[deivid-rodriguez]

Hi again @mokagio!

So it turns out that this was "accidentally" fixed by #3666. I meant to fix a related but different issue there, but hey, it sounds like the refactoring in there had other nice side effects :)

[mokagio]

Hey @deivid-rodriguez 👋 That's awesome!

I checked out rubygems at 6edf247, installed bundler from the source, and run my repro script. It worked as expected 🎉

Full output

+ ISSUE=3708
+ ROOT_DIR=/tmp/repro-3708
+ rm -rf /tmp/repro-3708
+ mkdir -p /tmp/repro-3708
+ cd /tmp/repro-3708
+ bundle env
## Environment

Bundler       2.2.0.dev
  Platforms   ruby, x86_64-darwin-19
Ruby          2.7.1p83 (2020-03-31 revision a0c7c23c9cec0d0ffcba012279cd652d28ad5bf3) [x86_64-darwin19]
  Full Path   /Users/gio/.rbenv/versions/2.7.1/bin/ruby
  Config Dir  /Users/gio/.rbenv/versions/2.7.1/etc
RubyGems      3.1.2
  Gem Home    /Users/gio/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0
  Gem Path    /Users/gio/.gem/ruby/2.7.0:/Users/gio/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0
  User Home   /Users/gio
  User Path   /Users/gio/.gem/ruby/2.7.0
  Bin Dir     /Users/gio/.rbenv/versions/2.7.1/bin
Tools         
  Git         2.26.2
  RVM         not installed
  rbenv       rbenv 1.1.2
  chruby      not installed

## Bundler Build Metadata

Built At          2020-06-12
Git SHA           6edf24739
Released Version  false
+ cat
+ bundle config set --local path vendor/bundle
+ cat .bundle/config
---
BUNDLE_PATH: "vendor/bundle"
+ bundle install
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Using bundler 2.2.0.dev
Fetching dotenv 2.7.5
Installing dotenv 2.7.5
Bundle complete! 2 Gemfile dependencies, 2 gems now installed.
Bundled gems are installed into `./vendor/bundle`
+ BUNDLE_WITH=optional_group
+ bundle install
Fetching gem metadata from https://rubygems.org/.........
Using bundler 2.2.0.dev
Using dotenv 2.7.5
Fetching safe_yaml 1.0.5
Installing safe_yaml 1.0.5
Bundle complete! 2 Gemfile dependencies, 3 gems now installed.
Bundled gems are installed into `./vendor/bundle`
+ cat .bundle/config
---
BUNDLE_PATH: "vendor/bundle"

Pretty cool how a refactor ended up fixing a bug. Proof of the value of looking after code quality 💪 !

Thank you so much for looking into this promptly. I'll leave it up to you whether to close this issue now or wait till 2.2.0 is out.

[deivid-rodriguez]

Yay! 🎉 I'm glad it works!

I think I'll close this now, because I'm going to forget to do it when we release :)

Thanks for the great report and your kind words!

[mokagio]

Hey @deivid-rodriguez 👋

I just tried this with Bundler 2.2.5, experienced the issue again.

I think I tracked it down to #4154. I run BUNDLE_WITH=group bundle _"$VERSION"_ install with VERSION values going from 2.2.0 to 2.2.5. Prior to 2.2.2, where #4154 landed, it behaves as I'd expect. That is, the given group is installed but the configuration is not persisted in either the local (./.bundle/config) or global (~/.bundle/config) Bundler config files.

The rationale of my expectation is that calling a CLI tool with an env var is a way to bypass existing configurations.

After chatting with @SViccari on Twitter I wonder if my expectation is incorrect 🤔

Can you shed some light on this? And do let me know if there's anything I can do to help solving this.

Thanks 🙌

[deivid-rodriguez]

Ha!

@mokagio My expectations are the same as yours, that's why I fixed this issue initially. However, I had forgotten about this issue. When I got reports about the fix for this issue causing trouble for users, I thought the PR where I fixed this was just something annoying that I found out about, and didn't link it to this bug report. So I decided to temporarily revert it.

Anyways, the idea here is to go with #4152, and then reapply the fix that was reverted since it will no longer cause issues.

[deivid-rodriguez]

I had to revert the fix to this bug, but I never reopened it.

[deivid-rodriguez]

@mokagio We'll be finally addressing this at #5335. Sorry for taking this long!

[mokagio]

@deivid-rodriguez no worries at all! You folks have released a bajillion versions in the meantime, thank you for all the improvements and fixes that came along the way. 🙇‍♂️