Odd behavior with/without --deployment option

[jasonrclark]

I noticed some peculiarities working with a local gem with and without using --deployment on installing our bundle. I've set up a demonstration repo at https://github.com/jasonrclark/odd-deployment.

Note that the setup of my gem isn't quite standard (see section below), so maybe this is expected but I found at least part of what happened surprising.

The Behavior

If I run a normal bundle install I get this list of gems:

~/source/odd-deployment:bundle install
Resolving dependencies...
Using rake 10.5.0
Using bundler 1.12.5
Using little-plugger 1.1.4
Using multi_json 1.12.1
Using minitest 5.9.0
Using zookeeper 1.4.11
Using logging 1.8.2
Using zk 1.9.4
Using nope 0.2.0 from source at `.`
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

However, if I run bundle install --deployment from a clean checkout I get a shorter list:

~/source/odd-deployment:bundle install --deployment
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing rake 10.5.0
Installing minitest 5.9.0
Using bundler 1.12.5
Bundle complete! 4 Gemfile dependencies, 3 gems now installed.
Bundled gems are installed into ./vendor/bundle.

Note that because of the version we're not getting our local gem from source, nor any of its dependencies.

What's Weird in the Setup

We use papers to do license validation tests against our gems. Given that, I found it useful to provide a Gemfile.lock so we wouldn't have versions drifting in the test environment. This is non-standard for a gem, and is likely involved in the problem.

The other oddity is that I've bumped the local VERSION constant for the gem, but haven't updated the Gemfile.lock yet. Obviously not the state things should be in longer term, but in another case I did check something in without updating and saw test failures because of missing dependencies... not what I'd expect.

What Did I Expect?

I was surprised by the different behavior when passing or not passing --deployment. I can buy that bundler can't proceed because of the version mismatch, but maybe it could give a warning or error if it's excluding a local gem in this fashion?

[segiddins]

Awesome, thanks for the repro case? Quick question -- does this still happen in 1.13.0.pre.1?

[jasonrclark]

@segiddins Went to try it out, but even though I have 1.13.0.pre.1 installed, every time I bundle it installes 1.12.5 and tries to use that instead. Is there some trick to persuade things to use the prerelease version?

[chrismo]

bundle update --bundler will upgrade the BUNDLED_WITH section to 1.13.0.pre.1

[jasonrclark]

Thanks @chrismo! That did the trick.

Results were still the same on latest pre-release version.

~/source/odd-deployment:bundle install --deployment
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing rake 10.5.0
Installing minitest 5.9.0
Using bundler 1.13.0.pre.1
Bundle complete! 4 Gemfile dependencies, 3 gems now installed.
Bundled gems are installed into ./vendor/bundle.

✔ ♥♥♥♡ [2.2.4] (master)*
~/source/odd-deployment:bundle --version
Bundler version 1.13.0.pre.1

[chrismo]

@jasonrclark I can't recreate. Am I missing any steps? If not can you post your bundle env?

https://gist.github.com/chrismo/f6583f3a7f530eaed715abb28de8f7be

[jasonrclark]

@chrismo Hmm, my steps seem to have gotten off the path partway through. Your first bundle install bumped the Gemfile.lock version to 0.2.0 makes things work.

Try this and see if it dups for you:

git checkout .
rm -rf .bundle vendor
bundle install --deployment

[chrismo]

Ah - that does recreate:

[chrismo@momac odd-deployment (master)]$ git checkout .
[chrismo@momac odd-deployment (master)]$ rm -rf .bundle vendor
[chrismo@momac odd-deployment (master)]$ bundle install --deployment
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing rake 10.5.0
Installing minitest 5.9.0
Using bundler 1.12.5
Bundle complete! 4 Gemfile dependencies, 3 gems now installed.
Bundled gems are installed into ./vendor/bundle.```

[chrismo]

This is a complicated one, /me taking a break. Been stepping through it for a while - somewhere nope is intentionally being skipped over because it's a locally pathed source is the best I can figure. This feels like a mismatch of intended functionality, similar to what you call out in What's Weird ... --deployment won't update the lockfile (I think - frozen and deployment are related - https://github.com/bundler/bundler/blob/a65a2db118ed714586ac4c56a0584c97bf0305df/lib/bundler/installer.rb#L73 -- https://github.com/bundler/bundler/blob/a65a2db118ed714586ac4c56a0584c97bf0305df/lib/bundler/cli/install.rb#L64) ... best guess: you're asking bundler to do two conflicting things "install the just updated locally sourced gem with the --deployment flag which presumes the current lockfile is good, when it's not" and an Arthur C. Clarke scenario a la giving HAL conflicting orders results.

But I could be wrong. I've been wrong about 3 times just today. ;)

[jasonrclark]

Thanks for all the digging @chrismo! I bet you're right that I'm asking something conflicting.

I did a bit of digging myself, and the main point of divergence between the bundle install and bundle install --deployment that I'm seeing is around https://github.com/bundler/bundler/blob/master/lib/bundler/definition.rb#L215-L227. In particular last_resolve only has minitest and rake in it before the if, but the merge in the else if we're not deploying (aka frozen) adds our local gem (nope) back in from the expanded_dependencies list. Not sure what to make of that, though, just yet.

[chrismo]

This method gets called many times, but the last time, as best I could tell, is where nope was getting dropped -- I think: https://github.com/bundler/bundler/blob/master/lib/bundler/spec_set.rb#L17-L55

[segiddins]

This correctly errors on bundle install --deployment in 1.15:

You are trying to install in deployment mode after changing
your Gemfile. Run `bundle install` elsewhere and add the
updated Gemfile.lock to version control.

The gemspecs for path gems changed