Gemfile.lock not created for second project in Dockerfile

[ShockwaveNN]

Hi, i notice some regression in update from ruby:2.4.1 to ruby:2.4.2
This dockerfile

FROM ruby:2.4.1

RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock

For ruby 2.4.1 it correctly show presence of /clipboard/Gemfile.lock
But if same Dockerimage created FROM ruby:2.4.2 this cause
ls: cannot access /clipboard/Gemfile.lock: No such file or directory

Probably somehow it related with update from bundler 1.15.4 to bundler 1.16.0 but even if I manully rollback to bundler 1.15.4 in ruby:2.4.2 error is the same.
This is actual for any project, i choose ruby-jwt and cliboard as they pretty small.

[gavinkflam]

I was suffered from this issue as well.

After hours of extensive research, it originates from an upstream issue of Bundler.

Please refer to rubygems/bundler#6154 and rubygems/bundler#6162.

===

The workaround provided by @lucasmazza is working.

You can add the environment variable BUNDLE_GEMFILE=./Gemfile to revert to the original behaviour.

[gavinkflam]

Should we introduce BUNDLE_GEMFILE=./Gemfile into all images with bundler ~1.16.0?

Or should we just wait for a fix in the upstream bundler?

[lucasmazza]

@gavinkflam thanks for mentioning this on this repository. Since we stumbled upon this on our app more and more I believe this should be fixed by removing the environment variables on the image rather than changing bundler - I believe that the image assumes too much of how the people are using the image and "installing" their apps with it.

[deivid-rodriguez]

@lucasmazza I totally agree with the image being too opinionated. I think the whole block I mention here could probably be removed except for the BUNDLE_SILENCE_ROOT_WARNING=1, and use default bundler's values.

[lucasmazza]

@yosifkit any thoughts on trimming down the environment variables on the images?

[yosifkit]

Current images with bundler 1.16.1 work fine with equivalent Dockerfile from above.

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ruby:2.4
 ---> dbe4f0ae7c72
Step 2/2 : RUN set -ex; 	git clone https://github.com/jwt/ruby-jwt.git; 	cd ruby-jwt; 		bundle install; 	cd ..; 	git clone https://github.com/janlelis/clipboard.git; 	cd clipboard; 		bundle install; 	cd ..; 	ls -la /clipboard/Gemfile.lock
 ---> Running in e5a0695ade91
+ git clone https://github.com/jwt/ruby-jwt.git
Cloning into 'ruby-jwt'...
+ cd ruby-jwt
+ bundle install
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.22
Installing ffi 1.9.22 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
+ cd ..
+ git clone https://github.com/janlelis/clipboard.git
Cloning into 'clipboard'...
+ cd clipboard
+ bundle install
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Fetching rake 11.3.0
Installing rake 11.3.0
Using bundler 1.16.1
Using clipboard 1.1.1 from source at `.`
Using diff-lcs 1.3
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
+ cd ..
+ ls -la /clipboard/Gemfile.lock
-rw-r--r-- 1 root root 661 Feb 23 20:23 /clipboard/Gemfile.lock
Removing intermediate container e5a0695ade91
 ---> 9d12bfabeb9b
Successfully built 9d12bfabeb9b

[ShockwaveNN]

Can confirm that issue is not actual for your variant of Dockerfile:

FROM ruby:2.4

RUN set -ex;\
    git clone https://github.com/jwt/ruby-jwt.git;\
    cd ruby-jwt;\
    bundle install;\
    cd ..;\
    git clone https://github.com/janlelis/clipboard.git;\
    cd clipboard;\
    bundle install;\
    ls -la /clipboard/Gemfile.lock

But issue still actual for me with with my original Dockerfile from first message

FROM ruby:2.4

RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock

I think issue should be reopened

[tianon]

I'm not able to reproduce with that exact Dockerfile: 😕

$ cat Dockerfile
FROM ruby:2.4

RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock

$ docker build --pull .
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM ruby:2.4
2.4: Pulling from library/ruby
4176fe04cefe: Already exists 
851356ecf618: Already exists 
6115379c7b49: Already exists 
aaf7d781d601: Already exists 
acd6992c338a: Pull complete 
91d9490effb0: Pull complete 
1deabe842a08: Pull complete 
Digest: sha256:eeb289458b9025d9c6c598739ca88e0e0e92e3fa28b8140b0e611f7363961eab
Status: Downloaded newer image for ruby:2.4
 ---> 4b7621478e79
Step 2/6 : RUN git clone https://github.com/jwt/ruby-jwt.git
 ---> Running in 06f72f0071b9
Cloning into 'ruby-jwt'...
Removing intermediate container 06f72f0071b9
 ---> c8a5341e65b9
Step 3/6 : RUN cd ruby-jwt && bundle install
 ---> Running in 4be1b26aa2b6
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.23
Installing ffi 1.9.23 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 4be1b26aa2b6
 ---> 9e59a5707974
Step 4/6 : RUN git clone https://github.com/janlelis/clipboard.git
 ---> Running in 7dbf43405646
Cloning into 'clipboard'...
Removing intermediate container 7dbf43405646
 ---> 83cc647e79c0
Step 5/6 : RUN cd clipboard && bundle install
 ---> Running in b3a1b93b38f2
Using rake 12.3.0
Using bundler 1.16.1
Using docile 1.1.5
Using json 2.1.0
Using simplecov-html 0.10.2
Using simplecov 0.13.0
Using codacy-coverage 1.1.8
Using codeclimate-test-reporter 1.0.8
Using diff-lcs 1.3
Using ffi 1.9.23
Using jwt 2.1.0 from source at `.`
Using rbnacl 5.0.0
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Using simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container b3a1b93b38f2
 ---> 72c29d7ea676
Step 6/6 : CMD ls -la /clipboard/Gemfile.lock
 ---> Running in ebc57e5b2b3e
Removing intermediate container ebc57e5b2b3e
 ---> e9519654e875
Successfully built e9519654e875

[ShockwaveNN]

@tianon Do you not forget to run your container? ls -la /clipboard/Gemfile.lock is CMD command, which execute only on docker run, not docker build

[tianon]

Oh bizarre -- something happens to the files when they get commit into a layer that breaks the second bundle install. There's nothing different between the two invocations except Docker's layering. 😕

[tianon]

For some reason I haven't figured out yet, when we run the second bundle install in a new layer, bundle env reports:

...
  Set via BUNDLE_GEMFILE: "/ruby-jwt/Gemfile"
...

... even though no such environment variable is or ever was set, and we're in the /clipboard directory, and it works fine if we run both bundle install commands in the same layer without round-tripping through Docker.

[tianon]

Ok, I've managed to reproduce with @yosifkit's single-layer image as well by adding hash -r in between the two bundle install invocations. What's happening here is that the image-provided /usr/local/bin/bundle script is getting shadowed by a new /usr/local/bundle/bin/bundle script (via binstubs) which is pointing at a specific Gemfile instead.

[lucasmazza]

@tianon as mentioned on previous comments, this isn't Docker fault by its own, but the combination of how this image configures Bundler (BUNDLE_BIN is always set) and upstream changes on Bundler (see rubygems/bundler#6154 and rubygems/bundler#6162).

[tianon]

Applying the following simple patch appears to do the trick:

-ENV PATH $BUNDLE_BIN:$PATH
+
+# bundler itself is in /usr/local/bin, and due to https://github.com/docker-library/ruby/issues/188 (and thus https://github.com/bundler/bundler/issues/6162) we need to ensure that we don't get a bundler-binstub'd version of bundler in the PATH before our global bundler (hence the explicit "/usr/local/bin" prefix on this PATH)
+ENV PATH /usr/local/bin:$BUNDLE_BIN:$PATH
+

Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM f43db8c0ae0b
 ---> f43db8c0ae0b
Step 2/6 : RUN git clone https://github.com/jwt/ruby-jwt.git
 ---> Running in 605212cce70f
Cloning into 'ruby-jwt'...
Removing intermediate container 605212cce70f
 ---> b96e0830368a
Step 3/6 : RUN cd ruby-jwt && bundle install
 ---> Running in 1fe833dc0c17
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.23
Installing ffi 1.9.23 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 1fe833dc0c17
 ---> 9fd2e01b9635
Step 4/6 : RUN git clone https://github.com/janlelis/clipboard.git
 ---> Running in bcda7be744b8
Cloning into 'clipboard'...
Removing intermediate container bcda7be744b8
 ---> 9930c870c723
Step 5/6 : RUN cd clipboard && bundle install
 ---> Running in 4db88df3ebff
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Fetching rake 11.3.0
Installing rake 11.3.0
Using bundler 1.16.1
Using clipboard 1.1.1 from source at `.`
Using diff-lcs 1.3
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 4db88df3ebff
 ---> 953376471d3b
Step 6/6 : RUN ls -la /clipboard/Gemfile.lock
 ---> Running in f7c7cfdd1c8f
-rw-r--r-- 1 root root 661 Feb 27 22:22 /clipboard/Gemfile.lock
Removing intermediate container f7c7cfdd1c8f
 ---> 1cd1366dee09
Successfully built 1cd1366dee09