BUNDLE_GLOBAL_GEM_CACHE breaks natively compiled libraries #6616

brodock · 2023-04-17T14:05:58Z

Describe the problem as clearly as you can

When setting up BUNDLE_GLOBAL_GEM_CACHE=1 it installs the gems correctly. In a CI environment you want to carry that cache over to a next build. When the next CI run happens (which contains only the cache folder from previous execution) the gems "install" correctly but when trying to execute the code it claims the gems with native extensions are missing the compiled extension/library.

You can see an example here:

https://gitlab.com/gitlab-org/gitlab-development-kit/-/jobs/4117614795

LoadError: cannot find 'llhttp-ext' library
/builds/gitlab-org/gitlab-development-kit/gdk/gitlab/vendor/bundle/ruby/3.0.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find'
/builds/gitlab-org/gitlab-development-kit/gdk/gitlab/vendor/bundle/ruby/3.0.0/gems/llhttp-ffi-0.4.0/lib/llhttp.rb:13:in `<module:LLHttp>'
/builds/gitlab-org/gitlab-development-kit/gdk/gitlab/vendor/bundle/ruby/3.0.0/gems/llhttp-ffi-0.4.0/lib/llhttp.rb:6:in `<top (required)>'
<internal:/home/gdk/.asdf/installs/ruby/3.0.5/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/home/gdk/.asdf/installs/ruby/3.0.5/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'

Did you try upgrading rubygems & bundler?

Yes, I've tried using latest versions of both:

Bundler version 2.4.12

$ gem --version
3.4.11

Post steps to reproduce the problem

I've tested this locally by setting:

export BUNDLE_GLOBAL_GEM_CACHE=1
export BUNDLE_USER_CACHE=~/.cache/bundle

And doing the following:

On the original project: bundle install
Go back to another project with fewer gems: bundle clean --force
Go back to the original project: bundle install
Try to run rails console (and get a lot of missing libraries errors, including pg)

Which command did you run?

bundle install
rails console

What were you expecting to happen?

It should load the code correctly

What actually happened?

I've got a missing (binary/compiled) library error

If not included with the output of your command, run `bundle env` and paste the output below

Environment

Bundler       2.4.11
  Platforms   ruby, arm64-darwin-22
Ruby          3.0.5p211 (2022-11-24 revision ba5cf0f7c52d4d35cc6a173c89eda98ceffa2dcf) [arm64-darwin-22]
  Full Path   /Users/brodock/.rvm/rubies/ruby-3.0.5/bin/ruby
  Config Dir  /Users/brodock/.rvm/rubies/ruby-3.0.5/etc
RubyGems      3.4.12
  Gem Home    /Users/brodock/.rvm/gems/ruby-3.0.5
  Gem Path    /Users/brodock/.rvm/gems/ruby-3.0.5:/Users/brodock/.rvm/rubies/ruby-3.0.5/lib/ruby/gems/3.0.0
  User Home   /Users/brodock
  User Path   /Users/brodock/.gem/ruby/3.0.0
  Bin Dir     /Users/brodock/.rvm/gems/ruby-3.0.5/bin
Tools
  Git         2.40.0
  RVM         1.29.12-next master
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2023-04-10
Git SHA           be1d1b4623
Released Version  true

Bundler settings

Environment

Bundler       2.4.11
  Platforms   ruby, arm64-darwin-22
Ruby          3.0.5p211 (2022-11-24 revision ba5cf0f7c52d4d35cc6a173c89eda98ceffa2dcf) [arm64-darwin-22]
  Full Path   /Users/brodock/.rvm/rubies/ruby-3.0.5/bin/ruby
  Config Dir  /Users/brodock/.rvm/rubies/ruby-3.0.5/etc
RubyGems      3.4.12
  Gem Home    /Users/brodock/.rvm/gems/ruby-3.0.5
  Gem Path    /Users/brodock/.rvm/gems/ruby-3.0.5:/Users/brodock/.rvm/rubies/ruby-3.0.5/lib/ruby/gems/3.0.0
  User Home   /Users/brodock
  User Path   /Users/brodock/.gem/ruby/3.0.0
  Bin Dir     /Users/brodock/.rvm/gems/ruby-3.0.5/bin
Tools
  Git         2.40.0
  RVM         1.29.12-next master
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2023-04-10
Git SHA           be1d1b4623
Released Version  true

Bundler settings

build.ffi
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): "--disable-system-libffi"
build.nokogiri
  Set for the current user (/Users/brodock/.bundle/config): "--use-system-libraries"
build.pg
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): "--with-pg-config=/opt/homebrew/bin/pg_config"
  Set for the current user (/Users/brodock/.bundle/config): "--with-pg-config=/opt/homebrew/bin/pg_config"
build.pg_query
  Set for the current user (/Users/brodock/.bundle/config): "--with-ldflags=-Wl,-undefined,dynamic_lookup"
build.re2
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): "--with-re2-dir=/opt/homebrew/opt/re2"
  Set for the current user (/Users/brodock/.bundle/config): "--with-re2-dir=/opt/homebrew/opt/re2"
build.thin
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): "--with-cflags=-Wno-error=implicit-function-declaration"
  Set for the current user (/Users/brodock/.bundle/config): "--with-cflags=-Wno-error=implicit-function-declaration"
build.thrift
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): "--with-cppflags=-Wno-error=compound-token-split-by-macro"
  Set for the current user (/Users/brodock/.bundle/config): "--with-cppflags=-Wno-error=compound-token-split-by-macro"
cache_all
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): false
gem.changelog
  Set for the current user (/Users/brodock/.bundle/config): true
gem.ci
  Set for the current user (/Users/brodock/.bundle/config): "gitlab"
gem.coc
  Set for the current user (/Users/brodock/.bundle/config): false
gem.mit
  Set for the current user (/Users/brodock/.bundle/config): true
gem.rubocop
  Set for the current user (/Users/brodock/.bundle/config): "true"
gem.test
  Set for the current user (/Users/brodock/.bundle/config): "rspec"
global_gem_cache
  Set via BUNDLE_GLOBAL_GEM_CACHE: true
jobs
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): 10
set
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): "without production"
user_cache
  Set via BUNDLE_USER_CACHE: "~/.cache/bundle"
without
  Set for your local app (/Users/brodock/Projetos/Gitlab/gdk-ee/gitlab/.bundle/config): [:production]
  Set for the current user (/Users/brodock/.bundle/config): [:production]

The text was updated successfully, but these errors were encountered:

stanhu · 2023-04-17T19:46:31Z

I think this issue could be better summarized, "Some native extensions not reinstalled properly with BUNDLE_GLOBAL_GEM_CACHE=1". Here's how I reproduced the problem:

docker run -it ruby:3.0 bash
apt update && apt install -y vim

vim /tmp/Gemfile with:

source 'https://rubygems.org'

gem 'llhttp-ffi'
gem 'http'

Then run:

cd /tmp
BUNDLE_USER_CACHE=/tmp/.cache/bundle BUNDLE_GLOBAL_GEM_CACHE=1 bundle install

Verify that http loads:
```
irb(main):001:0> require 'http'
=> true
```

Now uninstall llhttp-ffi and attempt to reinstall:

 gem uninstall llhttp-ffi
 BUNDLE_USER_CACHE=/tmp/.cache/bundle BUNDLE_GLOBAL_GEM_CACHE=1 bundle install

If you run bundle exec irb:

irb(main):001:0> require 'http'
/usr/local/bundle/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find': cannot find 'llhttp-ext' library (LoadError)
from /usr/local/bundle/gems/llhttp-ffi-0.4.0/lib/llhttp.rb:13:in `<module:LLHttp>'
from /usr/local/bundle/gems/llhttp-ffi-0.4.0/lib/llhttp.rb:6:in `<top (required)>'
from /usr/local/bundle/gems/http-5.1.1/lib/http/response/parser.rb:3:in `require'
from /usr/local/bundle/gems/http-5.1.1/lib/http/response/parser.rb:3:in `<top (required)>'
from /usr/local/bundle/gems/http-5.1.1/lib/http.rb:16:in `require'
from /usr/local/bundle/gems/http-5.1.1/lib/http.rb:16:in `<top (required)>'
from (irb):1:in `require'
from (irb):1:in `<main>'
from /usr/local/lib/ruby/gems/3.0.0/gems/irb-1.3.5/exe/irb:11:in `<top (required)>'
from /usr/local/bin/irb:23:in `load'
from /usr/local/bin/irb:23:in `<top (required)>'
from /usr/local/lib/ruby/3.0.0/bundler/cli/exec.rb:58:in `load'
from /usr/local/lib/ruby/3.0.0/bundler/cli/exec.rb:58:in `kernel_load'
from /usr/local/lib/ruby/3.0.0/bundler/cli/exec.rb:23:in `run'
from /usr/local/lib/ruby/3.0.0/bundler/cli.rb:479:in `exec'
from /usr/local/lib/ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
... 10 levels...

If I look at the --verbose log, without the feature flag, I see:

Installing llhttp-ffi 0.4.0 with native extensions
Building native extensions. This could take a while...
0:  llhttp-ffi (0.4.0) from /usr/local/bundle/specifications/llhttp-ffi-0.4.0.gemspec

However, with the feature flag, I don't see, "Building native extensions. This could take a while...":

Fetching llhttp-ffi 0.4.0
Using http-cookie 1.0.5
3:  http-cookie (1.0.5) from /usr/local/bundle/specifications/http-cookie-1.0.5.gemspec
Installing llhttp-ffi 0.4.0 with native extensions
2:  llhttp-ffi (0.4.0) from /usr/local/bundle/specifications/llhttp-ffi-0.4.0.gemspec

Is the idea the .so should be stored in this cache? Note the same problem doesn't happen with a native extension like pg, but I noticed for pg, the extension is stored in the cache:

root@d6eaab209c8f:/tmp# find /tmp/.cache -name \*.so
/tmp/.cache/bundle/extensions/x86_64-linux/ruby/3.0.0/rubygems.org.443.29b0360b937aa4d161703e6160654e47/unf_ext-0.0.8.2/unf_ext.so
/tmp/.cache/bundle/extensions/x86_64-linux/ruby/3.0.0/rubygems.org.443.29b0360b937aa4d161703e6160654e47/pg-1.4.6/pg_ext.so
/tmp/.cache/bundle/extensions/x86_64-linux/ruby/3.0.0/rubygems.org.443.29b0360b937aa4d161703e6160654e47/ffi-1.15.5/ffi_c.so

Whereas for llhttp-ffi, I see the .so is in the gem path:

root@d6eaab209c8f:/tmp# find /usr/local/bundle -name \*.so | grep http
/usr/local/bundle/gems/llhttp-ffi-0.4.0/ext/x86_64-linux/libllhttp-ext.so

brodock added the Bundler label Apr 17, 2023

brodock mentioned this issue Apr 17, 2023

Issues running it on Mac Intel (12.6) httprb/http#748

Closed

brodock mentioned this issue Apr 18, 2023

Ruby libffi.so.7: cannot open shared object file: ffi/ffi#881

Open

tpowell-progress mentioned this issue Aug 2, 2023

Bundle pristine to patch 3.1 ruby chef/ffi-libarchive#78

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUNDLE_GLOBAL_GEM_CACHE breaks natively compiled libraries #6616

BUNDLE_GLOBAL_GEM_CACHE breaks natively compiled libraries #6616

brodock commented Apr 17, 2023

stanhu commented Apr 17, 2023 •

edited

BUNDLE_GLOBAL_GEM_CACHE breaks natively compiled libraries #6616

BUNDLE_GLOBAL_GEM_CACHE breaks natively compiled libraries #6616

Comments

brodock commented Apr 17, 2023

Describe the problem as clearly as you can

Did you try upgrading rubygems & bundler?

Post steps to reproduce the problem

Which command did you run?

What were you expecting to happen?

What actually happened?

If not included with the output of your command, run bundle env and paste the output below

Environment

Bundler Build Metadata

Bundler settings

Environment

Bundler Build Metadata

Bundler settings

stanhu commented Apr 17, 2023 • edited

If not included with the output of your command, run `bundle env` and paste the output below

stanhu commented Apr 17, 2023 •

edited