Installing to vendor/bundle appears to be broken

[pixeltrix]

Whilst this appears to be similar to #646 this app does not have json_pure in the bundle.

Running bundle install inside a Debian Bullseye based docker image works fine, but running bundle install --path vendor/bundle breaks with the following error:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/app/vendor/bundle/ruby/3.0.0/gems/appsignal-3.13.1/ext
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.0.0 -r
./siteconf20241025-15-5udun9.rb extconf.rb
/usr/local/lib/ruby/3.0.0/json/common.rb:82:in `initialize': wrong number
of arguments (given 1, expected 0) (ArgumentError)
	from /usr/local/lib/ruby/3.0.0/json/common.rb:82:in `new'
	from /usr/local/lib/ruby/3.0.0/json/common.rb:82:in `create_fast_state'
	from /usr/local/lib/ruby/3.0.0/json/common.rb:75:in `generator='
	from /usr/local/lib/ruby/3.0.0/json/ext.rb:11:in `<module:Ext>'
	from /usr/local/lib/ruby/3.0.0/json/ext.rb:6:in `<module:JSON>'
	from /usr/local/lib/ruby/3.0.0/json/ext.rb:3:in `<top (required)>'
from
<internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
`require'
from
<internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
`require'
	from /usr/local/lib/ruby/3.0.0/json.rb:579:in `<module:JSON>'
	from /usr/local/lib/ruby/3.0.0/json.rb:575:in `<top (required)>'
from
<internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
`require'
from
<internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
`require'
from /app/vendor/bundle/ruby/3.0.0/gems/appsignal-3.13.1/ext/base.rb:7:in
`<top (required)>'
from
<internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
`require'
from
<internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
`require'
	from extconf.rb:3:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in
/app/vendor/bundle/ruby/3.0.0/gems/appsignal-3.13.1 for inspection.
Results logged to
/app/vendor/bundle/ruby/3.0.0/extensions/aarch64-linux/3.0.0/appsignal-3.13.1/gem_make.out

  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:93:in `run'
/usr/local/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:47:in `block
in build'
  /usr/local/lib/ruby/3.0.0/tempfile.rb:317:in `open'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
/usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:159:in
`build_extension'
/usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:193:in `block in
build_extensions'
  /usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in `each'
/usr/local/lib/ruby/3.0.0/rubygems/ext/builder.rb:190:in
`build_extensions'
/usr/local/lib/ruby/3.0.0/rubygems/installer.rb:837:in
`build_extensions'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/source/rubygems.rb:207:in
`install'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:54:in
`install'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:186:in
`do_install'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:177:in
`block in worker_pool'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/worker.rb:62:in
`apply_func'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/worker.rb:57:in `block
in process_queue'
  /usr/local/bundle/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in `loop'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in
`process_queue'
/usr/local/bundle/gems/bundler-2.3.26/lib/bundler/worker.rb:91:in `block
(2 levels) in create_threads'

An error occurred while installing appsignal (3.13.1), and
Bundler cannot continue.

It also fails with GHA running on Ubuntu 22.04 but works locally on my Mac using a chruby install of 3.0.7.

The app can be checked out here: SeneddCymru/e-petitions#220

[pixeltrix]

Weirdly, a fork of the app is fine here: ScottishParliament/e-petitions#308

The broken one does have multi_json in the bundle but doesn't appear in the backtrace.

[casperisfine]

Hum, I cloned the repo an am unable to repro :/

I'll try to figure out what's going on from the backtrace.

[pixeltrix]

Just reproduced from this exact steps:

git clone git@github.com:SeneddCymru/e-petitions.git
cd e-petitions
git checkout update-bundle-and-gha
docker compose build web --no-cache
docker compose run --rm web bash
bundle install --path vendor/bundle

[casperisfine]

/usr/local/lib/ruby/3.0.0/json/common.rb:82:in `initialize': wrong number
of arguments (given 1, expected 0) (ArgumentError)
	from /usr/local/lib/ruby/3.0.0/json/common.rb:82:in `new'
	from /usr/local/lib/ruby/3.0.0/json/common.rb:82:in `create_fast_state'
	from /usr/local/lib/ruby/3.0.0/json/common.rb:75:in `generator='
	from /usr/local/lib/ruby/3.0.0/json/ext.rb:11:in `<module:Ext>'
	from /usr/local/lib/ruby/3.0.0/json/ext.rb:6:in `<module:JSON>'
	from /usr/local/lib/ruby/3.0.0/json/ext.rb:3:in `<top (required)>'

So this is using the stdlib version bundled with Ruby 3.0, not the gem from rubygems. But if initialize take no argument, it suggest the extension from 2.7.3 was loaded.

Thanks for the compose repro, I'll see if it works for me

[pixeltrix]

The fork also fails with these steps now:

git clone git@github.com:ScottishParliament/e-petitions.git
cd e-petitions
git checkout update-bundle-and-gha
docker compose build web --no-cache
docker compose run --rm web bash
bundle install --path vendor/bundle

So this is using the stdlib version bundled with Ruby 3.0, not the gem from rubygems. But if initialize take no argument, it suggest the extension from 2.7.3 was loaded.

Appsignal does have some JSON utilities so wondered if those could be affecting it

[pixeltrix]

The following steps work fine so definitely seems to be related to the bundle path

git clone git@github.com:SeneddCymru/e-petitions.git
cd e-petitions
git checkout update-bundle-and-gha
docker compose build web --no-cache
docker compose run --rm web bash
bundle install

[casperisfine]

Unfortunately podman compose is giving me trouble, so I still can't repro.

But based on the backtrace I think it's quite clear the 2.7.3 extension is loaded with the bundled version of the .rb files. And this is happening in rubygems.

I'll try to reduce your repro to just a bundle install in your container instead of having to deal with the whole compose thing.

[casperisfine]

Alright, I'm able to repro now.

[casperisfine]

So the trigger is that appsignal load json from extconf.rb: https://github.com/appsignal/appsignal-ruby/blob/c615c09466e13e0f090464dabf44ad9957b5c960/ext/base.rb#L7

And as suspected if I inspect $LOADED_FEATURES there is a mix between the stdlib and the gem:

/usr/local/lib/ruby/3.0.0/json/generic_object.rb
/usr/local/lib/ruby/3.0.0/json/common.rb
/app/vendor/bundle/ruby/3.0.0/gems/json-2.7.3/lib/json/ext/parser.so
/app/vendor/bundle/ruby/3.0.0/gems/json-2.7.3/lib/json/ext/generator.so

[casperisfine]

If I upgrade the base image to ruby:3.3-bullseye, the bug is gone, so that sounds like a rubygems issue that may have been fixed. Digging more.

[casperisfine]

gem update --system on 3.0 also fixes it. Trying to see if something in the rubygems changelog matches.

[casperisfine]

Here we go: rubygems/rubygems#6490

But I can't really think of any reasonable work around for this bug that we could implement in a potential 2.7.4.

[pixeltrix]

Here we go: rubygems/rubygems#6490

But I can't really think of any reasonable work around for this bug that we could implement in a potential 2.7.4.

It's at this point I hate Ubuntu do this:

# gem update --system
ERROR:  Your RubyGems was installed trough APT, and upgrading it through RubyGems
itself is unsupported. If you really need the latest version of RubyGems (tip:
you usually don't), then you need to install RubyGems (and Ruby) manually,
maybe using tools like ruby-install, rvm, etc.

😭

This affects Ubuntu 24.04 as well with the default Ruby 3.2 packages

Anything you can think of to workaround this within the app?

[casperisfine]

Urk. That's awful.

It's tricky to workaround from the app itself, because it isn't loaded, you are just in bundler/rubygems.

I'll think some more.

[casperisfine]

So I was thinking of doing: casperisfine@5089c41, but I don't know how to test because just using json as a git gems already work around the bug. So I fear I may introduce another issue.

So I can propose you two workarounds:

gem 'json', '2.7.2' # Don't upgrade until rubygems is upgraded to >= 3.4.9

Or if you want the latest json:

gem 'json', github: "ruby/json", tag: "v2.7.3"

Would that be OK for you?