Fix installing gems with native extensions + transitive dependencies

[nicholasdower]

What was the end-user or developer problem that led to this PR?

I am seeing the following error during bundle install:

Gem::MissingSpecError: Could not find 'ffi' (>= 1.15.5) among 48 total gem(s) (Gem::MissingSpecError)

This is reproducible with:

source 'https://rubygems.org'

gem 'llhttp-ffi'

A binary search of this repo indicates the issue may have been introduced in #9381.

It seems only direct dependencies are checked when determining whether a Gem with native extensions is ready to install. I believe this can lead to a failure if a transitive dependency is not yet installed.

In the example above, llhttp-ffi depends on ffi-compiler, which depends on ffi. Since ffi-compiler has no extensions, it is installed immediately without waiting for ffi. When llhttp-ffi then checks its direct dependencies, ffi-compiler is already installed, so llhttp-ffi starts building its native extension. The build requires ffi, which may not have been installed yet.

What is your fix for the problem, implemented in this PR?

Check transitive dependencies for Gems with native extensions.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[nicholasdower]

cc: @Edouard-chin

[Edouard-chin]

Thank you for catching this and opening a fix ! Not sure how I missed this case 🤦 .
The only concern I have with this solution is that all gems with native extensions now have to pay the cost of waiting for all their dependencies (direct + transitive) to be installed. I think native ext gems that needs a transitive dependencies to be installtable are the minority, and I'd like to explore a different solution.

What I have in mind is basically to install any gems without waiting (even the one with native extensions), and if an error arise during the installation of a gem with native exts, we put it back in the queue and set a flag so that we only retry the installation once all its dependencies have been installed. Many gems with native extensions don't need any dependencies to be installed, even direct ones, so that will not penalize them.

I'll try this experiment this week, and if that doesn't work for whatever reason, let's go with your approach !

[nicholasdower]

Thank you very much for taking the time to review this.

Regarding the performance concern, since 4.0.8 effectively waited for transitive dependencies, my understanding is that this change would at worst be a return to 4.0.8 performance. The worst case I can think of is a chain where gem A with a native extension depends on pure Ruby gem B, which itself depends on gem C with a native extension. In that case, installs of gems A and C would be serialized.

I couldn't find guidance on how to benchmark this change, so I wrote a simple script and used it to compare Bundler 4.0.9 and this branch across multiple Gemfiles constructed from the 200 most-downloaded gems from the last day (https://bestgems.org/daily). I varied bundle sizes (10–200 gems) and parallelism levels.

In these tests, I wasn’t able to observe a significant or consistent performance difference between this branch and 4.0.9. Across runs, this branch was sometimes faster and sometimes slower. Most differences were small (generally within ~1s on ~10–20s installs), with a few outliers in both directions.

I may be mistaken, but my concern with the alternative approach is that it could introduce significant complexity and potentially mask build flakiness. Given that this is a regression causing build failures, it might be reasonable to prioritize a small fix first, even if there is some potential performance impact.