Platform-independent lockfiles for gems with pre-built binaries

[xanderflood]

I can't really provide reproduction instructions because of the nature of the issue, but the behavior seems well-documented. I'm including as much of the requested info as seemed relevant, please criticize me if you want more!

What you're trying to do: maintain an application which relies on a gem with pre-built binaries, and have the application's lockfile remain platform-independent so it can be checked into version control.
The command you ran: bundle install (for bundler version 1.14.0 through 1.16.0)
What you expected to happen: a platform-independent lockfile
What actually happened: a lockfile that contains more detailed information than needed

---

OUR ENVIRONMENT:
I work on a closed-source project that uses a private gemserver to deliver pre-built binaries to linux and OSX machines. The binaries are built by the user who pushes the gem, so each new version is pushed to the server once from a Linux machine and once from an OSX machine, using the CURRENT platform each time. Our gemserver's dependencies API returns two entries for each version, marked with "x86_64-linux" and "universal-darwin" as their platforms, as expected. It's essential that we support both platforms, and a pure-ruby gem with native extensions isn't an option, since the gem goes to many places where our source code can't.

OUR BUNDLER ISSUE:
The problem is that starting with the release of bundler 1.14, the name of the platform is included in the lockfile for the application using the gem, so the contents of the lockfile depends on which platform it was last bundled from. Before 1.14.0, the lockfile would read something like "our-great-gem (21.4.3)", and after that release it changed to "our-great-gem (21.4.4-x86_64-linux)", and it remains this way today in bundler 1.16.0.

PREVIOUS BUNDLER DEV DISCUSSION OF THIS FACT:
For this reason, we've stayed on Bundler 1.13 until now, and have finally decided to look seriously into what it would take to upgrade. In my research I found this thread in which it seems to have been decided that "pre-built binaries" is a feature of RubyGems that the Bundler team simply doesn't have plans to support because of how rarely it is used.

So here are my questions, thanks a lot to anyone who can help!

1. It seems like the lockfile "needs" this platform information because, in principle, the Darwin and Linux gems could have different dependencies and simply are not interchangeable. In our case, this isn't true - all the dependencies are open-source and pure ruby, and they are all the same dependencies. Is there something we can do (either in the gem-building step, our gemserver's dependencies API, or the application Gemfile) to inform bundler that these two gems are interchangeable from the point of view of dependency resolution, even though they differ in their contents?

2. This issue is tricky because it highlights a fundamental difference between what RubyGems tries to do and what Bundler tries to do. The real, honest, complete resolution of this problem would, it seems to me, require making changes to both of those tools: Bundler can only tell that the two gems are interchangeable if RubyGems tells it so, and RubyGems only cares about one platform at a time, so it has no reason to track that kind of information in the gem metadata. Is this something, as far as anyone knows, that is slated to be fixed when RubyGems 3.0 incorporates the Bundler project?

3. If not, our intention is probably to fork bundler to implement the feature we need. Would the Bundler team have any interest in supporting this functionality in the trunk? And if so, do you have any parameters for how to go about it? Obviously, as pointed out by question gem bundle doesnt work when you only have the latest prerelease version of the gem installed #1, platform information sometimes is relevant for dependency resolution, so turning it off for all gems would generally be a bad idea. Maybe it would be better to allow it as an option to an individual gem entry in the Gemfile. The way I envision it, you would use something like:

gem "our-great-gem", "21.4.4", ignore_platform: true

Bundler then queries the gemserver for our-great-gem, and gets two 21.4.4 entries back. If the two don't have identical dependencies, then "ignore_platform: true" shouldn't have been used, and Bundler fails with an informative message. If they have identical dependencies, then their platform difference is ignored and no distinction is made between them during dependency resolution or while writing to the lockfile. When it comes times to actually install the gem, the one for the current platform is used.

[colby-swandale]

Why not have both x86_64-linux and universal-darwin in the Gemfile.lock? That way your Gemfile.lock is not changing when you run bundle install?

See https://bundler.io/v1.16/man/bundle-lock.1.html#SUPPORTING-OTHER-PLATFORMS for more information

[colby-swandale]

I'm closing this. If you're still experiencing your original issue don't be afraid to re-open this ticket.

[xanderflood]

Hey thanks, we realized that solution shortly after I posted this, (sorry for not following up about that) and it's been working pretty well, but it has caused one issue we've not been able to resolve.

The issue is that bundle install will always fail unless both gems for both platforms are present on the gemserver. I totally understand why this happens (adding in both platforms instructs bundler that it should always verify dependencies for both platforms), but it'd be nice if there's a way to tell bundler to allow failures for the inactive platforms. The Darwin version is really just an auxiliary thing we create for the two developers who need it, and it's a shame that even in production environments, bundle install is checking for both of them now. I might be missing it, but I don't see an option here to do something like that.

Another case where this issue shows up is that there doesn't seem to be any way to use a local copy of the gem for development without disabling the other platform. If I use path: in my Gemfile to point to a local copy, bundle install will complain:

Could not find gem 'engine (= 22.1.10)' in source at ../engine. The source contains 'engine' at: 22.1.10 x86_64-linux

since of course my local copy only has the linux version of the gem, and I imagine it wants both of them.

The workaround for this is of course to bundle lock --remove-platform, and then add it back in when you're done. In addition to being a bit of a hassle, this is really precarious because it blows up if you do things in the wrong order. If I accidentally add in the path: option before removing the extra platform, it destroys my bundle, and every command I issue returns this error message:

/home/xflood/.rvm/rubies/ruby-2.2.2/lib/ruby/site_ruby/2.2.0/rubygems.rb:291:in 'find_spec_for_exe': can't find gem engine (>= 0.a) with executable bundle (Gem::GemNotFoundException) from /home/xflood/.rvm/rubies/ruby-2.2.2/lib/ruby/site_ruby/2.2.0/rubygems.rb:310:in 'activate_bin_path' from /home/xflood/.rvm/gems/ruby-2.2.2@patterns2/bin/bundle:23:in '<main>' from /home/xflood/.rvm/gems/ruby-2.2.2@patterns2/bin/ruby_executable_hooks:15:in 'eval' from /home/xflood/.rvm/gems/ruby-2.2.2@patterns2/bin/ruby_executable_hooks:15:in '<main>'

Once this happens, I haven't found any way to recover. So far, I have been completely deleting my gemset and re-installing all gems from scratch each time I make that mistake (:thankgodforrvm:). This bit might be a bug, but I don't understand the error message well enough to tell, and I haven't tried minimally verifying it yet.

[mikegee]

I'm closing this.

Should this be closed now?