Support cross-platform development

[xaviershay]

Simplest statement of problem: Allow windows developers to easily deploy to linux without having to do weird things to their lock file.

See rubygems/bundler#646 and rubygems/bundler#2040 for history, discussion, and some pointers for possible implementation.

[johnnyshields]

👍 to getting this resolved.

[sheerun]

+1. I would like to see something like this for Mac's too. Fot example I have problems installing libv8 3.11.8.13 under 10.9, so I use 3.16, but on the other hand on server 3.16 doesn't work on server, so I can't change it in Gemfile.lock.

[MichaelSp]

Suggestion: Maintain a dependency section for each platform in Gemfile.lock and warn/error if versions differ

[johnnyshields]

This one affected me again yesterday, when deploying cross platform I'm always forced to bundle update whether I want to or not. There's got to be a better way...

[phemmer]

As a possible workaround, how does bundler detect the architecture? Obviously you don't want to run bundle install for a foreign architecture, but when using bundle package --no-install faking the architecture would be a workaround.

[dcfsc]

+1 for @MichaelSp's suggestion. Easy to understand and obvious. We are using compass and need to lock down the SCSS version, so need the .lock file. We do not use rake, so a rake fix is not helpful.

[esumerfd]

This appears to have been a problem for a long time with no solution. As another sample of the problem I am running the pry gem on Mac, Linux and Windows. The Windows gem dependency includes win32console which causes the lock file difference.

Whether the Gemfile.lock is made smarter by isolating dependencies by platform or just allow me to add a override Gemfile..lock for additional platform specific gems. Either would work for me.

Since bundler creates this file, it should be able to diff the current version, know which platform it was created on, and make appropriate delta changes to the current platform. The output is a set of common gems and a set of platform specific gems.

[rb2k]

We ran into the same problem. I'd love to see something like "bundle package --architecture $ARCH"

[grossadamm]

👍 Would be very very helpful

[indirect]

This is very, very hard to fix in a reliable way. If you bundle install on a completely new platform, it means that you might end up with different code installed, and you have no idea if your application will work.

At a minimum, please run a CI server on the same platform as your production servers, and use the Gemfile.lock from a passing CI build when you deploy.

[moeffju]

👍

[janv]

👍

[salzig]

👍

[moeffju]

@indirect That would mean that your CI server would have to be able to commit to your repository. And you would not know the end result beforehand either.

The most common use case for this that I have seen is people using bundle package in combination with multi-platform gems. The easiest workaround currently is to just manually get the .gem file and put it in the vendor/cache directory. Adding an option to bundler to do just that, e.g. something like bundle package --arch=linux-x86_64,darwin-x86_64, should not be too hard, and solve the problem for a lot of people. The format of Gemfile.lock would not even need to be extended AFAICT.

tl;dr: Even if no solution is perfect, any solution is better than none.

[moeffju]

Addendum: For some gems, the version string that ends up in Gemfile.lock includes the platform (nokogiri comes to mind), for some (e.g. libv8), it doesn’t. I haven’t looked into why this is happening or even if this is still happening, but gems like libv8 are completely unproblematic here since all the different architecture’s gems fulfill the version condition. So maybe if bundler just always stored arch-independent version strings in the Gemfile.lock…?

[indirect]

@moeffju Darwin and Linux currently share gems only because the gem requires you to build C extensions at install-time. If there are different gems for different platforms, those gems can have completely different dependencies. For example, Nokogiri on windows depends on a gem that Nokogiri for Linux/Darwin does not depend on. Completely different dependencies means completely different lock file, and makes it a terrible idea to cache gems for a different platform, because you won't even know whether your production app can boot or not until after you've deployed it.

[moeffju]

@indirect Yes, but bundler can still pretend to run on a different platform and resolve those dependencies, can’t it (provided they are gem dependencies and not lib dependencies, but those are out of scope anyway)?

[indirect]

The entire reason Bundler exists is to make sure that you run the same code in development as in production. You're asking for Bundler to "just" let you run completely different code in development and production... at that point, you can just as well delete your Gemfile.lock before bundling during production deploys, and this entire discussion is pointless.

On Thu, Feb 26, 2015 at 2:55 PM, Matthias Bauer notifications@github.com
wrote:

@indirect Yes, but bundler can still pretend to run on a different platform and resolve those dependencies, can’t it (provided they are gem dependencies and not lib dependencies, but those are out of scope anyway)?

Reply to this email directly or view it on GitHub:
#4 (comment)

[johnnyshields]

Why not add a declaration cross_platform: true or something of that nature?

Node NPM's Shrinkwrap feature supports cross platform with no problem.

[TimMoore]

In Bundler 1.8, bundle package supports an --all-platforms option: rubygems/bundler@347c004

It looks like this hasn't made its way into the docs yet, though. Help with that would be appreciated!

[moeffju]

@TimMoore According to the spec and a cursory local testing, that only includes any other ruby platforms, but not all cpu arches. I.e. it does not install libv8-3.16.14.7-x86_64-linux.gem when run from OS X, but only libv8-3.16.14.7-x86_64-darwin-14.gem.

[moeffju]

@indirect I’m aware of what bundler’s intentions are, but what is the alternative? The current situation is that lots of people develop on a platform that is different from their deployment platform. Ruby as an interpreted language facilitates this, because your Ruby code usually runs no matter what the underlying platform is (with some caveats). Your scenario of “completely different code” running in dev and prod, how often does that really happen? libv8 on linux does the same as libv8 on darwin. It could theoretically generate PNGs of kittens instead, but – why would it? Maybe if you could give an example of a gem that works noticeably different on a different platform, that would be helpful for our understanding.

There is a pain point here, and all the suggested workarounds come with their own problems:

• deleting Gemfile.lock loses most of the benefits of bundler and makes deployments slower, network dependent and potentially brittle;
• having a CI on the target platform overwrite Gemfile.lock seems awfully indeterministic and would mean giving your CI server write access to your repo, which unnecessarily increases attack surface;
• manually editing Gemfile.lock is brittle and does not cover all cases;
• manually downloading the gem for the other platform works pretty much fine and makes the user responsible for testing that the combination works (which the CI would do anyway), but runs the risk of forgetting to update the gem version
• anything else?

Of all the alternatives, having bundler download and cache the gem for different platforms seems like the most obvious, simplest, and least brittle option. The only failure case here is if a) a gem does wildly different things on different platforms, which afaik is comparatively unlikely, and b) people deploy to a different platform for production without testing the app on that platform first, whether on a CI or through manual QA, which is not a great idea anyway.

If people want to shoot themselves in the foot, they will do it no matter whether bundler supports them or not. Adding multi-arch support to bundler at least hands them an unloaded gun pointing away from their body. So my recommendation would be to add an option (--arch=x,y,z or --all-arch), make it off by default, and document it well. No regression for those who don’t have the problem, a simple, straightforward, and officially sanctioned way to solve the problem for those who do.

[ignisf]

a) a gem does wildly different things on different platforms, which afaik is comparatively unlikely

For example libv8 runs fine on 64-bit Linux and segfaults on 32-bit Linux.

[moeffju]

@ignisf Well, while I would agree that that is wildly different per platform, it’s not “doing” different things, it just has bugs on some platforms. Bundler caching different platform’s gems does not preclude people from proper testing and QA – e.g. we run CI builds on the same platform/arch that we deploy on, plus run staging environments. If you don’t do that, your deployment may break for a variety of reasons.

[johnnyshields]

My project has 200+ gems and we have developers on Windows, Mac, and Linux without issue. Gem not working on different platforms with MRI is the exception not the rule.

[MichaelSp]

I think there are quite a few ways to go. What I think we should not do is to be awestruck facing the problem. While there may be bugs on different platforms, bundler should support those too. It's not bundlers fault. And it might help gem-devs to be more aware of the cross platform developers if it's officially supported by bundler.
So I agree to both of the opinions:

1. Bundler should make sure to have the same code in production as in development
2. Bundler should support cross platform development.

And it seems like to reach the second we have to sacrifice the first. But that should be the way to go, because as others already said:
Problems are the exception, cross platform development is not.
And bundler doesn't say you don't have to run CI, staging-area and manual testing.

TL;DR: Support and document cross-platform-development and write a big ⚠️ warning ⚠️ in the docs about the implications.

[TimMoore]

My own opinion (for what it's worth) is that I can understand the unmet need here, and an opt-in feature to support cross-platform development would be a reasonable compromise.

In practical terms, someone who has this need would need to commit to implementing, documenting and owning this feature long term. None of the currently active contributors to Bundler have this need and so we are not well-equipped to design this feature or support it. If our hunch is right that this would lead to a significant increase in issues filed against the project caused by misuse of the feature, we'd need help triaging and troubleshooting those issues as well.

(The above could be said about most open feature requests, to be honest.)

[moeffju]

@TimMoore Well, that sounds reasonable. Could you quickly summarize why the --all-platforms option does not satisfy this requirement and maybe give me a pointer where to start looking in the bundler source to properly implement this? I’ve had a look through the source and I’m not sure where to start. Pointers would be appreciated so I don’t have to look through the entirety of the source just to begin… :)

[TimMoore]

@moeffju I'm not sure why the existing --all-platforms option isn't getting gems for all architectures. Maybe it should be a separate option? It would probably be more useful to list the specific ones you want.

I think most of the logic would be in definition.rb.

[johnnyshields]

@TimMoore thank you for helping find a reasonable comprise here 🍻

[coezbek]

Suggested steps towards a solutions:

1.) Add option to pretend to be on another platform than current. (we can resolve for multiple platforms without much pain)
2.) Add option to merge different Gemfile.lock`s for an integrated file, with conflicts being highlighted.

[johnnyshields]

There should be an option at the top of your Gemfile multiplatform! which causes gems with multiple platforms, for example Nokogiri, to only consider the version number and ignore the platform-specificity for the purpose of Gemfile locking.

Since this change is opt-in, it would not affect existing users.

[indirect]

@coezbek there is no way to handle gems with different dependencies on different platforms, so that seems like a partial solution at best. :/ additionally, if you resolve a lock for a platform other than the platform you are on, you have never even tried executing the code that will be run when you deploy. you should be (at minimum) running bundle install and all of your tests on the same platform as the platform you deploy to, which would give you a correct lockfile.

@johnnyshields that's impossible, because some versions of nokogiri have completely different dependencies on different platforms.

[coezbek]

@indirect I guess I am getting closer to a conceptual understanding: You suggest that it is a bad smell when I want to lock down dependencies for another platform that I have not run on yet. My question is: How can I run on another platform without bundle install and pushing the resulting Gemfile.lock to the platform I would like to test? Our dev ops run a tight ship on the CI server (which is identical to the production platform, but not to the development platform): it is strictly non-interactive. They would likely kill me if I suggest that I would like to run a dependency resolution as part of testing/staging.

Again the question: How do I resolve for another platform, so I can actually try the code / run the tests on this platform?

[segiddins]

You don't at the moment -- you just delete the lock and run bundle install. Maybe once the plugin system is done you can write a plugin that does this for you?

[indirect]

@coezbek wait, are you saying that you don't run bundle install on your CI server? Because that does a dependency resolution.

The most practical solution that I am currently aware of to this issue is to set versions as required in your Gemfile, and then generate a new lock for your deployment platform as part of your CI process. It's not great, but we don't have any better solutions yet.

[tmbv93]

Here's a workaround that I quite like. It allows you to specify the platforms you don't want to stage Gemfile.lock files for, and unstages them on commit.

Benefits:

• You avoid cluttering up your repo with Gemfile.lock changes you don't want
• You can keep the Gemfile.lock files generated by certain platforms
• You avoid manually unstaging or ignoring Gemfile.lock
• You can override it when necessary

Drawbacks:

• Needs to be added to .git/hooks for every user
• Users of the excluded platform still need to run 'bundle install' after pull

[chrisgit]

Would love to see a solution to this as we develop on a Windows platform but have a need to vendor Linux gems.

The workaround is to go on to a Linux box, bundle on the Linux box and copy the files back to the Windows machine.

[coilysiren]

Closing in favor of rubygems/bundler#4517, please continue discussion there!

[phemmer]

Just curious, but why? This issue has all the discussion history.

[coilysiren]

@phemmer we are going to archive this repository, keeping it active in addition to bundler/bundler would require me to duplicate all the labels / milestones. Also having the features exist in a different repo from the bug reports is confusing to users (including myself, a few days ago).

[rebase-master]

As of now, I am yet to find a way to make bundle install work successfully on a linux server. The Gemfile was created on windows so every time it runs bundle install on linux I get the following error:

Your bundle only supports platforms ["x86-ming32"] but your local platforms 
are ["ruby","x86_64-linux"], and there's no compatible match between these two lists.