Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Double Rainbow #552

Closed
wants to merge 2 commits into from
Closed

Fix Double Rainbow #552

wants to merge 2 commits into from

Conversation

schneems
Copy link
Contributor

@schneems schneems commented Mar 24, 2017

A "double rainbow" deploy is when someone is using the same buildpack twice. Normally this is by accident. There are failure conditions defined in #511 the app has to have a json gem and a version of Ruby that is different from the version of Ruby being used by the buildpacks.

When this happens, this exception will occur when the second buildpack tries to execute:

remote: -----> Ruby app detected
remote: /tmp/tmp.Se0nOtPeri/bin/ruby: symbol lookup error: /tmp/build_c85cc44940028913e25caa54b9ff2142/vendor/bundle/ruby/2.3.0/gems/json-1.8.6/lib/json/ext/generator.so: undefined symbol: rb_data_typed_object_zalloc
remote:  !     Push rejected, failed to compile Ruby app.

If you look at the output, the failure is happening extremely early, this indicated that it was happening before any significant code had been run, otherwise we would have expected to see

remote: -----> Compiling Ruby

The exception only occurred when using v155 of the buildpack as the second buildpack (via heroku buildpacks:add).

Here's what was happening: The first buildpack runs and installs the app's json gem. The buildpacks are designed so that they make their contents available for the next buildpack. I.e. if you install node via a buildpack, then the next buildpack to execute is expected to have node on the PATH. The same is true of ruby and any gems installed.

So now the second buildpack executes. To do this it's own source code is loaded into memory, when it does this it hits a require "json" call that was introduced in v155 and tries find the json library. While this library ships with Ruby it is also a gem, if the gem is present on the system it will prefer the gem. So it loads the json gem from a different version of Ruby that is currently executing, it can't because the symbols are different, it errors out.

We can avoid this completely by not loading the json library, instead we can vendor in https://github.com/kr/okjson.

We mistakenly added yarn to the path even when it was not on the system.
This is super UGHHH. There are cases when a `json` gem has been installed already. One case is when two ruby buildpacks run.

Here's what I think is happening: The first buildpack sets the PATH so that future buildpacks can execute Ruby code. This means that now all ruby gems installed from the customer app are on GEM_PATH. Now when the next invocation of building ruby comes up it fires `require 'json'` ruby sees that there's a gem installed and tries to load it, unfortunately it's for a different version of ruby than the one the buildpack is running on.

One option to solve is by not requiring `json` but to vendor in a json library such as https://github.com/kr/okjson.
@joshwlewis
Copy link
Member

Tested, it does fix the example app mentioned in #551.

@joshwlewis
Copy link
Member

I agree with the sentiment here... ugh. Weird stuff is weird. Thanks for figuring it out.

@tt
Copy link
Member

tt commented Mar 27, 2017

Rather than using a different gem should the buildpack try to isolate itself from changes in the build environment? Could it set the appropriate GEM_PATH to only look where it wants to look?

@schneems
Copy link
Contributor Author

@tt here's an implementation #553

@schneems schneems closed this Apr 4, 2017
@tt tt deleted the schneems/fix-double-rainbow branch April 4, 2017 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants