Ruby 3.3.0-preview2 take 3

[schneems]

Binary and buildpack Context

Here's how we provide build ruby binaries. First this repo will:

• Download a ruby source tarball
• Build it into a binary (make install etc.)
• Zip/tar that binary up and upload it to S3 (filename is coupled to buildpack logic)

Then later the buildpack has to:

• Take the output of bundle platform --ruby and turn that into an S3 url
• Download and unzip that tarball and place it on the path

That means that this repo is coupled to:

• Ruby source conventions (like filenames)
• The bundler output (bundle platform --ruby)
• Any logic the buildpack uses to convert bundle platform --ruby to a download URL

Another big piece of context is that this is the first year we're trying fully automated binary build and upload steps. I knew when I automated the regular builds that we would likely have to come back to this logic. In prior years we've been able to manually adjust file names which meant inconsistencies were worked around manually.

First problem - ruby-3.3.0-preview2.tgz

I've tried to release Ruby 3.3.0-preview2 twice now. The first attempt changed no logic for the binary builder or the buildpack. It produced a file that was uploaded to S3:

$ curl -I  https://heroku-buildpack-ruby.s3.us-east-1.amazonaws.com/heroku-22/ruby-3.3.0-preview2.tgz
HTTP/1.1 200 OK

This doesn't work for us for several reasons. You cannot actually run bundle install with Ruby 3.3.0-preview2 locally or you would get an error:

$ cat Gemfile | grep ruby
ruby '3.3.0-preview2'
$ bundle install
Your Ruby version is 3.3.0.preview2, but your Gemfile specified 3.3.0.pre.preview2

Note that it says there's a difference in the input string in the Gemfile and what bundler thinks we put in our gemfile (-preview2 versus .pre.preview2). This is due to bundler replacing the dash with .pre..

Second problem ruby-3.3.0.tgz

When we started supporting prerelease versions of Ruby we used to use the eventual version for pre-releases. So in that scenario Ruby 3.3.0-preview2 would be uploaded to ruby-3.3.0.tgz. Then we asked people to put that version in their Gemfile like ruby "3.3.0". Seeing problem number one, I recalled this era and thought we had some edgecase tooling for it.

However, this strategy of using the plain version stopped working with Ruby 3.2 when bundler was checking and erroring on Ruby versions and a pre-release version. We maintained that strategy until Ruby 3.1, see a 3.1 changelog (https://devcenter.heroku.com/changelog-items/2292).

In Ruby 3.2 we could not longer ask people to use 3.2.0 to work around limitations in bundler (that it does not recognize the dash). Our solution below looks a lot like it did for our solution in Ruby 3.2 where we asked people to put an extra specifier in the Gemfile like ruby "3.2.0.preview3" (https://devcenter.heroku.com/changelog-items/2499). Note that this is a different string than 3.3.0-preview2 (dot, which is what bundler needs versus a dash which is the source file from the ruby ftp site).

Without recalling this change I put in the work to automate generation of a ruby-3.3.0.tgz file which uploaded fine:

$ curl -I  https://heroku-buildpack-ruby.s3.us-east-1.amazonaws.com/heroku-22/ruby-3.3.0-preview2.tgz
HTTP/1.1 200 OK

And even can be downloaded fine if you put ruby "3.3.0" in a Gemfile (and do not try installing locally). However you'll get an error when you try to deploy:

remote: -----> Using Ruby version: ruby-3.3.0
remote: -----> Installing dependencies using bundler 2.3.25
remote:        Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4
remote:        Your Ruby version is 3.3.0.preview2, but your Gemfile specified ~> 3.3.0
remote:        Bundler Output: Your Ruby version is 3.3.0.preview2, but your Gemfile specified ~> 3.3.0
remote:

In essence I had to re-learn what I had already learned last year (in the changelog above) that ruby "3.2.0.preview3" will work with both bundler and the buildpack. (Note that we're using a dot instead of a dash). You can see that's the name of the binary on S3:

$ curl -I  https://heroku-buildpack-ruby.s3.us-east-1.amazonaws.com/heroku-22/ruby-3.2.0.preview3.tgz
HTTP/1.1 200 OK

This year I refactored and re-wrote the binary build system to add tests (previously there were none) and learn how it all works. Last year I only observed inputs and outputs. I think what happened is that I manually changed the filename to match the output of bundle platform --ruby when running those versions locally and just matched the file and instructions to that. I didn't put in the time to understand the intricies of the behavior of all the systems involved.

This year, I correctly recalled that there was an edgecase, and remembered one of the solutions, but I failed to remember that specific solution didn't work with newer bundler versions.

Instead of simply getting things working, I'm investing in understanding and writing down WHY some of this behavior exists and where certain limitations are coming from.

Here we are today

That brings us to this PR. The solution looks a lot like Ruby 3.2. We will ask customers to put `ruby "3.3.0.preview2" in their Gemfile. Which works locally without raising a bundler error:

$ bundle exec rake "generate_image[heroku-22]"
$ bash rubies/heroku-22/ruby-3.3.0-preview2.sh
$ docker run -it -v $(pwd)/builds/heroku-22:/tmp/output hone/ruby-builder:heroku-22  bash
root@440d92753881:/# mkdir /tmp/unzipped && tar xzf /tmp/output/ruby-3.3.0.preview2.tgz -C /tmp/unzipped &&
export PATH="/tmp/unzipped/bin/:$PATH" &&
echo "ruby '3.3.0.preview2'" > Gemfile &&
bundle install
Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this machine.
[DEPRECATED] This Gemfile does not include an explicit global source. Not using an explicit global source may result in a different lockfile being generated depending on the gems you have installed locally before bundler is run. Instead, define a global source in your Gemfile like this: source "https://rubygems.org".
The Gemfile specifies no dependencies
Resolving dependencies...
Bundle complete! 0 Gemfile dependencies, 1 gem now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

And importantly, it produces the same version output:

$ bundle platform --ruby
ruby 3.3.0.preview2

While it's not revelatory, it is not automated and tested. I also wrote the system to convert these version strings between the ruby source download and bundler version strings. That means that a Heroku developer using 3.3.0-preview2 or 3.3.0.preview2 will get the same output (correct) result uploaded to S3 and a working changelog to go with it.

[schneems]

I would still like to do a few things:

• Refactor the RubyVersion class into other more focused classes. For example:
  • TarRubyForS3
  • RubySourceContents
  • ComparableRubyVersion
• Make the DockerCommand module more of a builder style class where it helps to concatenate commands and deals with shell escaping
• Add either a test or output to actually use the suggested ruby version in a Gemfile within the container to ensure that what we're telling customers to do actually works (this would be easier if DockerCommand was a builder instead of a static set of hardcoded commands.

That being said. I want to ship preview3 sooner than later and this gets us there.