Updating to Ruby 2.3.x

[ctompkinson]

Is there any plans to update travelling ruby to 2.3.x or later versions?

This is important as 2.2.x is difficult to use now that the FileUtils gem has been pulled from gem repositories by the maintainers (ruby/fileutils#2).

[bethesque]

Very interested in having 2.3 support.

[billiam13s]

+1. I am interested in 2.4 support. #62

[igolden]

+1

[SethWilson]

+1

[dsimm]

+1

[bethesque]

Given the last commit on this project was April 2016, I take it this is a dead project. I wonder if we can put out the call for some new maintainers? @FooBarWidget it looks like you've done the lion's share of the work, but are now otherwise engaged. Thank you for your work. Are you willing to work with some new maintainers if they can be found?

[bradland]

@ctompkinson Just a side note, FileUtils is in Ruby 2.2.x Std-lib, so you don't need gems: https://ruby-doc.org/stdlib-2.2.0/libdoc/fileutils/rdoc/FileUtils.html

[FooBarWidget]

Hi, sorry for the delayed response, my mailbox has been so full of old emails thanks to the fact that I am becoming a father. Yes I would very much welcome others to take over maintenance of this project. Sent from my Android phone. Op 23 sep. 2017 12:48 a.m. schreef "Beth Skurrie" <notifications@github.com

…

: Given the last commit on this project was April 2016, I take it this is a dead project. I wonder if we can put out the call for some new maintainers? @FooBarWidget <https://github.com/foobarwidget> it looks like you've done the lion's share of the work, but are now otherwise engaged. Thank you for your work. Are you willing to work with some new maintainers if they can be found? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#88 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADM-_NfgAgNmbWosL9O0rWVMUE7uQJks5slDlJgaJpZM4OteSU> .

[bradland]

Oh wow! Congrats, Hongli!

[bethesque]

Congratulations! You'll definitely need help!

Trying to think of where we could advertise for anyone interested. I'd love to, but already up to my eyeballs maintaining other libraries. Will put some feelers out.

[bethesque]

I know you'll by super busy @FooBarWidget, but a high level description of what would be involved to build a 2.3/2.4 package would be very helpful for helping us find someone. How would you describe it? Devilishly tricky? Just a matter of going through a known process? Does it involve a lot of trail and error? What sort of skillz does the candidate need? Ruby/C/anything else?

[FooBarWidget]

Let me do a quick braindump here on how things work on a high level.

Linux

The idea is that we first build a predictable, isolated, controlled build environment, and then use that controlled environment to build Ruby. The build environment is called the "runtime", which is created by the script linux/setup-runtime. The script linux/build-ruby uses the runtime and to build Ruby binaries from source.

The runtime is based on CentOS 5, because it contains an old glibc. The weirdest part is that we don't use this runtime directly. The linux/setup-runtime and linux/build-ruby scripts both invoke a Docker phusion/traveling-ruby, which is based on CentOS 6. Inside this container, we invoke the mock tool in order to spawn a CentOS 5 chroot. That chroot is the actual runtime that we use.

The reason why we do it like this, instead of invoking a CentOS 5 Docker container directly, is because back when Traveling Ruby was written there were no good CentOS 5 Docker containers available. Now that there are, this method is ripe for an update. See section "Holy Build Box" in this post.

Anyway, linux/setup-runtime (at the bottom) invokes the Docker container, which (inside the Docker container) calls linux/internal/setup-runtime. That script in turn spawns a CentOS 5 mock chroot which in turn calls linux/internal/setup-runtime-inside-mock. It is that script which is truly responsible for setting up the runtime. As you can see in that script, it installs a bunch of stuff that are necessary for compiling Ruby and various native extensions, such as autoconf, automake, SQLite, PostgreSQL client libraries, etc. If I remember correctly, mock already automatically installs GCC.

The biggest problem of the runtime, besides that it is invoked through a weird chroot-inside-Docker indirection, is that CentOS 5 is so old that its OpenSSL library does not support SNI. Many open source projects' tarballs nowadays are hosted on HTTPS servers with SNI. This means that linux/internal/setup-runtime-inside-mock is probably broken and is unable to download various source files. But see section "Holy Build Box" for more discussion about this aspect.

linux/build-ruby works in much of the same way. It invokes a Docker container, which uses mock to eventually invoke linux/internal/build-ruby-inside-mock inside a CentOS 5 chroot.

linux/build-ruby also builds native extensions. This is done by bundle installing the various Gemfiles under shared/gemfiles.

macOS

Things on macOS are simpler (and unfortunately, also more brittle) because we cannot use Docker and chroots. Like the Linux version, there is also a runtime and also a build-ruby script.

What we do here is to reset environment variables to predictable values, in the hope that this is enough to setup a predictable, controlled build environment. It is best run on a macOS system that's as empty as possible.

Tricks we use to make Ruby portable

The main trick we use to make Ruby portable is this: statically link all dependencies, except for ones that we can reasonably assume is already installed everywhere.

On Linux, we statically link everything except glibc (and related libraries such as libm, libdl, libpthread and a bunch of others), as well as libreadline (required by bash), libstdc++, libtermcap (lots of existing binaries already use this) and a bunch others.

On macOS, we statically link everything except libSystem and related stuff.

In linux/internal/build-ruby-inside-mock there is a sanity check at the end of verify that Ruby is indeed not dynamically linked to anything besides what we know is safe. MacOS's build-ruby script currently has no such check, so there is room for improvement there.

The build-ruby scripts also ensure that native extensions also statically link their dependencies.

It's usually a huge pain to tell compilers to statically link to a library. If you pass -lfoo to the compiler and there is both a dynamic and a static version, then the compiler will chose the static version. Also most open source projects' build systems don't have good support for linking to static versions of libraries; some of them contain bugs.

We deal with this pain in two ways. The controlled build environment is very important: you don't want the compiler to accidentally find any dynamic versions of libraries. Inside the build environment we ensure that as few dynamic libraries are installed as we can, and we use various compiler environment variables and flags (such as LIBRARY_PATH, -L) to ensure that compilers find the static libraries that we want. The order in which -L is passed in the compiler invocation matters, so we have to be careful and test things well.

Some open source projects' build systems refuse to play nicely with this, so we monkey patch them to do what we want.

It is especially a pain to tell gems with native extensions to link to the static libraries that we want. So far I've been able to manage this by meddling with BUNDLE_BUILD__* environment variables, but this feels very brittle.

Cannot use OS-provided static libraries

CentOS provides static libraries via YUM. Unfortunately we cannot use them for two reasons:

1. They are usually too old.
2. They are not built with -fvisibility=hidden. This is very important to avoid symbol conflicts. More about this later.

Therefore our setup-runtime scripts build the static libraries that we need, from source, while passing -fvisibility=hidden as compiler flag.

Some dependencies are actually dynamically linked

We don't actually statically link to all non-system dependencies. There are a couple of dependencies that we link dynamically to, such as libffi. The reason why I chose this is because there are multiple native extensions out there that use libffi. Having everybody statically link to libffi wastes space. So I have chosen to allow dynamic linking to libffi, while also distributing libffi with the Traveling Ruby binary tarball.

There is one caveat: when the user starts Ruby, the OS must be able to locate libffi. We don't want the OS to load the libffi that the user has already installed (which may be an incompatible version); we want the OS to load the libffi that we shipped. This is why the user doesn't invoke the Ruby executable directly. Instead the user invokes a wrapper script, which sets up a bunch of environment variables (such as LD_LIBRARY_PATH) to prior to executing the actual Ruby binary. You can see how this wrapper script is generated in linux/internal/build-ruby-inside-mock and osx/build-ruby, create_wrapper and create_environment_file.

If my memory is correct, I think that on the macOS side we used to link Ruby with -rpath and $ORIGIN, but I abandoned that approach for reasons I can't quite remember. I think it has got to do with not being able to specify the correct paths using this approach. So now we use DYLD_LIBRARY_PATH.

fvisibility=hidden

Thanks to the way dynamic libraries work on ELF systems such as Linux, symbol conflicts is a real concern. Suppose a program foo is linked to liba.so and libb.so. liba.so is linked to libcar.so and calls create_vehicle() from that library which is supposed to print "car". libb.so is linked to libbus.so and calls create_vehicle() from that library which is supposed to print "bus". What do you think happens?

1. The output is car, bus (or vice versa).
2. The output is car, car.
3. The output is bus, bus.

The answer is: either 2 or 3. But we actually want 1. As you can imagine this is very bad, and something we don't want to deal with.

By compiling as many things as possible with -fvisibility=hidden, we ensure that the dynamic linker does not lookup symbols from a library that we don't intend to. This eliminates symbol conflicts.

This also has the side effect of making executables smaller because no space is wasted on the dynamic symbol table.

See also http://blog.fesnel.com/blog/2009/08/19/hiding-whats-exposed-in-a-shared-library/

Adding a new Ruby version

Adding a new Ruby version should be very simple, and just a matter of extending the current process. But the devil is in the details. You have to look at all the notes that I wrote above and double check that everything is done is correctly. The ecosystem changes all the time, build systems change, so sometimes they break the mechanisms with which I try to ensure that dependencies are statically linked.

Windows

Things on Windows work completely differently. We don't try to build Ruby there at all because that's a too big can of worms. Instead our Windows support consists of a bunch of scripts that download and package binaries that have already been built by other people.

Holy Build Box

After Traveling Ruby, I started working on various Passenger-related packaging projects which also run into similar problems that I encountered with Traveling Ruby. Passenger provides generic Linux and macOS binaries, so I also have to think about setting up a controlled build environment there, and ensuring that Passenger is statically linked to the dependencies that I want.

I figured that it would be a good idea to extract a general system to its own project. Enter Holy Build Box. It works and it's used by Passenger. It does not use the weird chroot-inside-Docker trick, and instead uses CentOS 5 images directly. There is a lot of documentation. I even solved the OpenSSL problem there, by building the latest OpenSSL version from source and using that to build a new curl, which I then use to download other source files.

I've tried for a while to port Traveling Ruby's Linux side to Holy Build Box, but the effort stalled. What I've done so far is published in the 'holy-build-box' branch. Maybe someone can take over this.

I hope this brain dump helps. What skills a maintainer needs? I'd say Ruby and C, at minimum. A good understanding of how linkers and binary compatibility works is also a very good idea.

[FooBarWidget]

I just realized I forgot to say something extra regarding why we build our own static libraries instead of getting them from YUM. PIC code.

Ruby native extensions are shared libraries. On many operating systems, shared libraries must be compiled with - fPIC, meaning position independent code. That means if you link any static libraries into a shared library, those static libraries have to be compiled with - fPIC as well. The ones provided by YUM are likely not PIC.

[bethesque]

Wow, thanks for that. It's as complicated as I suspected. I'm asking around to see if I can find anyone.

[tigris]

there is a pre-existing fork somewhere that has ruby 2.3.1 as a release

in Jan this year i tried to fork that and do a ruby 2.4.0 build, but ran into a lot of issues during the mocking / chroot phases that blew my mind, not having much docker knowledge at the time

i shall endeavour to try again over the next few weeks given the amazingly detailed information above