Binary packages #42

Open
sstephenson opened this Issue Sep 17, 2011 · 47 comments

Projects

None yet
@sstephenson

I'd like to add binary package support to ruby-build. I'd guess that most ruby-build installations are on one of two platforms—Mac OS X x86_64 or Linux x86_64—and most people really don't need to compile their own Ruby runtimes. Binary packages would make it much quicker to get up and running.

In order to do this, we'll need a command to build a definition into a temporary directory and make a tarball. We'll also need a way to download and install package tarballs from the internet. This implies having package repository sources, an architecture naming system, and an index of packages with checksums.

@sstephenson

Looks like --prefix is used to define the initial load path. It's compiled into the ruby binary itself:

$ strings bin/ruby | grep rbenv
/Users/sam/.rbenv/versions/1.9.3-dev
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby/1.9.1/x86_64-darwin11.1.0
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin11.1.0
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/1.9.1/x86_64-darwin11.1.0

Obviously these paths aren't going to be the same for everyone. Can we work around this?

@sstephenson

Crazy idea: build Ruby with an unreasonably long --prefix, like

/ruby-build/placeholder----------------------------------------------------------------/1.9.3-rc1

...and search-and-replace that with the actual installation location after extracting the package.

We'd need to do this for the shebangs for commands like irb too.

@josh

Skeptical about that. Compiled constants are going to be offset by their length. So replacing the placeholder with a shorter string is probably going to fuck up the memory addresses off all the other constants after it in the data segment.

@sstephenson

We'd keep the strings the same length, obviously, and just replace the rest of the space with null bytes.

Proof of concept: https://gist.github.com/a95fd5e818f9237a1c9f

Save as rewrite.sh and run like so:

$ ./rewrite.sh < bin/ruby > bin/ruby2
@clowder

I was digging round the MacRuby source today and came across install_name_tool and instantly thought of this ticket.

On MacRuby it is being used to replace linking to the system MacRuby with one packaged inside the executable (see MacRuby/MacRuby@master).

EDIT: Also useful headerpad_max_install_names

@fschwahn

so... what happened to this? isn't it ready to go, I remember you tweeting about this a few month ago? if not, is there anything that needs to be done?

I really like the prospect to install ruby in a few seconds instead of a few minutes :)

@john-griffin

Is there anything we can do to help out with this and get it finished?

@haberbyte

Here is an interesting blog post: http://yehudakatz.com/2012/06/

I think Yehuda has solved that problem already!

@petejkim

is it possible by any chance to use rvm's precompiled packages?

@rosenfeld

When using precompiled packages ruby-build should probably build them with optimization flags (-O3 or -O2) turned on.

Also, they should be also the default when compiling from source. In that case, it would be a good idea to also add -march=native.

@pranas

Any progress? It's highly desirable feature.

@ptrf

👍 Any progress? Post a gist maybe?

@barttenbrinke

+1 Rubber (Ec2 automation) recently switched to Ruby-build (which is great!) but compiling Ruby 2.0 for each Amazon instance you spin up is something the world can really do without.

@loopj

👍 would love to see support for this

@sheerun

Support for precompiled ruby would be awesome. +1

@sheerun

I made plugin for that https://github.com/sheerun/ruby-download It uses binaries precompiled by Boxen.

@jessmartin jessmartin referenced this issue in virtual-world-framework/vwf Nov 12, 2013
Closed

Improve the installation experience #69

@garnieretienne

I also made a plugin to download precompiled rubies from the RVM repository: https://github.com/garnieretienne/rvm-download. It's pretty basic but I can add functionnalities if other people are interrested (like listing available precompiled rubies or checksum verifications)

@kiranos

as @habermann24 stated in his link: http://yehudakatz.com/2012/06/

Fortunately, the C Ruby folks know about this problem, and include an (undocumented?) 
flag that you can pass to ./configure, --enable-load-relative. This flag fixes the problems
with hardcoded paths:

Adding precompiled binaries would absolutely bee a great new feature. What technical difficulties are still present to make this happen other than structuring it?

@mislav

I'd love if we had precompiled binaries support. It seems that a lot of people on this thread did some research, and it would be great if someone submitted a pull with a proof-of-concept:

  1. What are the flags that we should use when compiling Ruby to make it reusable for other people
  2. How do we match a user's system to a precompiled Ruby version?
  3. What about dependencies such as libyaml and openssl? We install those explicitly for some Ruby definitions but also sometimes use system/Homebrew ones. For precompiled Rubies, we must include all dependencies. How do we precompile a Ruby version with consistent zlib/libyaml/openssl support?
@errm

So I had a look at what rvm are doing, with https://github.com/rvm/rvm-binary, they are setting a bunch of flags with rmv install --movable I have had a trawl through the source to work out what flags they are setting but I can't quite work it out, it looks like they are setting CFLAGS="-fPIC" LDFLAGS="-fPIC". Also what they are doing only seems to work for 1.9.3 and above. They are using smf to install dependencies.

I created a gist with the config from the binaries built in this way on ubuntu 14.04 https://gist.github.com/errm/75474a643b8e5d09a444

The generated binaries work fine once the simlink in /lib/ruby/gems/*/cache is removed and replaced with a dir . . .

@michiels

I am currently working on this problem and my goal is to be able to provide/host Ruby binaries that are built with ruby-build so they can be used as rbenv binaries.

The first step was testing if the compile options in the previous comments on this issue would produce a sharable Ruby directory.

I have set up a a build infrastructure that can build Ruby versions on Ubuntu 12.04 and Ubuntu 14.04. I am publishing those binaries at this location: https://intercityup.com/binaries/ruby/ with PGP signed SHA1 checksums.

The binaries are compiled with

RUBY_CONFIGURE_OPTS=--enable-load-relative ruby-build <version> <version>

It would be great if someone could test those binaries as "drop-in" placement into their .rbenv/versions directory to see if those work. I have already confirmed them personally by downloading them into my rbenv installation, installing some gems like bundler and rails and set up a Rails app with the native mysql2 extension.

On a Ubuntu 14.04 box or Vagrant you can test a binary version with:

$ cd ~/.rbenv/versions/
$ wget https://intercityup.com/binaries/ruby/ubuntu/14.04/x86_64/ruby-2.1.2.tar.bz2
$ tar jxf ruby-2.1.2.tar.bz2
$ rbenv versions
# Verify 2.1.2 shows up!
$ rbenv shell 2.1.2
$ ruby -v

I would really like to hear if they work and what people's thoughts are on:

  • Hosting (I'm happy to sponsor the primary CDN from https://intercityup.com/binaries/) but I can understand if the best place for this is the mirror 37s sponsors via CloudFront.
  • Implementing as a plugin (like rbenv download) or
  • Maybe implement rbenv install --binary

Thanks for testing in advance and I hope we can make rbenv binaries happen real soon!

Michiel

@mislav
@michiels
@mislav
@michiels
@brocktimus

lsb_release sounds alright for Ubuntu and similar but I'm not sure if all Linux have them.

Can confirm it isn't available on the RHEL variants, at least it can't be relied upon. It's available in an RPM but not installed by default on minimal at least.

Some details available here http://serverfault.com/questions/89654/what-version-of-rhel-am-i-using.

@michiels
@brocktimus

Thanks! Is the file /etc/lsb_release available? We might need to build a lot of logic just for OS recognition. Or we can use an external dependency.

Nope.

I don't know if a dependency of lsb-release would be fine if you wanted the extra functionality. But I'm unsure how far it gets you compared to the alternatives. The other alternatives don't seem that bad since you can use lsb_release if it exists or otherwise check /etc/debian_version or /etc/redhat-release to grab the entire family. I have no idea about OSX though ;-).

http://danielgibbs.co.uk/2013/04/bash-how-to-detect-os/
http://unix.stackexchange.com/questions/6345/how-can-i-get-distribution-name-and-version-number-in-a-simple-shell-script

@michiels
@garnieretienne

For info, this is how RVM detect the current OS to download its precompiled ruby in bash:
https://github.com/wayneeseguin/rvm/blob/98c0a3aadfcade0b9b34701978c4eacfbcc17eba/scripts/functions/utility_system#L3

They seems to get the following data: system info, system type, system name, system version, system arch.

  • They use the uname command to get the system type (Linux, Darwin, Windows...)
  • If Linux is detected as the system type, they use a number of file name and content to find out which is the system name and version (and sometime system arch) (/etc/lsb-release, /etc/altlinux-release, /etc/os-release, /etc/debian_version, ...)
  • If non Linux they use dedicated command for each system to find system name, version and arch

At the end they sanitarize each metadata and build the download URL using them.

@mislav

Here's how I intend to extract basic OS information in ruby-build for build output purposes:

os_information() {
  if type -p lsb_release >/dev/null; then
    lsb_release -sir | xargs echo
  elif type -p sw_vers >/dev/null; then
    echo "OS X $(sw_vers -productVersion)"
  else
    local os="$(cat /etc/{centos,redhat,fedora,system}-release /etc/debian_version 2>/dev/null | head -1)"
    echo "${os:-$(uname -sr)}"
  fi
}

Of course, such a simplistic approach won't be enough for differentiating between binary builds. For example, it misses the information about system arch.

@martinb3

@sheerun Did ruby-download go away? It sounds useful!

@sheerun

You probably mean ruby-download. I removed it because Boxen removed most of their builds.

I could reintroduce it, and change provider to http://phusion.github.io/traveling-ruby/ BUT it supports only Ruby 2.1. So it all depends whether @FooBarWidget wishes to support more ruby versions.

@martinb3

@sheerun It looks like TravisCI and rvm.io also provide binary builds.

@sheerun

@FooBarWidget Do you know the difference between Travelling Ruby build & Travis?

I can't find rvm's builds. @martinb3 Do you know the API for them?

@martinb3

@sheerun I know they're hosted here: https://rvm.io/binaries/

@FooBarWidget

@sheerun No. Traveling Ruby's scope is limited to distributing apps. It's not meant to distribute Ruby binaries for general-purpose or development use. For that reason, we also have no intention to distribute more than a single version.

@garnieretienne

@sheerun @martinb3 I've made this plugin to download and install rvm binaries for rbenv:
https://github.com/garnieretienne/rvm-download

It use code from rvm to generate an URL and download the requested version of ruby from https://rvm.io/binaries/.

@jasonkarns jasonkarns referenced this issue in nodenv/node-build Mar 27, 2015
Closed

Download official binaries #11

@schlueter

Any movement on this?

@vemv vemv referenced this issue in rbenv/rbenv Nov 8, 2015
Closed

are you serious guys? #794

@cybertk

+1

@mconigliaro

Wow, this still isn't solved? To be honest, I don't even care about getting your binaries to work on my machines. If there was just a simple way to check for a precompiled version of Ruby before building (e.g. RUBY_BUILD_BINARY_URL?), I'd just build my own and host them myself.

@md5
md5 commented Apr 4, 2016

@mconigliaro If you want to do that, you can use rbenv by itself and not use ruby-build since its purpose is to build Ruby for rbenv.

From the rbenv docs:

Alternatively to the install command, you can download and compile Ruby manually as a subdirectory of ~/.rbenv/versions/. An entry in that directory can also be a symlink to a Ruby version installed elsewhere on the filesystem. rbenv doesn't care; it will simply treat any entry in the versions/ directory as a separate Ruby version.

@mconigliaro

@md5 oh yeah, that's a good point. I could always just have my config management system download and unpack a set of precompiled versions to ~/.rbenv/versions/. Thanks!

@jasonkarns
jasonkarns commented May 18, 2016 edited

Super old issue, but I just wanted to pop in and say that node-build (direct fork, minimally modified) has added support for installing precompiled binaries. nodenv/node-build@474e872 (source has been modified slightly since that PR was merged, though)

We grab the system info via:

platform(){
  local arch="$(uname -m)"
  local os="$(uname -s | tr '[:upper:]' '[:lower:]')"

  case "$arch" in
    x86_64 | amd64 | i686-64 ) arch=x64 ;;
    i[36]86* | [ix]86pc )      arch=x86 ;;
  esac

  echo "${os}-${arch}"
}

And a build definition looks like:

binary darwin-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-darwin-x64.tar.gz#658686795fac9669d411ac5c5be2da8159058e386767322d8d8151dcdb4810b9"
binary linux-arm64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-arm64.tar.gz#05df4aeb8a53798f8b10074600518040fc317f2919f9755aeab57b0aaf7227b0"
binary linux-armv6l "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-armv6l.tar.gz#6f29286464efda4a362d9464d82733398b98882051c20c0f190004480b6e506a"
binary linux-armv7l "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-armv7l.tar.gz#fb4fbef9306962e800804ab5ba615c06bba28deb4e7a3e945a291dba986ef816"
binary linux-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x64.tar.gz#e766e387934e17daaad92d0460ed76f756655da62b627a5c9cc07faea4a0b824"
binary linux-x86 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x86.tar.gz#97b5ccea7044073c87a21bcc4b0762f4a6bd77db9cc958206f684ecdfeb89b1f"
binary sunos-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-sunos-x64.tar.gz#acec44790ff0069620c0fd03945d14b9f97c7ccb0a0450f766a5cbe4a906510b"
binary sunos-x86 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-sunos-x86.tar.gz#d7bf8dcf353115b5e55bf64d25c2c34fa2d237e201e46dab97ae3d8e3f051583"

install_package "node-v4.2.1" "https://nodejs.org/dist/v4.2.1/node-v4.2.1.tar.gz#8861b9f4c3b4db380fcda19a710c0430c3d62d03ee176c64db63eef95a672663"

How it works:

The binary function is invoked multiple times from the build definition; given the platform and package url. If the binary's platform matches the current system, the package url is saved to a global variable.

Then when install_package is invoked, if a global BINARY_URL has been set (and the user hasn't forced a compile with --compile), it does an install_package ... copy.

It's a pretty hacky workflow, and I would love to find a more elegant solution. But it works and, I think, could be adopted for ruby-build.

@mislav

@jasonkarns Thanks for sharing! That's an interesting approach to fetching precompiled binaries.

In my opinion, it's not such a problem to make ruby-build fetch precompiled binaries as it is to make a system to package and publish precompiled binaries in the first place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment