Permalink
Find file
Fetching contributors…
Cannot retrieve contributors at this time
1048 lines (778 sloc) 39.1 KB

Supporting (native) gems in GNU Guix

Table of Contents

Introduction

This document explains how to go from creating a software package to creating a non-native software packaging system in GNU Guix. Examples of such systems are Python pip, emacs ELPA packages and (here) Ruby gems.

Ruby developers have multiple needs when running a system when it comes to Ruby versions and modules (called gems). GNU Guix has great support for versioning and control of the dependency graph. Here we discuss gem support in GNU Guix (this is a work in progress).

Basically we are aware of the following use cases after installing a version of Ruby (say 2.1.6):

  1. A user/sysadmin wants to install a gem in the GNU Guix store
  2. A user wants to install a gem using rubygems outside GNU Guix and local to $HOME
  3. A developer wants to use bundler which installs gems in the source tree for development and testing

In all cases, when Ruby code needs to include a file, resolving the location of the gem should start in 3, next 2, then 1. As an example of supporting a complex gem we use the example of Nokogiri which has over 1,000 installation related questions on Stack Overflow.

Current state

GNU Guix has native gem support through the rubygems-uri procedure and ruby-build-system!

That means you can create a simple package description which can install a native gem:

(define-public ruby-json
  (package
    (name "ruby-json")
    (version "1.8.3")
    (source
     (origin
       (method url-fetch)
       (uri (rubygems-uri "json" version))
       (sha256
        (base32
         "1nsby6ry8l9xg3yw4adlhk2pnc7i0h0rznvcss4vk3v74qg0k8lc"))))
    (build-system ruby-build-system)
    (arguments '(#:tests? #f)) ; dependency cycle with sdoc
    (synopsis "JSON library for Ruby")
    (description "This Ruby library provides a JSON implementation written as
a native C extension.")
    (home-page "http://json-jruby.rubyforge.org/")
    (license (list license:ruby license:gpl2)))) ; GPL2 only

More examples can be found here

The rest of this document describes the discovery path of building the nokogiri software through GNU Guix. The final package description is fairly simple!

GNU Guix installed gem

GNU Guix now build gems from a tar-ball using a Ruby build environment. Installing ruby-hoe

guix package -i ruby-hoe

creates a symlink to the executable sow in

/home/pjotrp/.guix-profile/bin/sow -> /gnu/store/ccx2ri5l8y6qbrzkw4p0gq8l647kxn0g-ruby-hoe-3.13.1/bin/sow

and the libraries are available through the symlink in

$HOME/.guix-profile/lib/ruby/gems/2.2.0/gems/hoe-3.13.1/ -> /gnu/store/ccx2ri5l8y6qbrzkw4p0gq8l647kxn0g-ruby-hoe-3.13.1/lib/ruby/gems/2.2.0/gems/hoe-3.13.1

Running sow means you’ll need to add the GNU Guix gem locations to the GEM_PATH, e.g.,

env GEM_PATH=~/.guix-profile/lib/ruby/gems/2.2.0 sow

Make the GNU Guix gem path automatically available

At this point the GEM_PATH is not set automatically. Guix prints the PATH after installing a package so you can add it to you environment. You can view the recommended paths for your profile with

guix package --search-paths

For end-users we may want to automate that.

To see the actual paths and libraries that a Ruby uses, check

guix environment ruby-2.1 --search-paths

Currently, to find gems, you have to set the GEM_PATH. I use a script for this.

Gem in $HOME

The gem tool comes with the GNU Guix Ruby installation. This means you can run gem after tweaking the local PATHs. An example can be found here. Run it as

. ruby-guix-env

Note: it also sets the path for GNU Guix global gems.

Supporting native gems

Native gems include some C or C++ code.

Currently the GNU Guix gem tool does build native gems. See below the infamous Nokogiri example.

Bundler

Bundler works in GNU Guix, both installed though Guix and through gems. See the Nokogiri example below. I prefer, however, not to use bundler at all. GNU Guix makes bundler obsolete. See the section on that below.

Important packages

GNU Guix provides a number of native gems. On my system I use

bigdecimal (1.2.6)
hoe (3.13.1)
i18n (0.6.11)
io-console (0.4.3)
json (1.8.1)
log4r (1.1.10)
minitest (5.4.3)
nokogiri (1.6.6.2)
power_assert (0.2.2)
psych (2.0.8)
rake (10.4.2)
rake-compiler (0.9.5)
rdoc (4.2.0)
rspec (3.2.0)
rspec-core (3.2.3)
rspec-expectations (3.2.1)
rspec-mocks (3.2.1)
rspec-support (3.2.2)
test-unit (3.0.8)

Note the notoriously difficult to support nokogiri gem. It was added through the exercise below.

The local gem installer (after setting paths) installs other gems, including the natively building cucumber gems. E.g.

builder (3.2.2)
cucumber (2.0.2)
cucumber-core (1.2.0)
diff-lcs (1.2.5)
gherkin (2.12.2)
io-console (0.4.3)
json (1.8.1)
multi_json (1.11.2)
multi_test (0.1.2)

Package Nokogiri

Introduction

Nokogiri is a great test case because the gem contains both Ruby and C files (that need to be compiled into shared library nokogiri.so) and it depends on external C libraries which are not always on a system (libxml2 and libxslt, for example).

But first a tip: because the build is reproducible you don’t have to fix a package in one go. Personally, I like to work incrementally at fixing dependencies. One at a time. GNU Guix always stops where you were last.

Set up the environment

First using the script I set up the Guix environment so it looks like

set|grep guix
ACLOCAL_PATH=$HOME/.guix-profile/share/aclocal
BASH=$HOME/.guix-profile/bin/bash
CPATH=$HOME/.guix-profile/include
GEM_PATH=$HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0:$HOME/.guix-profile/lib/ruby/gems/2.1.0/
GUILE_LOAD_COMPILED_PATH=$HOME/.guix-profile/share/guile/site/2.0
GUILE_LOAD_PATH=$HOME/.guix-profile/share/guile/site/2.0
LIBRARY_PATH=$HOME/.guix-profile/lib
LOCPATH=$HOME/.guix-profile/lib/locale
PATH=$HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0/bin:$HOME/.guix-profile/bin:$HOME/.guix-profile/sbin:/usr/bin:/bin
PKG_CONFIG_PATH=$HOME/.guix-profile/lib/pkgconfig

Note the PATH still contains /usr/bin for convenience, though you should be able to do without.

‘gem install nokogiri’

Trying a naive

gem install nokogiri

results in ‘ERROR: Failed to build gem native extension’ due to a failing libxml2 build. Nokogiri packages its own version of libxml2 and tries to compile that. One way to solve this error is by fixing the compile problem (the logs say it is libtool related), the other way is to install libxml2 in Guix and tell nokogiri where to find it.

guix package -i libxml2

Guix symlinks both static and shared libraries in ~/.guix-profile/lib/ so we tell nokogiri where to find them

gem install nokogiri -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xml2-lib=$HOME/.guix-profile/lib

First Nokogiri complains ‘libxml2 version 2.9.2 or later is highly recommended, but proceeding anyway’ - we can fix that later. Next error is missing libxslt, but now we know what to do

guix package -i libxslt

and

gem install nokogiri -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib

and now the build succeeded

nokogiri -v
# Nokogiri (1.6.6.2)

That was rather easy for a notoriously difficult gem! And it looks like we can formalize this in a Guix package. Note that I cheated a little. Since /usr/bin is still in the path I am (probably) still using some of the build tools of the underlying distribution (running gem on its own does not isolate the build). I could aim to fix that, but it will come out when we add a proper guix package anyway (guix builds are fully isolated).

One interesting check is to see what the nokogiri.so shared library that we built links against, e.g.

ldd $HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0/extensions/x86_64-linux/2.1.0-static/nokogiri-1.6.6.2/nokogiri/nokogiri.so

and validate all the paths are pointing at the GNU Guix store. You don’t want to mix in libraries that are non-guix because it suggests things are missing. Note that the Nokigiri documentation also suggests gem path options for

--with-iconv-dir=/path/to/dir --with-zlib-dir=/path/to/dir

as well as

--with-exslt-dir=/path/to/dir --with-exslt-config=/path/to/exslt-config.

But none of these were needed here.

‘bundle install’

Running bundler naively

 bundle install

results in the same library issues with ‘Gem::Ext::BuildError: ERROR: Failed to build gem native extension’. Bundler also needs to be told where to find the libraries.

The first try was to configure bundler by adding to .bundle/config

BUNDLE_BUILD__NOKOGIRI: "--use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib"

Unfortunately, this does not work as it does not prevent bundler for starting to build the libxml2. This should not happen with the –use-system-libraries option. To check the bundler setting see

bundle config build.nokogiri

:

Set for your local app (app/.bundle/config): "--use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib"

But somehow these do not get honoured by extconf.rb. After reading the source and some trying inside the build dir I found the environment variable

~/.gems/bundler/ruby/2.1.0/gems/nokogiri-1.6.1/ext/nokogiri$ env NOKOGIRI_USE_SYSTEM_LIBRARIES=1 ruby extconf.rb  --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib

resulted in

/usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory

predefs is part of the GNU C library (libc6), so it is perhaps strange it does not get picked up (well, Guix even isolates away the native system - go the Guix gcc compiler does not see /usr/include). Adding –with-opt-include=/usr/include/x86_64-linux-gnu does find it. Added that to bundler’s config

BUNDLE_PATH: $HOME/.gems/bundler/
BUNDLE_DISABLE_SHARED_GEMS: '1'
BUNDLE_BUILD__NOKOGIRI: " --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib --with-opt-include=/usr/include/x86_64-linux-gnu"

and ran

env NOKOGIRI_USE_SYSTEM_LIBRARIES=1 bundle

and the thing builds. Better even, also Cucumber builds and all the test pass for bio-vcf (the tool I want to ultimately package).

You may want to check config settings with

bundle config

Note we should have used predefs.h from the store glibc-2.21/include/stdc-predef.h instead. It looks like Nokogiri is using an older include. This suggests what needs to be done:

error: #error "Never use <bits/predefs.h> directly; include <stdc-predef.h> instead."

Creating the tgz-derived package

Fetch and unpack the tar ball

Now we now how gem/bundler builds Nokogiri we have a chance at building the package from source and bundling it into GNU Guix. The tar ball can be found on https://github.com/sparklemotion/nokogiri/releases.

Unpack the tar ball and extconf.rb builds with

cd ext\nokogiri
env LD_LIBRARY_PATH=$HOME/.guix-profile/lib LIBRARY_PATH=$HOME/.guix-profile/lib \
  NOKOGIRI_USE_SYSTEM_LIBRARIES=1 ruby extconf.rb \
  --with-xml2-include=$HOME/.guix-profile/include/libxml2 \
  --with-xslt-include=$HOME/.guix-profile/include/libxslt \
  --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib \
  --with-opt-include=/usr/include/x86_64-linux-gnu \
  --with-opt-include=$HOME/.guix-profile/include

and make

env LIBRARY_PATH=$HOME/.guix-profile/lib  make

check the linked paths

ldd nokogiri.so
      linux-vdso.so.1 (0x00007ffc9f3e1000)
      libexslt.so.0 => $HOME/.guix-profile/lib/libexslt.so.0 (0x00007fb6c45aa000)
      libxslt.so.1 => $HOME/.guix-profile/lib/libxslt.so.1 (0x00007fb6c436b000)
      libxml2.so.2 => $HOME/.guix-profile/lib/libxml2.so.2 (0x00007fb6c4006000)
      libpthread.so.0 => $HOME/.guix-profile/lib/libpthread.so.0 (0x00007fb6c3de9000)
      libdl.so.2 => $HOME/.guix-profile/lib/libdl.so.2 (0x00007fb6c3be4000)
      libcrypt.so.1 => $HOME/.guix-profile/lib/libcrypt.so.1 (0x00007fb6c39ad000)
      libm.so.6 => $HOME/.guix-profile/lib/libm.so.6 (0x00007fb6c36ab000)
      libc.so.6 => $HOME/.guix-profile/lib/libc.so.6 (0x00007fb6c330a000)
      libgcc_s.so.1 => /gnu/store/76afr0pfbnimz7rdad35y5yd753myjhk-gcc-4.9.2-lib/lib/libgcc_s.so.1 (0x00007fb6c30f4000)
      liblzma.so.5 => /gnu/store/h86jd7lyd6lny3yz30d44gi4b0mz73in-xz-5.0.4/lib/liblzma.so.5 (0x00007fb6c2ed1000)
      libz.so.1 => /gnu/store/yx7c449ds3psyrn40h4nfvsb7xqqzziy-zlib-1.2.7/lib/libz.so.1 (0x00007fb6c2cb8000)
      libgcrypt.so.20 => /gnu/store/r16v30hlw2d6n232rm37p53qy8rpr7f2-libgcrypt-1.6.3/lib/libgcrypt.so.20 (0x00007fb6c29db000)
      libgpg-error.so.0 => /gnu/store/63lp72xz64axrbrlvpyln449v42h0zbh-libgpg-error-1.18/lib/libgpg-error.so.0 (0x00007fb6c27ca000)
      /gnu/store/wiqbxcvzj3r35hd55yxzz919b1dv1hnv-glibc-2.21/lib/ld-linux-x86-64.so.2 (0x00007fb6c49de000)

as it should be - though with the GNU Guix package the .guix-profile’s will point to proper store locations.

Create the gem from source

nokogiri.so is the C-part of the gem. The Ruby part sits in ./bin and ./lib in the tarball. These can simply be copied into the GEM_HOME. But reading the current implementation of the GNU Guix ruby-build-system, it creates a gem first using a gemspec

rake gem:spec

create the gem

gem build nokogiri.gemspec

install using our earlier trick

env C_INCLUDE_PATH=$HOME/.guix-profile/include gem install --local nokogiri-1.6.6.2.20150629081149.gem -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib --with-opt-include=$HOME/.guix-profile/include

which (now) fails with

38:26: fatal error: linux/limits.h: No such file or directory
 #include <linux/limits.h>

Actually, this is not so bad. The environment gets picked up in a GNU Guix package, so let’s move on. The install path (mostly) works.

Write the GNU Guix package

Update and build the GNU Guix source

In the next step we start with an existing GNU Guix package so we can just fill in the missing pieces. First I synchronized the Guix source and checked out a new branch named nokogiri

git pull --recurse-submodules guix master
git checkout -b nokogiri

now we need to make sure the environment is correct (as described in ./HACKING.org)

make

Download the Nokogiri tar ball

make sure gnutls is installed

guix package -i gnutls
guix download  https://github.com/sparklemotion/nokogiri/archive/v1.6.6.2.tar.gz

which gives

/gnu/store/v2hc2imgzgar4srfh64svkvas4ha07xz-v1.6.6.2.tar.gz
1dpmmxr8azbyvhhmw9hpyk3dds577vsd6c312gh2s7kgjd98nd9j

Start writing the GNU Guix Nokogiri package

Then I copied an existing package from gnu/packages/ruby.scm and started filling in

(define-public ruby-nokogiri
  (package
    (name "ruby-nokogiri")
    (version "1.6.6.2")
    (source (origin
              (method url-fetch)
              (uri (string-append
                    "https://github.com/sparklemotion/nokogiri/archive/v"
                    version ".tar.gz"))
              (file-name (string-append name "-" version ".tar.gz"))
              (sha256
               (base32
                "1dpmmxr8azbyvhhmw9hpyk3dds577vsd6c312gh2s7kgjd98nd9j"))))
    (build-system ruby-build-system)
    (arguments
     `(#:tests? #f)) ; no test suite
    (synopsis "Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser")
    (description "Nokogiri parses and searches XML/HTML very quickly, and also has correctly implemented CSS3 selector support as well as XPath 1.0 support.")
    (home-page "http://www.nokogiri.org/")
    (license license:expat)))

Note the MIT license is also known as the X11 or expat license.

Test run the package

Now we have the package let’s see if it is there

./pre-inst-env guix package -A ruby-nokogiri
ruby-nokogiri   1.6.6.2 out     gnu/packages/ruby.scm:504:2

now build it

./pre-inst-env guix package -K -i ruby-nokogiri

the -K switch will keep the unpacked build directory. The first error pops up

ERROR: No files matching pattern:  "\\.gemspec$"

which makes sense, because earlier we had to run first

rake gem:spec

Check out the build by hand

The builder says that it kept build directory `/tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-0’. So in a different terminal do

cd /tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-0
. environment-variables

and you are at the state of the error (with environment). Running

rake gem:spec

it complains Gem::LoadError: Could not find ‘hoe’ (>= 0) among 9 total gem(s). It is interesting to note that the build is completely isolated from the rest of the system, so any dependencies not explicitely added will fail. To check run

set

And when you do add it, it will be visible to the package forever and this is key to the build being reproducible.

Note: you may need to change the permissions of the build directory to try stuff by hand. As root

chown user -R /tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-*

Fix dependencies

We have to add the hoe dependency first.

   (native-inputs
    `(("ruby-hoe" ,ruby-hoe)))

and retry the build. Now the new build is in /tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-1. This way we keep reiterating until the package works. One of the interesting errors was LoadError: cannot load such file – rake/extensioncompiler since we had not seen that earlier. That is part of the rake-compiler gem. The gems are listed in the Rakefile as

["hoe-bundler",     ">= 1.1"],
["hoe-debugging",   "~> 1.2.0"],
["hoe-gemspec",     ">= 1.0"],
["hoe-git",         ">= 1.4"],
["minitest",        "~> 2.2.2"],
["rake",            ">= 0.9"],
["rake-compiler",   "~> 0.9.2"],
["racc",            ">= 1.4.6"],
["rexical",         ">= 1.0.5"]

We have to add the necessary missing package(s) to GNU Guix. The rake-compiler package becomes something like:

(define-public ruby-rake-compiler
  (package
    (name "ruby-rake-compiler")
    (version "0.9.5")
    (source (origin
              (method url-fetch)
              (uri (string-append
                    "https://github.com/rake-compiler/rake-compiler/archive/v"
                    version ".tar.gz"))
              (file-name (string-append name "-" version ".tar.gz"))
              (sha256
               (base32
                "07lk1vl0jqcaqwjjhmg0qshqwcxdyr5kscc9xxm13m03835xgpf3"))))
    (build-system ruby-build-system)
    (arguments
     '(#:tests? #f ; needs cucumber
       #:phases (modify-phases %standard-phases
                  (add-before 'build 'remove-cucumber-rake-task
                    (lambda _
                      ;; Remove cucumber test file because the
                      ;; dependencies are not available right now.
                      (delete-file "tasks/cucumber.rake")))
                  (replace 'build
                    (lambda _ (zero? (system* "rake" "gem")))))))
    (synopsis "Building and packaging helper for Ruby native extensions")
    (description "Rake-compiler proivides a framework for building and
packaging native C and Java extensions in Ruby.")
    (home-page "https://github.com/rake-compiler/rake-compiler")
    (license license:expat)))

Note it needs to remove tasks/cucumber.rake to prevent those tasks from running. Also we override the build system because this package runs

rake gem

to create the gem instead of the default ‘gem build $package.gemspec’ as defined in ./guix/build/ruby-build-system.scm. After successfully installing that package we simply add the dependency to the nokogiri package.

Next phase the build complains that Errno::ENOENT: No such file or directory @ rb_sysopen - ports/archives/libxml2-2.9.2.tar.gz. This is because Nokogiri build wants to find the source and patch libxml2 for itself. Previously we used rake gem:spec, but that is no longer available with this later version, check rake task options with

rake -T

Running the Nokogiri build with

rake newb

results in a mini_portile (LoadError). Now mini-portile we don’t need (it is another packaging system). But I found out that running

rake gem

twice will generate the gem. Now the install phase fails on mini-portile. That means we need to replicate the earlier gem install command with its switches. First we need to add libxml2 and libxslt as dependencies.

This requires adding at the top of ruby.scm

#:use-module (gnu packages xml)

and inside the nokogiri package definition

   (inputs
    `(("zlib" ,zlib)
      ("libxml2" ,libxml2)
      ("libxslt" ,libxslt)))

resulting in a build with

The following files will be downloaded:
   /gnu/store/s4vwk3f0ainazh2czl5k5gsainpiby6i-libxml2-2.9.2
   /gnu/store/sprxqr56hm7p9wcy17bm2vj99k1mr779-libxslt-1.1.28

Nice. The build fails (of course), but now inside the build directory you can find the settings. E.g.

cat environment-variables

shows build variables, such as

export LIBRARY_PATH=(...):/gnu/store/s4vwk3f0ainazh2czl5k5gsainpiby6i-libxml2-2.9.2/lib

where dependencies can be found. To reference such a dependency you can add variables in the package. One example could be

(let ((libxml2 (assoc-ref inputs "libxml2"))
  do something

So the earlier –with-xml2-include switch can become something like

(string-append "--with-xml2-include=" libxml2 "/include/libxml2")

Note: At some point the guile REPL may come in handy to see what is happening. See the guix-notes HACKING guide for more information.

It is interesting to see what other packages implement. The ruby-git package adds an absolute path for the git binary with

(arguments
 '(#:phases (modify-phases %standard-phases
              (add-before 'build 'patch-git-binary
                (lambda* (#:key inputs #:allow-other-keys)
                  ;; Make the default git binary an absolute path to the
                  ;; store.
                  (let ((git (string-append (assoc-ref inputs "git")
                                            "/bin/git")))
                    (substitute* '("lib/git/config.rb")
                      (("'git'")
                       (string-append "'" git "'")))
                    ;; Fix a test that expects the binary to be simply
                    ;; 'git'.
                    (substitute* '("tests/units/test_logger.rb")
                      (("def test_logger")
                       (string-append
                        "def test_logger\n"
                        "Git::Base.config.binary_path = 'git'")))
                    #t)))
              (add-before 'check 'create-fake-home
                (lambda _
                  ;; The test suite runs 'git config --global' commands,
                  ;; so a fake home directory is needed for them to
                  ;; succeed.
                  (let ((fake-home (string-append (getcwd) "/fake-home")))
                    (mkdir fake-home)
                    (setenv "HOME" fake-home)))))))

After patching out the ‘mini_portile’ dependency from the Rakefile and adapting the gem install –local the next error was ERROR: While executing gem … (Gem::FilePermissionError) You don’t have write permissions for the /gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/gems/2.2.0 directory.

Great! We do not want the library installed inside Ruby, but in its own store. Thanks GNU Guix for pointing that out! We need to override the install-dir.

Inside the build directory the following worked after disabling the libxml2 check in extconf.rb (so another patch is required).

gem install --install-dir /tmp --local pkg/nokogiri-1.6.6.2.gem --
  --use-system-libraries
  --with-xml2-include=/gnu/store/s4vwk3f0ainazh2czl5k5gsainpiby6i-libxml2-2.9.2/include/libxml2

Note that the libxml2 include file is in a non-standard place, so it needs to be defined. Even so, less configuration is needed than the earlier build-by-hand exercise. Remember we had to specify

env C_INCLUDE_PATH=$HOME/.guix-profile/include gem install --local nokogiri-1.6.6.2.20150629081149.gem -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib --with-opt-include=$HOME/.guix-profile/include

GNU Guix’ input variable resolves the standard library and include paths. Rather then using /tmp, we also use the targetdir out.

Next thing we know the ruby-nokogiri package installs!

Fix the runtime gem path

After installing Nokogiri

./pre-inst-env guix package -i ruby-nokogiri
  The following package will be upgraded:
  ruby-nokogiri        1.6.6.2 → 1.6.6.2       /gnu/store/ynwfr9mfs5w3xhbxn1sgbcqrq0mh4gdx-ruby-nokogiri-1.6.6.2

the binary complains

$HOME/.guix-profile/bin/nokogiri
/gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/2.2.0/rubygems/dependency.rb:315:in `to_specs': Could not find 'nokogiri' (>= 0) among 9 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=$HOME/.gem/ruby/2.2.0:/gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/gems/2.2.0', execute `gem env` for more information
        from /gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/2.2.0/rubygems/dependency.rb:324:in `to_spec'
        from /gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:64:in `gem'
        from $HOME/.guix-profile/bin/nokogiri:22:in `<main>'

which means the nokogiri module is not found. Doing a

find ~/.guix-profile/lib/ruby/ -name nokogiri*

finds nothing. That is disappointing. The binary nokogiri wants to run the module as a gem from a GEM_PATH. In the store we find

/gnu/store/ynwfr9mfs5w3xhbxn1sgbcqrq0mh4gdx-ruby-nokogiri-1.6.6.2/gems/nokogiri-1.6.6.2/lib/nokogiri.rb

On my underlying Debian system the gem path is:

selinunte:~$ gem env
  RubyGems Environment:
  - RUBYGEMS VERSION: 1.8.23
  - RUBY VERSION: 1.9.3 (2012-04-20 patchlevel 194) [x86_64-linux]
  - INSTALLATION DIRECTORY: /var/lib/gems/1.9.1
  - RUBY EXECUTABLE: /usr/bin/ruby1.9.1
  - EXECUTABLE DIRECTORY: /usr/local/bin
  - GEM PATHS:
     - /var/lib/gems/1.9.1
     - $HOME/.gem/ruby/1.9.1

which means we need a system-wide path for gems - which does not exist in Guix. The Guix way is to create symlinks in a profile (usually ./guix-profile/), so the logical thing is to either symlink gnu/store/ynwfr(…)-ruby-nokogiri-1.6.6.2/gems/nokogiri-1.6.6.2/lib onto ~.guix-profile/lib/ruby/2.2.2/ or symlink a gems directory and add ~/.guix-profile/lib/gems/2.2.2/ to the GEM_PATH. Since this is clearly a gem path, I favour the latter. So we have to add that gem support to GNU Guix.

When I look into a standard bundler install it has the identical shared library 3x in

./bundler/ruby/2.1.0/gems/nokogiri-1.6.1/lib/nokogiri/nokogiri.so
./bundler/ruby/2.1.0/gems/nokogiri-1.6.1/ext/nokogiri/nokogiri.so
./bundler/ruby/2.1.0/extensions/x86_64-linux/2.1.0-static/nokogiri-1.6.1/nokogiri/nokogiri.so

not sure why that is - probably an artifact of nokogiri’s build system (can we now state it is a mess?). Only the first one is probably required.

Adding gem support to Guix profiles

Interestingly I find gems (and other directories) are symlinked in ./guix-profile! But they are at the root of the profile. To fix this all we need to do is ‘hoist’ the relevant directories inside the package into ./lib/gems/#{version}.

The ruby-rspec-core package does that. And reading the ruby-build system it does that (we overrode that with ruby-nokogiri):

(define* (install #:key source inputs outputs #:allow-other-keys)
  (let* ((ruby-version
          (match:substring (string-match "ruby-(.*)\\.[0-9]$"
                                         (assoc-ref inputs "ruby"))
                           1))
         (out (assoc-ref outputs "out"))
         (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0")))
    (setenv "GEM_HOME" gem-home)
    (mkdir-p gem-home)
    (zero? (system* "gem" "install" "--local"
                    (first-matching-file "\\.gem$")
                    ;; Executables should go into /bin, not /lib/ruby/gems.
                    "--bindir" (string-append out "/bin")))))

So, rather than overriding the install phase, we would be better of adding the one option it introduces for finding the libxml2 include! You can see the install phase has #:allow-other-keys, so we can modify the ruby-build-system to allow for an option we can name #:gem-install-option. The xpdf package does something similar

#:phases
 (alist-replace
  'install
  (lambda* (#:key outputs inputs #:allow-other-keys #:rest args)
   (let* ((install (assoc-ref %standard-phases 'install))
          (out (assoc-ref outputs "out"))
          (xpdfrc (string-append out "/etc/xpdfrc"))
          (gs-fonts (assoc-ref inputs "gs-fonts")))
         (apply install args)
         (substitute* xpdfrc
          (("/usr/local/share/ghostscript/fonts")
           (string-append gs-fonts "/share/fonts/type1/ghostscript"))
          (("#fontFile") "fontFile"))))
  %standard-phases)))

Note the ‘apply install’ calling back into the build-system.

Fixing it for Nokogiri is also a simplification because the underlying install method is used. We pass in the extra gem-flags parameter.

		 (alist-replace
		  'install
		  (lambda* (#:key inputs outputs #:allow-other-keys #:rest args)
			   (let* ((out (assoc-ref outputs "out"))
				  (libxml2 (assoc-ref inputs "libxml2"))
				  (gem-flags (string-append
					      "--use-system-libraries --with-xml2-include="
					      libxml2 "/include/libxml2"))
				  (install (assoc-ref %standard-phases 'install)))
			     (apply install #:gem-flags gem-flags args)))

We got a build!

Now we can install Nokogiri and run it after the GEM_PATH is set:

export GEM_PATH=$HOME/.guix-profile/lib/ruby/gems/2.2.0/
nokogiri

    Nokogiri: an HTML, XML, SAX, and Reader parser
    Usage: nokogiri <uri|path> [options]

    Examples:
      nokogiri http://www.ruby-lang.org/
      nokogiri ./public/index.html
      curl -s http://nokogiri.org | nokogiri -e'p $_.css("h1").length'

    Options:
	    --type type                  Parse as type: xml or html (default: auto)
	-C file                          Specifies initialization file to load (default $HOME/.nokogirirc)
	-E, --encoding encoding          Read as encoding (default: none)
	-e command                       Specifies script from command-line.
	    --rng <uri|path>             Validate using this rng file.
	-?, --help                       Show this message
	-v, --version                    Show version

Success!!

Feedback

After sending the patch around I got feedback from others. Most importantly, rather than overriding the install phase I should be using arguments.

IN PROGRESS Isolation of rubies

One interesting point of note is that nokogiri is using ruby 2.2.2 while I have ruby 2.1.6 in my profile. We’ll look into that later because we don’t want to mix the two. The resolution will be similar to that of Dave’s ‘gem-with-ruby’ procedure hanging around somewhere that does something like:

(define-public ruby1.9-nokogiri
  (gem-with-ruby ruby-nokogiri ruby-1.9))

It recursively overrides the Ruby version used for packages that use the Ruby build system.

Creating the gem-derived package

In the next phase we are going to modify the build system. In the Nokogiri exercise we were working from a downloaded tar-ball not native to rubygems.

Because nokogiri is such a complex beast, let’s try first with a simpler gem that has only a few dependencies.

Gem install log4r

Log4r is a long standing popular Ruby gem with few dependencies. Doing a simple

 gem install log4r

on my GNU Guix ruby install rapidly with

Fetching: log4r-1.1.10.gem (100%)
Successfully installed log4r-1.1.10
Parsing documentation for log4r-1.1.10
Installing ri documentation for log4r-1.1.10
Done installing documentation for log4r after 0 seconds

That looks promising. Now we want to make it into a proper GNU Guix package for posterity. The command line steps

wget https://rubygems.org/downloads/log4r-1.1.10.gem
gem install --ignore-dependencies --install-dir ~/tmp/gems log4r-1.1.10.gem

With Guix the destination will be ~/.guix-profile/lib/ruby/gems/2.2.0 or similar.

The ignore-dependencies switch is important because it prevents gem from fetching dependencies on its own. The GNU Guix rubygem packages we are going to create should take care of that.

Create a new rubygem-build-system

Basically I copied the existing ruby-build-system to facilitate gem installs. I’ll probably have to make one later again, but this is a good strategy for testing new things.

Getting rid of Bundler

Bundler aims to solve a problem that Guix solves - i.e. dependencies. The good thing about bundler is that it creates an environment which can be replicated for continuous integration (Travis CI). But it is not as robust as Guix can do - still, and a surprising amount of tooling depends on bundler. Checking an average bundler managed Gemfile of one of my projects it lists over 70 lines of dependencies. Seriously, that is the full gem dependency graph (but not the actual graph because it does not list lower depencies, such as the Ruby interpreter itself and glibc, which are indeed part of the more robust Guix graph).

The whole Ruby developer community uses bundler. With Guix, however, we have profiles and bundler can and should be thrown out.

To get rid of bundler in an existing project, remove the .bundler dir, the Gemfile and Gemfile.lock and remove bundler binary from the path.

gem uninstall bundler
guix package -r bundler (maybe ruby-bundler later)

Make sure it is gone

gem list
which bundle
bundle
  command not found

Next start removing bundler dependencies in the files. I had some in my test initialisation - make sure it is removed from all sources! Note, to get decent errors you may want to (temporarily) remove the exception catching in the cucumber source gems/cucumber-2.0.2/lib/cucumber/cli/main.rb.

After the fixes I could run cucumber without bundler (without the cucumber modification)! Make a note of the gems you need to pull in by ‘hand’.

Annoyingly bundler gets pulled in by some other gems (jeweler for one). So every time you run gem install locally make sure it is not there.

Note at any time you can remove the local gems and run gem list to see what Guix gives us:

rm -rf ~/.gem/p3vzqwxavyfchwjw2bxnq365sr1ap99b-ruby-2.2.2/
gem list

without jeweler, bundler and dependencies the list became quite a bit shorter.

Continuous integration (CI)

Travis CI without Guix

To test Rubygems in Travis with bundler, simply provide the Gemfile. Alternatively pull in dependending gems separately with gem in the .travis.yml file:

install:
  - gem install cucumber rspec regressiontest

This has the advantage of being faster and purer. Even better go the Guix way:

The Guix way

Guix has a build farm with CI. It even provides build logs, e.g., http://hydra.gnu.org/build/573030/log/raw

The final nokogiri package description

(define-public ruby-nokogiri
  (package
    (name "ruby-nokogiri")
    (version "1.6.6.2")
    (source (origin
              (method url-fetch)
              (uri (rubygems-uri "nokogiri" version))
              (sha256
               (base32
                "1j4qv32qjh67dcrc1yy1h8sqjnny8siyy4s44awla8d6jk361h30"))))
    (build-system ruby-build-system)
    (arguments
     ;; Tests fail because Nokogiri can only test with an installed extension,
     ;; and also because many test framework dependencies are missing.
     '(#:tests? #f
       #:gem-flags (list "--" "--use-system-libraries"
                         (string-append "--with-xml2-include="
                                        (assoc-ref %build-inputs "libxml2")
                                        "/include/libxml2" ))))
    (native-inputs
     `(("ruby-hoe" ,ruby-hoe)
       ("ruby-rake-compiler", ruby-rake-compiler)))
    (inputs
     `(("zlib" ,zlib)
       ("libxml2" ,libxml2)
       ("libxslt" ,libxslt)))
    (propagated-inputs
     `(("ruby-mini-portile" ,ruby-mini-portile)))
    (synopsis "HTML, XML, SAX, and Reader parser for Ruby")
    (description "Nokogiri (鋸) parses and searches XML/HTML, and features
both CSS3 selector and XPath 1.0 support.")
    (home-page "http://www.nokogiri.org/")
    (license license:expat)))

The latest version you can find in git.

Notes

Why is bundler installing in .gems?

It says so in the ./bundle/config file.

Add GNU Guix to the Nokogiri installation page

Nokogiri lists many solutions here. We should add ours.