Create a package for Rubinius.

[ghost]

I am working on creating a Nixpkgs package for the Rubinius language interpreter:
Rubinius GitHub. Here is my progress so far: https://gist.github.com/zacts/d9cf55a7312e78699e478be431fe2ec9 and https://gist.github.com/zacts/5b5a8a64ad7941821ba34e0f78404746.

[manveru]

I played a bit with the derivation and got something that gets a bit further in compiling:

{ clangStdenv, fetchurl, llvm, openssl, bundlerEnv, git, which }:

let env = bundlerEnv {
  name = "rubinius-env";

  gemfile = ./Gemfile;
  lockfile = ./Gemfile.lock;
  gemset = ./gemset.nix;
};

in clangStdenv.mkDerivation rec {
  name = "rubinius-${version}";
  version = "4.3";

  src = fetchurl {
    url = "https://github.com/rubinius/rubinius/releases/download/v${version}/${name}.tar.bz2";
    sha512 = "27033e1529c5c85f387af47d6cfadab4d55fd8214eda33670d43b8214cdb8a0c887f5343e7cda84483988430cb7808d80a7574d3d4b713cadaeecf97ebc70b69";
  };

  doCheck = true;

  nativeBuildInputs = [ env.wrappedRuby env which llvm git openssl ];

  patchPhase = ''
    patchShebangs .
  '';

  configurePhase = ''
    ruby configure
  '';

  buildPhase = ''
    ./build.sh
  '';

  installPhase = ''
    mkdir -p $out
    cp -r $src $out
  '';

  CXXFLAGS = "-Wno-unused-command-line-argument";

 meta = with clangStdenv.lib; {
  description = "Rubinius is a modern language platform that supports a number of programming languages, including Ruby";
  homepage = https://github.com/rubinius/rubinius;
  license = with licenses; [ mpl20 bsd3 ];
  paltforms = platforms.linux;
 };
}

[ghost]

@manveru is there a particular reason you are invoking ruby configure? I think that manually invoking configure is being depreciated in the source tree. The README.md metions only invoking ./build.sh --prefix=/path/to/install/dir as the preferred way to do this. (Note: I'm testing your above version now).

[brixen]

@Zacts thanks for starting this!

@manveru hey! I was surprised to find you here!

A couple notes about building:

1. We really really really want to remove the Ruby dependency, and that's part of the reason I created build.sh, which for will forward everything along, so if you use it, you'll be future-proofing the recipe, but we can always assist with changes;
2. By really really want to remove Ruby, I mean that there will be zero need for Ruby to build Rubinius, and that has already been started, which means that to bootstrap Rubinius requires an existing CodeDB archive. The release tarballs contain it, so that shouldn't be a problem. But if there's some testing phase, etc. that doesn't use the tarball, the process will need a network connection (something other package managers frown upon);
3. It's unlikely that configure will remain, but a first step will likely be converting configure to shell. Again, using build.sh (even though it's literally a shell of a shell right now) is going to be better;
4. There are other shell scripts in the ./scripts directory that may need to run, so beware those shebangs.

Is there anything I can help with?

[manveru]

Hey Brian, long time no see :)

The problem on darwin is that it doesn't accept scripts as shebangs, and the bundlerEnv.wrappedRuby we're using is a script that makes sure the paths for the gems are set correctly before invoking Ruby itself. That's why the shebang patching patching doesn't work.

Since I'm forced to use darwin this week for work anyway, I might as well try to get it to also work on that platform, but it will require some changes in how we setup environment variables and use bundlerEnv. I haven't used rbx or darwin for ages now, so will take a bit of time to get up to speed again.

I'm sure the work we're doing here can benefit a lot of people, that's why I'm chiming in :)

[ghost]

Thanks @manveru! 😄

[manveru]

So here's the current progress: https://gist.github.com/manveru/b8090e1ff0c614b11094c7e6ec113f55 I'm not sure why it currently fails (some other missing executable?), but at least it gets pretty far.

[manveru]

I probably won't get time today to continue much further, given that the C++ takes ages to compile as usual, so feel free to continue on it as you like.

[ghost]

I'm testing your latest version on NixOS Linux now.

[ghost]

Ok, here is what I'm getting: https://gist.github.com/zacts/05814823b4d81fe251c850a11013015c.
I modified default.nix to contain rake, which prevented an error. I also used: $ $(nix-build '<nixpkgs>' -A bundix --no-out-link)/bin/bundix --magic to generate the gemset.nix. I'll look into this more tomorrow. Thanks.

[ghost]

I don't know if rake conflicts with the Nix philosophy of packaging ruby dependencies. I can post the error that I get without rake as well.

[manveru]

My Gemfile is just

source 'https://rubygems.org' do
  gem 'rake'
end

I then run nix-shell -p bundler bundix --run 'bundle lock && bundix'.
There's no need for the rubinius gem to be used, afaict since it's all already part of the tarball.

[brixen]

@manveru from your gist above, Rubinius built completely and installed the pre-installed gems, so that is great success! The build failed when trying to compile the VM (C++) tests, but there's no indication of why it failed.

If you want to test that part without rebuilding everything (does Nix blow away the build dir?), you can run rake vm:test, which will only rebuild Rubinius C++ components that need to be built and then compile the C++ tests.

[brixen]

@Zacts did adding perl to the build fix the issue generating the VM C++ tests?

[ghost]

@brixen I'm testing the build now. I'll post the results in just a sec.

[ghost]

@brixen I'm getting a build error: gist

[brixen]

@Zacts hm, that's odd, @manveru's gist shows a complete build.

[ghost]

I am on NixOS Linux and from what I gather @manveru is on MacOS. I don't know if this might make a difference here.

[ghost]

Let me try this with a fresh nixpkgs repo. I guess my previous above Gemfile that included gem 'rubinius' left behind a vendor/ directory in pkgs/development/interpreters/rubinius/. I don't know if this might be conflicting in any way. I'm going to try everything from a fresh nixpkgs repo.

[ghost]

I have forked nixpkgs. I wonder if we should eventually move to using that instead of gist.

[ghost]

gist is good for quick edits though.

[ghost]

I wasn't expecting to find a vendor/ directory appear in there, which I assume was created by the bundix command that I ran. I want to make sure that we don't have slightly different setups from things like that now that we are dealing with multiple files. I'll have to be careful of this.

[ghost]

@brixen I'm still getting the same error. I don't know what the issue might be. I'll try to catch both of you on the Gitter channel.

[manveru]

Since I got pretty far, i made a branch for my progress: https://github.com/manveru/nixpkgs/tree/add-rubinius

The results are pretty encouraging, with only a handful of test failures which I might work on when I get a bit of spare time again (though feel free to have a go at them, they might even turn out to be bugs in rbx ;)

https://gist.github.com/manveru/6f1dedccb9da2aa874cd3d5f04bd3ff3

[manveru]

Little note, never use bundix --magic, it's really not good for anything other than a very specialized use-case and I'll remove it in the next major version of Bundix.

[manveru]

after adding a HOME directory, i got the following backtrace: https://gist.github.com/manveru/fe0c870206c9a7e53057a1bb390d5ca2

[ghost]

@manveru I got the bundix --magic line from the Nixpkgs manual: https://nixos.org/nixpkgs/manual/#sec-language-ruby

[ghost]

My latest build log via the 'add home for rubinius tests' commit. This is a freshly cloned version of @manveru's nixpkgs repo.

[ghost]

It appears to be the same build error that I have been previously getting on NixOS Linux.

[brixen]

@Zacts @manveru thank you both for getting this started and mostly working. I'll use this Vagrant box to try to get a completely working build on Linux first, and then I'll take a look at macOS.

[brixen]

@manveru 4.5 builds on Linux now but has an issue building the C-exts, investigating.

[brixen]

@Zacts you need to update to 4.5 in the build.

@manveru where are the build artifacts? I'd like to be able to work on a partially completed build. How would I do that?

[brixen]

@manveru would it be possible to have the build succeed even if there are some spec failures? Would Nix accept the package in that case?

The issue is that some specs can be very sensitive to env issues, and if Rubinius builds and installs the pre-installed gems, it should be working fine.

Alternatively, we could use configure and rake directly right now instead of build.sh and avoid running the specs completely.

[brixen]

@manveru left you a comment on your add-rubinius branch. Can we merge with those changes? If not, what else can I help with? Thanks!

[brixen]

@jc00ke perhaps you may be able to help get this completed and Nix could be a way to address rubinius/rubinius#3819 since you could easily install Nix and then Rubinius if it were cached as a binary package, probably faster than RVM does whatever it does on Travis.

[brixen]

@manveru bumping this... anything I can do to help get your patch merged?

[alyssais]

My build of @manveru’s branch has been stuck for a couple of hours. There are lots of rbx -v mspec-ci :ci_files -d —background processes using 0.0% CPU. Guessing that’s not expected?

[alyssais]

@manveru where are the build artifacts? I'd like to be able to work on a partially completed build. How would I do that?

Pass -K to nix-build.

@manveru would it be possible to have the build succeed even if there are some spec failures? Would Nix accept the package in that case?

The issue is that some specs can be very sensitive to env issues, and if Rubinius builds and installs the pre-installed gems, it should be working fine.

Alternatively, we could use configure and rake directly right now instead of build.sh and avoid running the specs completely.

We could accept without specs being run, yes. Although it would be preferable not to.

[brixen]

@alyssais hi, thanks for your help on this!

If you see my comments here and here, the build should complete with those changes.

As for the specs running during package build, here are the issues:

1. we run the specs on Travis and those must pass before we tag a release;
2. there are numerous specs that depend on certain environment configurations (because Ruby is very hard to test for things that interact with the environment, like spawning new processes) and we won't be able to control that when a user installs on some arbitrary system;
3. the specs take at least a few minutes to run, and imposing that burden (not to mention use of resources like electricity) on every install does not seem prudent.

Of course, there will be issues found from time to time, and that's the unavoidable nature of software. Running the specs on install won't prevent that. Helping people get a version installed will broaden the number of people who can help with issues.

Since there's no PR actually associated with this ticket, would you like me to make a PR based on @manveru's patch and submit that?

[ghost]

@alyssais @brixen I'm willing to help with anything as well if needed. I hope to see this patch merged.

[ghost]

@alyssais Is there anything we need to do to merge this?

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[brixen]

@alyssais @manveru howdy! Is there anything I can do to help wrap this up? We'd love to have this package. 🙏

[stale]

I marked this as stale due to inactivity. → More info