Add crosstool-ng package

[woehr]

This pull request adds a package for crosstool-ng that can be run standalone. The previous pull request also included several toolchain packages that depended on crosstool-ng but they will be submitted as a separate pull request if there is interest in the toolchains (they were the arm v5, v6, and v7 toolchains from Arch Linux Arm).

[bjornfor]

Cool. I like crosstool-NG. I don't think anyone has packaged it before, in any package system :-)

It looks to me like ct-ng isn't fully wrapped. Is that intended? You don't want it to be installable/useable directly with nix-env -i crosstool-ng?

IMHO, it'd be nice to split this PR in two commits. One that adds crosstool-ng and another that adds the toolchains. Well, at least if ct-ng is supposed to be installable by itself. If not, I understand the one-commit approach.

I originally thought of crosstool-NG as something quite foreign (or incompatible) with nix, so to see that you build toolchains with it, in nix, was a bit of a surprise. Now that I've thought about it, it makes more sense: If you have gotten so far as to create a good toolchain, you'd probably want it to be documented and repeatable (i.e. build it with nix).

I'm less sure about the usefulness of the predefined crosstool-NG toolchains. It seems to me that there are so many ways to build the toolchains (configuration variations) that the current set is rather arbitrary...

I don't really have anything to add regarding your questions. Possibly someone else can comment on whether to add those utility functions to lib.

[woehr]

On 26 March 2015 at 13:43, Bjørn Forsman notifications@github.com wrote:

Cool. I like crosstool-NG. I don't think anyone has packaged it before, in
any package system :-)

It looks to me like ct-ng isn't fully wrapped. Is that intended? You don't
want it to be installable/useable directly with nix-env -i crosstool-ng?

I had intended to allow it to be usable with nix-env. It can be installed
although I've now realized I don't think I ever built a toolchain with it
in that manner (which I'll try doing tonight). What do you mean when you
say it isn't fully wrapped?

IMHO, it'd be nice to split this PR in two commits. One that adds
crosstool-ng and another that adds the toolchains. Well, at least if ct-ng
is supposed to be installable by itself. If not, I understand the
one-commit approach.

I'm more than happy to split the PR.

I originally thought of crosstool-NG as something quite foreign (or
incompatible) with nix, so to see that you build toolchains with it, in
nix, was a bit of a surprise. Now that I've thought about it, it makes more
sense: If you have gotten so far as to create a good toolchain, you'd
probably want it to be documented and repeatable (i.e. build it with nix).

I'm less sure about the usefulness of the predefined crosstool-NG
toolchains. It seems to me that there are so many ways to build the
toolchains (configuration variations) that the current set is rather
arbitrary...

It is rather arbitrary. For myself I just wanted an armv7 toolchain but
could see the others being useful. For the three included, I chose the Arch
Linux Arm configurations because (I believe) they're fairly heavily used
and very likely to be better than anything I could come up with myself.

I found them at http://archlinuxarm.org/developers/distcc-cross-compiling

[bjornfor]

What do you mean when you say it isn't fully wrapped?

That ct-ng will not see all its needed dependencies, if installed with nix-env.

[woehr]

On 26 March 2015 at 14:29, Bjørn Forsman notifications@github.com wrote:

What do you mean when you say it isn't fully wrapped?

That ct-ng will not see all its needed dependencies, if installed with
nix-env.

Yeah ... that should have been obvious. I'll fix it up.

[woehr]

I've updated this pull request to only include the crosstool-ng package (and fixed it so that it is wrapped sufficiently to be run stand-alone). However, depending on what toolchain is being built, additional dependencies may need to be installed into the environment (e.g., perl to install linux kernel headers). Furthermore, builds may not be successful because sources aren't patched to work on nix (e.g., ct-ng downloads the Linux tarball which contains a Makefile that tries to invoke /bin/pwd).

There are two additional components that I can submit separate from the stand-alone ct-ng package (if desired):

1. A function that creates a cross-compiled toolchain derivation from a ct-ng config file. This function takes care of modifying ct-ng options to successfully build and install on nix (hopefully in a generic enough manner to work with any configuration). It also takes care of patching sources as necessary to build in a nix environment and only depends on files in the nix store.
2. Packages for arm v5, v6, and v7 toolchains based on the Arch Linux Arm ct-ng configurations (that use the above mentioned function to build successfully on nix).

[bjornfor]

Oh, I didn't mean for you to split the PR, just the commits (now where are the toolchains?). But ok.

Hm, I didn't think of that problem, that toolchain building will try to use programs from /bin. Even if we make crosstool-ng work properly stand-alone, possibly by using proot, what will happen with the generated toolchains? I fear that they will refer to /nix/store/* paths that might be garbage collected at any time, because now Nix doesn't know about them.

I think, either we make it work pretty well out-of-the box, or not expose it as a package at all. IOW, just have the ct-ng.mkToolchain function. Ugh, but then again, how to create the configs? You'd need to poke around in menuconfig for that...

About the extra deps, are there any more in addition to perl? What about just depending on perl?

[woehr]

Oh, I didn't mean for you to split the PR, just the commits (now where are
the toolchains?). But ok.

Oops. My misunderstanding.

Hm, I didn't think of that problem, that toolchain building will try to
use programs from /bin. Even if we make crosstool-ng work properly
stand-alone, possibly by using proot, what will happen with the generated
toolchains? I fear that they will refer to /nix/store/* paths that might be
garbage collected at any time, because now Nix doesn't know about them.

I think, either we make it work pretty well out-of-the box, or not expose
it as a package at all. IOW, just have the ct-ng.mkToolchain function. Ugh,
but then again, how to create the configs? You'd need to poke around in
menuconfig for that...

About the extra deps, are there any more in addition to perl? What about
just depending on perl?

I haven't seen any other dependencies. So just depending on them might not
be too bad.

[woehr]

What if we patched crosstool-ng to do additional patching that removes the /bin dependencies? That way it should work standalone, but it will also simplify the mkCrossToolchain function since it won't have to extract and patch everything itself.

[bjornfor]

Yes that is an option. Either make ct-ng patch its sources after download, or use proot (http://proot.me/) to patch paths at runtime.

[bjornfor]

It is generally preferred to take each dependency as an argument, instead of the whole pkgs package set.

[woehr]

I've nearly got ct-ng working standalone but am running into a problem. One of the components (cross-gdb) of the toolchain sample I'm trying to build is failing during its configure because it can't the termcap library. It seems this should be provided by ncurses, but it isn't (except the header in its include). Any ideas?

Once I have ct-ng working standalone I'll push it along with the function for creating derivations from configs and the arm cross toolchains (as separate commits this time 😄 ).

[bjornfor]

@woehr: No ideas, sorry. Interesting, grepping for "termcap" in nixpkgs results in many like these:

sed -i -e "s|-ltermcap|-lncurses|" configure

In fact, all hits are like that. Maybe try that on ct-ng too?

[bjornfor]

termcap is apparently obsolete, see first line of "man termcap". And apparently ncurses uses terminfo instead of termcap.

http://en.wikipedia.org/wiki/Ncurses#Terminal_database

[woehr]

I grep'ed -ltermcap and it isn't being used in the configure script that is failing. I'm fairly certain now that this actually an issue with finding ncurses. A simple gdb derivation with ncurses in its buildInputs avoids the error, so it seems just adding ncurses to PATH is insufficient. I'm trying to figure out what being in buildInputs does differently than just being added to PATH. Any ideas? (PS, thanks for all the help so far @bjornfor).

[bjornfor]

buildInputs typically does more than just adding packages to PATH. See https://nixos.org/nixpkgs/manual/#ssec-stdenv-attributes.

I guess ct-ng needs to see pkg-config and $PKG_CONFIG_PATH at runtime, to find ncurses compile flags.

[woehr]

I think ct-ng is now fairly usable stand-alone. It has successfully built all the sample configurations I've tried (which is not to say it'll build any configuration). Two minor annoyances remain:

1. Setting up stdenv/setup causes all make targets to be printed which clutters up the output (particularily when ct-ng list-samples is run).
2. Installing multiple of the same type of toolchain fails because they have items with the same name. For example, ct-ng names the binaries of both hard float Arch Linux Arm toolchains (6l and 7l) the same (arm-unknown-linux-gnueabihf-) and then symlinks the "specialized name" to the "generic name".

I've also committed the function that creates derivations from ct-ng config files and the Arch Arm toolchains.

[bjornfor]

The ct-ng wrapper ends up with hardcoded check against "/tmp/nix-build-crosstool-ng-1.20.0.drv-0". That doesn't look right.

I wonder if the wrapper can be simplified, and the "source $stdenv/setup" line be removed, by simply setting PKG_CONFIG_PATH=$PKG_CONFIG_PATH. $PKG_CONFIG_PATH would be expanded at package build time, so all build inputs would be available for pkg-config and ct-ng at runtime. You'll need to add pkgconfig as buildInput of course.

I'm actually surprised that this worked without pkgconfig...! "make menuconfig" requires pkg-config to find headers/libs, unless you have stuff installed in /usr/{lib,include}.

[woehr]

Oops. $NIX_BUILD_TOP was suppose to be escaped.

Including pkgconfig and setting PKG_CONFIG_PATH from the build environment seems to work. I'm building a few toolchains now to make sure.

PATH also needs to be set since setup would do that too. There are two options here, either set PATH from the build environment to make it explicit, or to a user's existing PATH. Personally setting PATH explicitly seems better so a user's environment doesn't affect ct-ng. The downside is then they'll have to override ct-ng (or submit a fix) if they try to build a configuration that has dependencies we haven't accounted for.

[bjornfor]

I think prefixing PATH sounds best. It ensures that needed deps are found and user env can supplement (append to) PATH if needed.

[woehr]

Hmmm. Something else is going on in setup. ncurses/lib/pkgconfig is in PKG_CONFIG_PATH but without running setup I'm back to it not finding termcap (because ncurses can't be found).

[woehr]

I'm not convinced that prefixing PATH is best. It's a subjective point, but running ct-ng such that the user environment can't affect ct-ng builds seems beneficial to me. Sure, a user would have to override the derivation if it's missing something, but I think we've already accounted for most external dependencies ct-ng needs such that in the common case an override won't be necessary.

[bjornfor]

That's weird. I'd expect ncurses to be found if you have pkg-config in path and $PKG_CONFIG_PATH includes $ncurses/lib/pkgconfig.

[woehr]

I've fixed the un-escaped NIX_BUILD_TOP bug and the test condition which happened to be wrong as well. It is functional, but it's using the source stdenv method for setting things up to run ct-ng standalone. Unfortunately I don't have any more time to put into this (at the present moment) to investigate why it doesn't work with the pkg-config stuff setup.

[woehr]

This pull request needs to be updated to work again. I'd like to do so one day, but am unable to at the moment, so I will close it for the time being.