Make nix available in the container

[wereHamster]

This makes nix available in the build image. It will allow users to use nix-shell or nix-build to build the websites.

I tested the image by building it and then executing nix-shell inside it:

docker run -ti <image> \
  nix-shell -p nodejs-10_x \
    --command "node -e 'console.log(process.version)'"

(here nix-shell is instructed to make node.js v10 available and then run "node -e 'console.log(process.version)'").

[brycekahle]

@wereHamster you can use node.js 10 by setting an environment variable of NODE_VERSION=10. See https://www.netlify.com/docs/continuous-deployment/#set-node-ruby-or-python-version for more information.

[wereHamster]

Don't pay too close attention to the fact that I used node in the test, nix actually provides thousands of packages. To test the build-image I could also have used docker run -ti <image> nix --version.

[grossbart]

Having Nix available in the build image would be wonderful! We've been using Nix very successfully to pin down (system or exotic) dependencies for currently ~40 projects, making it possible for team members to easily jump into a project, say a 5 year old Ruby app, without having to fiddle with the environment; with Nix the environment is precisely reproducible, which removes so many problems.

With Nix in the build image we could use this same configuration on our machines and the Netlify machines instead of trying to manually keep the configurations in sync.

Here's a more involved example that pins dependencies for a project that uses GDAL, dos2unix, Node with Yarn, and Ruby with Gems:

let
  unpinned = import <nixpkgs> {};
  pkgs = import (unpinned.fetchFromGitHub
    { owner = "NixOS";
      repo = "nixpkgs";
      rev = "e90412debb744dfc481cebf4243e48aef512c726";
      sha256 = "1bclz2xvk3w942d3kb5lzhzahfwmna8npcasq7n1m766h6yz2yj8";
    }) {};
  name = "netlify-project";

  ruby = pkgs.ruby_2_2;
  nodejs = pkgs.nodejs-8_x;
  yarn = pkgs.yarn.override { inherit nodejs; };

in pkgs.stdenv.mkDerivation rec {
  inherit name;
  gems = pkgs.bundlerEnv {
    inherit name ruby;
    gemdir = ./.;
  };

  buildInputs = [
    gems
    ruby
    nodejs
    yarn
    pkgs.gdal
    pkgs.dos2unix
  ];

  shellHook = ''
    yarn
  '';
}

What's currently unclear for me is what happens if we pin down a specific Node version and Netlify detecs a Node/Ruby/whatever app and installs dependencies on its own – there could be a conflict.

[wereHamster]

Dependencies that are specified in nix files take precedence over whatever is installed by the system. Furthermore, nix-shell has the --pure flag to explicitly ignore anything that is installed globally. I can't find that same flag in the nix-build documentation, but I'd expect it to behave like nix-shell --pure by default.

[brycekahle]

A major downside of this, is having to install those packages for every build. Our current setup goes through a lot of effort to prevent that.

[grossbart]

I'm not sure I understand the Netlify architecture well enough, but I would assume that the Docker image is built only once per project? If that is the case then Nix will take care of your concern: it builds all dependencies only once and stores them in the Nix store, each subsequent start of a Nix shell will then use the already built dependencies, which takes no time at all.

[brycekahle]

@grossbart That isn't how our CI system works. We have a global base docker image, then we save/restore caches of certain directories from blob storage during each build.

[wereHamster]

Nix makes it possible to redirect all artefacts (dependencies and intermediate build products) to a separate directory (see my last commit). So all good… except…

…there doesn't seem to be a way to skip the pre-processing and dependency installation that the build script does on its own. I tested it with one of my projects where nix reliably works, but it fails inside the netlify build container because the build script tries to (automagically) install dependencies using the wrong node/yarn versions. I was able to work around this by setting NODE_VERSION and YARN_VERSION, but that kindof defeats the purpose of using nix (one of the reasons to use nix was so that we don't have to specify the dependencies in multiple places).

[brycekahle]

@wereHamster We are tracking the ability to skip certain steps in #141

[grahamc]

A minimal approach to implementing this would be just creating the /nix directory as root. This would allow Nix to be installed as the user, in their own netlify.toml file. Would that be acceptable, @brycekahle?

[bcomnes]

Supporting a local nix and homebrew would be very cool indeed. I'm taking over for bryce on the buildbot work. So, open to it! But I don't have any kind of timeline yet.

[bcomnes]

I changed the base to xenial, since if we do this, we'll do it on that branch.

[thalesmg]

I was about to request Nix support too. I'm building some simple pages using it, and I'm currently only able to deploy that using Travis' dpl tool. But I miss using Netlify's Pull Request previews and integrations...

[domenkozar]

Is there a sensible path forward here?

[wereHamster]

Someone else can pick this up. I no longer plan working on this.