Cannot build useful docker images on OS X (darwin)

[jml]

Issue description

I am trying to build docker images using nix on my Macbook Pro, running Darwin. When I try to do so, nix tries to build linux-pam on my Darwin laptop, rather than on the docker-machine VM that I normally use for Docker. This fails, and so I can't build docker images.

Normally, I use the docker-machine VM for building images. The docker command on OS X works transparently with that VM, provided certain environment variables are set. I would have thus expected Nix to use the VM for building images.

I tried to find documentation on how to build docker images on OS X via the docker-machine VM, but couldn't find anything.

Steps to reproduce

Using the redis-small.nix example on @lethalman's blog, repeated below:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;
dockerTools.buildImage {
  name = "redis";
  runAsRoot = ''
    #!${stdenv.shell}
    ${dockerTools.shadowSetup}
    groupadd -r redis
    useradd -r -g redis -d /data -M redis
    mkdir /data
    chown redis:redis /data
  '';

  config = {
    Cmd = [ "${goPackages.gosu.bin}/bin/gosu" "redis" "${redis}/bin/redis-server" ];
    ExposedPorts = {
      "6379/tcp" = {};
    };
    WorkingDir = "/data";
    Volumes = {
      "/data" = {};
    };
  };
}

And then running nix-build...

$ nix-build --show-trace ./redis-small.nix
error: while evaluating the attribute ‘buildCommand’ of the derivation ‘redis.tar.gz’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/trivial-builders.nix:10:14:
while evaluating the attribute ‘shadowSetup’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/docker/default.nix:57:3:
while evaluating the attribute ‘nativeBuildInputs’ of the derivation ‘shadow-4.2.1’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:18:3:
while evaluating ‘optional’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/lists.nix:186:20, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:25:17:
while evaluating ‘callPackageWith’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:93:35, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/top-level/all-packages.nix:11081:9:
while evaluating ‘makeOverridable’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:54:24, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:97:8:
while evaluating anonymous function at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/pam/default.nix:1:1, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:56:12:
while evaluating ‘mkDerivation’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/stdenv/generic/default.nix:95:5, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/pam/default.nix:3:1:
while evaluating ‘throwEvalHelp’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/stdenv/generic/default.nix:120:23, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/stdenv/generic/default.nix:169:21:
Package ‘linux-pam-1.2.1’ in ‘/nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/pam/default.nix:53’ is not supported on ‘x86_64-darwin’, refusing to evaluate.
a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build` or any other Nix command you can add
  { allowBroken = true; }
to ~/.nixpkgs/config.nix.

When I add allowBroken, I get the following:

$ nix-build --show-trace ./redis-small.nix
error: while evaluating the attribute ‘buildCommand’ of the derivation ‘redis.tar.gz’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/trivial-builders.nix:10:14:
while evaluating the attribute ‘shadowSetup’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/docker/default.nix:57:3:
while evaluating the attribute ‘preBuild’ of the derivation ‘shadow-4.2.1’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:18:3:
assertion failed at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:8:10

Technical details

• System: Darwin: 10.11.5
• Nix version: 1.11.2
• Nixpkgs version: 16.09pre83147.df89584
• Docker version:

$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      linux/amd64

[garbas]

you would need to perform the build from linux-x86-64. i guess you can use nixos/nix docker image and run nix-build from there.

[jml]

Thanks for replying. I currently use a similar work-around.

However, the bug still stands:

Normally, I use the docker-machine VM for building images. The docker command on OS X works transparently with that VM, provided certain environment variables are set. I would have thus expected Nix to use the VM for building images.

[garbas]

i would actually expect that dockerTools.buildImage to raise an error that you can not build on darwin platforms.

[garbas]

@lethalman should we add such assert statement?

[jml]

Docker have done a lot of work to make it easy to build & run docker images on OS X. It would be a shame to have to throw all that away to use Nix.

[domenkozar]

I don't see why it shouldn't work, docker is not even involved here.

[lucabrunox]

@garbas there's no reason for a specific assertion. If a package is not supported on darwin there's little I can do, and I don't want to add extra friendly magic error messages. It's quite clear there, linux-pam is not supported. Yet, it's perfectly possible to build images on OS X without involving PAM stuff, pretty limited but possible.

@jml The main difference between our dockerTools and docker itself is that with dockerTools we take advantage of Nix build artifacts outside of the container. That means linux-pam is supposed to be available on OS X, which isn't the case obviously. While Docker builds its stuff within the container.

[lucabrunox]

To be clear, might be fixable by using shadow without PAM on OS X, and that might work for most cases.

[jml]

OK, I think I understand now. dockerTools re-implements docker build, rather than using docker to build images. This means that it cannot take advantage of things like Docker Machine to build images.

Even if the PAM thing got sorted out, dockerTools would still not be useful on OS X, because it would produce darwin binaries that couldn't run inside a Linux container.

I still want to use Nix expressions to build Docker images while running on OS X. I still think that's a laudable goal.

Would it be possible to make something that provides a similar interface to dockerTools, but uses the docker executable itself to build the containers, passing along the relevant environment variables (DOCKER_HOST primarily)?

[mightybyte]

I would like to second @jml's desire to be able to use Nix expressions to build Docker images while running on OS X.

[garbas]

@jml @mightybyte i dont see building docker images from other docker images as a work around, but exactly what docker cli is doing.

[jml]

It's a significantly worse user experience than the Docker CLI, which does all of that for you automatically.

[copumpkin]

I wouldn't say dockerTools should use the native docker command, because it's fundamentally impure, but try this from @LnL7: https://github.com/LnL7/nix-docker#running-as-a-remote-builder

I'd like to make something like that easier to use for OSX users, but it should get you reasonably transparent cross-builds like that today.

[charles-dyfis-net]

Using a remote builder as suggested by @copumpkin above gets closer, but not to the point of successful builds.

After configuring an appropriate builder (either with @LnL7's tools or linuxkit-builder), we succeed in building x86_64-linux dependencies, but then in one of the late stages run a command on the MacOS host invoking an x86_64-linux version of bash.

nix-build -j 1 --system x86_64-linux -E 'with import <nixpkgs> {}; pkgs.dockerTools.buildImage { name = "nix-htop"; contents = pkgs.htop; config = { Cmd = [ "/bin/htop" ]; }; }'

...thus fails with:

building path(s) ‘/nix/store/gz4lrsjcmxbcmdfpmazwz0wqnb5pbw8k-nix-htop-config.json’
/nix/store/nkq0n2m4shlbdvdq0qijib5zyzgmn0vq-bash-4.4-p12/bin/bash: /nix/store/nkq0n2m4shlbdvdq0qijib5zyzgmn0vq-bash-4.4-p12/bin/bash: cannot execute binary file
builder for ‘/nix/store/487mmw8kql56q7h6iq4c7hfzh4k0gv50-nix-htop-config.json.drv’ failed with exit code 126

(...as also brought up on Unix & Linux StackExchange at https://unix.stackexchange.com/questions/470420/bash-cannot-execute-binary-file-during-nix-dockertools-cross-build-from-macos).

[matthewbauer]

Closing for now as we have https://github.com/nix-community/linuxkit-nix which is doing the equivalent of what Docker does.

[anka-213]

Has anyone resolved the

/nix/store/nkq0n2m4shlbdvdq0qijib5zyzgmn0vq-bash-4.4-p12/bin/bash: cannot execute binary file

issue that @charles-dyfis-net mentioned? I still can't build docker images, even with linuxkit-nix.

[anka-213]

This answer helped for me. Instead of using --system x86_64-linux, you can use --argstr system x86_64-linux with a nix expression like this

{ system ? "x86_64-linux", pkgs ? import <nixpkgs> { inherit system; } }:
...

For example

nix-build -j 1 --argstr system x86_64-linux -E '{ system ? "x86_64-linux", pkgs ? import <nixpkgs> { inherit system; } }: pkgs.dockerTools.buildImage { name = "nix-htop"; contents = pkgs.htop; config = { Cmd = [ "/bin/htop" ]; }; }'