Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot build useful docker images on OS X (darwin) #16696

Closed
jml opened this issue Jul 4, 2016 · 15 comments
Closed

Cannot build useful docker images on OS X (darwin) #16696

jml opened this issue Jul 4, 2016 · 15 comments

Comments

@jml
Copy link
Contributor

@jml jml commented Jul 4, 2016

Issue description

I am trying to build docker images using nix on my Macbook Pro, running Darwin. When I try to do so, nix tries to build linux-pam on my Darwin laptop, rather than on the docker-machine VM that I normally use for Docker. This fails, and so I can't build docker images.

Normally, I use the docker-machine VM for building images. The docker command on OS X works transparently with that VM, provided certain environment variables are set. I would have thus expected Nix to use the VM for building images.

I tried to find documentation on how to build docker images on OS X via the docker-machine VM, but couldn't find anything.

Steps to reproduce

Using the redis-small.nix example on @lethalman's blog, repeated below:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;
dockerTools.buildImage {
  name = "redis";
  runAsRoot = ''
    #!${stdenv.shell}
    ${dockerTools.shadowSetup}
    groupadd -r redis
    useradd -r -g redis -d /data -M redis
    mkdir /data
    chown redis:redis /data
  '';

  config = {
    Cmd = [ "${goPackages.gosu.bin}/bin/gosu" "redis" "${redis}/bin/redis-server" ];
    ExposedPorts = {
      "6379/tcp" = {};
    };
    WorkingDir = "/data";
    Volumes = {
      "/data" = {};
    };
  };
}

And then running nix-build...

$ nix-build --show-trace ./redis-small.nix
error: while evaluating the attribute ‘buildCommand’ of the derivation ‘redis.tar.gz’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/trivial-builders.nix:10:14:
while evaluating the attribute ‘shadowSetup’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/docker/default.nix:57:3:
while evaluating the attribute ‘nativeBuildInputs’ of the derivation ‘shadow-4.2.1’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:18:3:
while evaluating ‘optional’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/lists.nix:186:20, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:25:17:
while evaluating ‘callPackageWith’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:93:35, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/top-level/all-packages.nix:11081:9:
while evaluating ‘makeOverridable’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:54:24, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:97:8:
while evaluating anonymous function at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/pam/default.nix:1:1, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/lib/customisation.nix:56:12:
while evaluating ‘mkDerivation’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/stdenv/generic/default.nix:95:5, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/pam/default.nix:3:1:
while evaluating ‘throwEvalHelp’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/stdenv/generic/default.nix:120:23, called from /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/stdenv/generic/default.nix:169:21:
Package ‘linux-pam-1.2.1’ in ‘/nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/pam/default.nix:53’ is not supported on ‘x86_64-darwin’, refusing to evaluate.
a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build` or any other Nix command you can add
  { allowBroken = true; }
to ~/.nixpkgs/config.nix.

When I add allowBroken, I get the following:

$ nix-build --show-trace ./redis-small.nix
error: while evaluating the attribute ‘buildCommand’ of the derivation ‘redis.tar.gz’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/trivial-builders.nix:10:14:
while evaluating the attribute ‘shadowSetup’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/build-support/docker/default.nix:57:3:
while evaluating the attribute ‘preBuild’ of the derivation ‘shadow-4.2.1’ at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:18:3:
assertion failed at /nix/store/phqb57liyaqm9jxlawk47y3d97whq2bv-nixpkgs-16.09pre83147.df89584/nixpkgs/pkgs/os-specific/linux/shadow/default.nix:8:10

Technical details

  • System: Darwin: 10.11.5
  • Nix version: 1.11.2
  • Nixpkgs version: 16.09pre83147.df89584
  • Docker version:
$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      linux/amd64
@garbas

This comment has been minimized.

Copy link
Contributor

@garbas garbas commented Aug 30, 2016

you would need to perform the build from linux-x86-64. i guess you can use nixos/nix docker image and run nix-build from there.

@jml

This comment has been minimized.

Copy link
Contributor Author

@jml jml commented Aug 30, 2016

Thanks for replying. I currently use a similar work-around.

However, the bug still stands:

Normally, I use the docker-machine VM for building images. The docker command on OS X works transparently with that VM, provided certain environment variables are set. I would have thus expected Nix to use the VM for building images.

@garbas

This comment has been minimized.

Copy link
Contributor

@garbas garbas commented Aug 30, 2016

i would actually expect that dockerTools.buildImage to raise an error that you can not build on darwin platforms.

@garbas

This comment has been minimized.

Copy link
Contributor

@garbas garbas commented Aug 30, 2016

@lethalman should we add such assert statement?

@jml

This comment has been minimized.

Copy link
Contributor Author

@jml jml commented Aug 30, 2016

Docker have done a lot of work to make it easy to build & run docker images on OS X. It would be a shame to have to throw all that away to use Nix.

@domenkozar

This comment has been minimized.

Copy link
Member

@domenkozar domenkozar commented Aug 30, 2016

I don't see why it shouldn't work, docker is not even involved here.

@lethalman

This comment has been minimized.

Copy link
Contributor

@lethalman lethalman commented Sep 5, 2016

@garbas there's no reason for a specific assertion. If a package is not supported on darwin there's little I can do, and I don't want to add extra friendly magic error messages. It's quite clear there, linux-pam is not supported. Yet, it's perfectly possible to build images on OS X without involving PAM stuff, pretty limited but possible.

@jml The main difference between our dockerTools and docker itself is that with dockerTools we take advantage of Nix build artifacts outside of the container. That means linux-pam is supposed to be available on OS X, which isn't the case obviously. While Docker builds its stuff within the container.

@lethalman

This comment has been minimized.

Copy link
Contributor

@lethalman lethalman commented Sep 5, 2016

To be clear, might be fixable by using shadow without PAM on OS X, and that might work for most cases.

@jml

This comment has been minimized.

Copy link
Contributor Author

@jml jml commented Dec 6, 2016

OK, I think I understand now. dockerTools re-implements docker build, rather than using docker to build images. This means that it cannot take advantage of things like Docker Machine to build images.

Even if the PAM thing got sorted out, dockerTools would still not be useful on OS X, because it would produce darwin binaries that couldn't run inside a Linux container.

I still want to use Nix expressions to build Docker images while running on OS X. I still think that's a laudable goal.

Would it be possible to make something that provides a similar interface to dockerTools, but uses the docker executable itself to build the containers, passing along the relevant environment variables (DOCKER_HOST primarily)?

@mightybyte

This comment has been minimized.

Copy link
Contributor

@mightybyte mightybyte commented Dec 28, 2016

I would like to second @jml's desire to be able to use Nix expressions to build Docker images while running on OS X.

@garbas

This comment has been minimized.

Copy link
Contributor

@garbas garbas commented Dec 29, 2016

@jml @mightybyte i dont see building docker images from other docker images as a work around, but exactly what docker cli is doing.

@jml

This comment has been minimized.

Copy link
Contributor Author

@jml jml commented Dec 30, 2016

It's a significantly worse user experience than the Docker CLI, which does all of that for you automatically.

@copumpkin

This comment has been minimized.

Copy link
Member

@copumpkin copumpkin commented Mar 14, 2017

I wouldn't say dockerTools should use the native docker command, because it's fundamentally impure, but try this from @LnL7: https://github.com/LnL7/nix-docker#running-as-a-remote-builder

I'd like to make something like that easier to use for OSX users, but it should get you reasonably transparent cross-builds like that today.

@copumpkin copumpkin added this to Improve functionality in Amazing images Apr 18, 2017
@charles-dyfis-net

This comment has been minimized.

Copy link
Contributor

@charles-dyfis-net charles-dyfis-net commented Sep 21, 2018

Using a remote builder as suggested by @copumpkin above gets closer, but not to the point of successful builds.

After configuring an appropriate builder (either with @LnL7's tools or linuxkit-builder), we succeed in building x86_64-linux dependencies, but then in one of the late stages run a command on the MacOS host invoking an x86_64-linux version of bash.

nix-build -j 1 --system x86_64-linux -E 'with import <nixpkgs> {}; pkgs.dockerTools.buildImage { name = "nix-htop"; contents = pkgs.htop; config = { Cmd = [ "/bin/htop" ]; }; }'

...thus fails with:

building path(s) ‘/nix/store/gz4lrsjcmxbcmdfpmazwz0wqnb5pbw8k-nix-htop-config.json’
/nix/store/nkq0n2m4shlbdvdq0qijib5zyzgmn0vq-bash-4.4-p12/bin/bash: /nix/store/nkq0n2m4shlbdvdq0qijib5zyzgmn0vq-bash-4.4-p12/bin/bash: cannot execute binary file
builder for ‘/nix/store/487mmw8kql56q7h6iq4c7hfzh4k0gv50-nix-htop-config.json.drv’ failed with exit code 126

(...as also brought up on Unix & Linux StackExchange at https://unix.stackexchange.com/questions/470420/bash-cannot-execute-binary-file-during-nix-dockertools-cross-build-from-macos).

@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Oct 31, 2018

Closing for now as we have https://github.com/nix-community/linuxkit-nix which is doing the equivalent of what Docker does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Amazing images
Improve functionality
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.