Multiple layers docker images don't work

[nlewo]

Issue description

I created an Docker image with a nix content and ubuntu as a base layer. When I run this image, Docker only see the base layers.

It seems this is because the base image contains an image config.json file which is also imported in the created image. In this case, it seems Docker uses this file instead of the repositories generated file.

Steps to reproduce

Consider the file multiple-layers.nix:

{ pkgs ? import <nixpkgs> {} }:

rec {
  ubuntu = pkgs.dockerTools.pullImage {
    imageName = "ubuntu";
    imageTag = "14.04";
    sha256 = "01v514h449pmx8rggkpbrfgy3i8yy8dljrkjc5h0fdr870nsvwbg";
  };
  bash = pkgs.dockerTools.buildImage {
    name = "bash";
    contents = pkgs.bashInteractive;
  };
  bashOnUbuntu = pkgs.dockerTools.buildImage {
    name = "bashOnUbuntu";
    contents = pkgs.bashInteractive;
    fromImage = ubuntu;
  };
}

Then, build the image and load it

    docker load < ./result
    docker run bashonubuntu ls | grep nix

The last output is empty while it should show the nix folder.

Technical details

• System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)
• Nix version: (run nix-env --version)
• Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)
  97544a6
• Sandboxing enabled: (run grep build-use-sandbox /etc/nix/nix.conf)

[fpletz]

cc @lo1tuma

[nlewo]

Maybe this feature has been working with older images which did't have this configuration file (docker load works well without it). This file could be related to the registry v2, but I'm really not sure about this.

[lo1tuma]

So far I’ve only used base images that were also built with nixpkgs dockerTools which works fine. I’ve never used it in combination with pullImage.

[nlewo]

@lo1tuma I think it works well because nixpkgs dockerTools doesn't create this image configuration file.

[Fuuzetsu]

This broke docker builds for us

these derivations will be built:
  …
building path(s) ‘/nix/store/….tar.gz’
Unpacking base image...
rm: cannot remove 'image/f3f1075ae114c0d3fa3716c89529ce25beefe34d9cfca8fd67401541e93f4790.json': Permission denied
rm: cannot remove 'image/manifest.json': Permission denied

The config is trivial, we use fromImage = busybox-docker; and basically just serve static web content with caddy, nothing fancy.

[nlewo]

@Fuuzetsu I cannot reproduce your issue. I successfully built:

  bashOnBusybox = pkgs.dockerTools.buildImage {
    name = "bashOnBusybox";
    contents = pkgs.bashInteractive;
    fromImage = pkgs.dockerTools.pullImage {
      imageName = "busybox";
      imageTag = "latest";
      sha256 = "0dg6wlsdh4f76rin9j0shbzmlfmviv0a2m2yb06843m6y0fxni7k";
    };
  };

Can you provide me the base image you are using? Or could you upgrade it ? :/

[Fuuzetsu]

@nlewo I did a little more digging and the problem seems to be that images can't be composed; that is both your image above and the very basic one we use work but they break once they get used in fromImage; see below on how to replicate. Here's the test file

{ bootstrap_pkgs ? import <nixpkgs> {}
, pkgs ? import (bootstrap_pkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "93d364f4f50ce54cd216361e4a4dd683c1933a10";
    sha256 = "058q8b105acrhy0v5lbd1w83zw2jwh06c9hzxlxw7zkx93sxk3nr";
  }) {}
}:

let mkImage = name: img: pkgs.dockerTools.buildImage {
      inherit name;
      fromImage = img;
      contents = [];
    };
    your_image = pkgs.dockerTools.buildImage {
      name = "bashOnBusybox";
      contents = pkgs.bashInteractive;
      fromImage = pkgs.dockerTools.pullImage {
        imageName = "busybox";
        imageTag = "latest";
        sha256 = "0dg6wlsdh4f76rin9j0shbzmlfmviv0a2m2yb06843m6y0fxni7k";
      };
    };
    our_image = pkgs.dockerTools.buildImage {
      name = "busybox";
      contents = pkgs.busybox;
    };
in {
  our_image = mkImage "our_image" our_image;
  your_image = mkImage "your_image" your_image;
}

and the output

[shana@lenalee:/tmp]$ nix-build busybox.nix --no-out-link -A your_image
building path(s) ‘/nix/store/k5gxjy66y1h9b2rkqvr7gikdfdrqkpzn-nixpkgs-93d364f4f50ce54cd216361e4a4dd683c1933a10-src’

trying https://github.com/NixOS/nixpkgs/archive/93d364f4f50ce54cd216361e4a4dd683c1933a10.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   155    0   155    0     0    412      0 --:--:-- --:--:-- --:--:--   413
100 11.6M  100 11.6M    0     0  4859k      0  0:00:02  0:00:02 --:--:-- 6251k
unpacking source archive /tmp/nix-build-nixpkgs-93d364f4f50ce54cd216361e4a4dd683c1933a10-src.drv-0/93d364f4f50ce54cd216361e4a4dd683c1933a10.tar.gz
these derivations will be built:
  /nix/store/3qbx902q6ysrr4jxjavj2xirzlxkpfxh-your_image-config.json.drv
  /nix/store/wj420dvhwp604mcvxzx9hsn9y8klywga-docker-layer-your_image.drv
  /nix/store/sglsiiqp6zckfm645nrh7mq39p8hgkc9-runtime-deps.drv
  /nix/store/znx482nkaf4pvz819bq5pms77gwxq65w-docker-image-your_image.tar.gz.drv
building path(s) ‘/nix/store/rbkc07k6lmhlh37w4iwszcq2wlyd8nb5-your_image-config.json’
building path(s) ‘/nix/store/bqx33zgsbg93xcwndm6azhs2dr2vsjac-docker-layer-your_image’
No contents to add to layer.
Packing layer...
Computing layer checksum...
Finished building layer 'your_image'
building path(s) ‘/nix/store/c60p2i6ipr6sk191n9csf0wsf5fjh6da-runtime-deps’
building path(s) ‘/nix/store/6miklnlc5lzwbz5pqkjssraf6cfiwrzl-docker-image-your_image.tar.gz’
Unpacking base image...
rm: cannot remove 'image/776a182be0245b0a5010c661def24741d485c301d179f623a8273718e50415c6.json': Permission denied
rm: cannot remove 'image/manifest.json': Permission denied
builder for ‘/nix/store/znx482nkaf4pvz819bq5pms77gwxq65w-docker-image-your_image.tar.gz.drv’ failed with exit code 1
error: build of ‘/nix/store/znx482nkaf4pvz819bq5pms77gwxq65w-docker-image-your_image.tar.gz.drv’ failed

[shana@lenalee:/tmp]$ nix-build busybox.nix --no-out-link -A our_image
these derivations will be built:
  /nix/store/r69hr7nl0wgcn1xcbdlbc4a3pijz8328-our_image-config.json.drv
  /nix/store/yw7hnsbhllmd827n6b4xyadv01rbs1cs-docker-layer-our_image.drv
  /nix/store/6qxig836daz3wyq142xvzb6q64r59682-runtime-deps.drv
  /nix/store/h1kxyjq6019prkyhaqkqksiqbs0d8csb-docker-image-our_image.tar.gz.drv
building path(s) ‘/nix/store/iljglmrgf3z74f7ixw3421m6p4g5lwh3-our_image-config.json’
building path(s) ‘/nix/store/lb7gvsqwm9zfnrx6n3zlg78mq6wb51z5-docker-layer-our_image’
No contents to add to layer.
Packing layer...
Computing layer checksum...
Finished building layer 'our_image'
building path(s) ‘/nix/store/230l2h9n2sbpcpn5xw3r50im2h7qgk4r-runtime-deps’
building path(s) ‘/nix/store/gx28v6wfjaxkyncrvq0cx3jpf84vsgbj-docker-image-our_image.tar.gz’
Unpacking base image...
rm: cannot remove 'image/f3f1075ae114c0d3fa3716c89529ce25beefe34d9cfca8fd67401541e93f4790.json': Permission denied
rm: cannot remove 'image/manifest.json': Permission denied
builder for ‘/nix/store/h1kxyjq6019prkyhaqkqksiqbs0d8csb-docker-image-our_image.tar.gz.drv’ failed with exit code 1
error: build of ‘/nix/store/h1kxyjq6019prkyhaqkqksiqbs0d8csb-docker-image-our_image.tar.gz.drv’ failed

[nlewo]

@Fuuzetsu Thanks for your example. I also reproduce with nix-build ./ -A dockerTools.examples.redis and I submitted a PR to fix this.