Support buildkit

[massimiliano-mantione]

I am using minikube 1.0.0 with kubernetes v1.14.0 (on Fedora Linux, xen2 driver).

I am using a docker client connected to the minikube docker, and issueing docker build commands, but I really need to use buildkit.

If I set "DOCKER_BUILDKIT=1" the docker client complains because the daemon does not support buildkit.
I tried all the different cointainer engines supported by minikube, but the result is always the same.

Then I tried running docker in experimental mode inside the minikube vm:

• ssh into the vm
• created the docker config file with the experimental=true option
• restarted the docker service with systemctl

Then the error I got on a docker build was "pivot_root invalid argument".
I tried setting DOCKER_RAMDISK to true or false, but nothing changed.

It might be that docker 18.09 fixes this, or it might be a tiny core linux issue, but understanding that docker engine subtlety is beyond me.

What I know is that the same docker build succeeds on my workstation (Fedora 29 with docker 18.09.3).

Given clear instructions I could try building minikube with docker 18.09 and test it with that.
IMHO supporting buildkit in local dev environments should be desirable...

[afbjorklund]

This looks to be a bug with the new "builder-next" (buildkit) builder in Docker upstream.

It is supposed to pass the --no-pivot parameter to runc, but doesn't do so.
That would have the same effect as the NoPivotRoot used by libcontainerd:

https://github.com/moby/moby/blob/7e03619c1c71eb7774b1cf53318af565df0952e7/libcontainerd/remote/client.go#L205

moby/moby@8a4225c

The root cause for this is that minikube runs on rootfs, which doesn't allow pivot_root(2)

---

Minikube doesn't run tiny core linux anymore, which means the Boot2Docker fix is not here:

$ docker-machine ssh
   ( '>')
  /) TC (\   Core is distributed with ABSOLUTELY NO WARRANTY.
 (/-_--_-\)           www.tinycorelinux.net

docker@default:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   890.4M    229.6M    660.8M  26% /
tmpfs                   494.7M         0    494.7M   0% /dev/shm
/dev/sda1                17.8G    247.9M     16.7G   1% /mnt/sda1
...

tmpfs on / type tmpfs (rw,relatime,size=911788k)

$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ df -h / /dev/shm /mnt/sda1
Filesystem      Size  Used Avail Use% Mounted on
rootfs             0     0     0    - /
tmpfs           996M     0  996M   0% /dev/shm
/dev/sda1        17G  1.4G   15G   9% /mnt/sda1

rootfs on / type rootfs (rw)

They have moved away from rootfs, which is something that minikube.iso also should do...

See #3512

[afbjorklund]

Should have mention that the end result of all this, is that BuildKit works fine in Boot2Docker 18.09.5:

docker@default:~$ DOCKER_BUILDKIT=1 docker build .
[+] Building 0.2s (6/6) FINISHED                                                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                                                0.0s
 => => transferring dockerfile: 64B                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                   0.0s
 => [1/2] FROM docker.io/library/busybox                                                                                                                            0.0s
 => => resolve docker.io/library/busybox:latest                                                                                                                     0.0s
 => [2/2] RUN true                                                                                                                                                  0.2s
 => exporting to image                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                             0.0s
 => => writing image sha256:44f8812a320e821b7ad6036e4a75b7741d12139ab0edc62c4bff211b4445f810                                                                        0.0s

But probably we need to do more in the minikube.iso, than just upgrading Docker from 18.06 to 18.09

[tstromberg]

Related to #3512 and #3519

[afbjorklund]

Depends on #4180

[AkihiroSuda]

PR: moby/buildkit#980

[afbjorklund]

@AkihiroSuda : thanks, that should do the trick until we get #3512 sorted out

[AkihiroSuda]

moby PR: moby/moby#39202

[antonmarin]

Same with 1.1.0

[afbjorklund]

We did not have time to patch and rebuild docker from source or to switch away from using rootfs for this minikube release.
So there is no buildkit support (yet), even though Docker was upgraded to 18.09.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[demisx]

When do you expect minikube to support BuildKit?

[afbjorklund]

We are aiming to move away from rootfs for v1.4.0

/remove-lifecycle stale

[afbjorklund]

Confirmed that building images with BuildKit works OK, with the new ISO:

$ DOCKER_BUILDKIT=1 docker build .
[+] Building 3.1s (6/6) FINISHED                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                     0.0s
 => [internal] load build definition from Dockerfile                                                                                                                0.0s
 => => transferring dockerfile: 65B                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                   1.9s
 => [1/2] FROM docker.io/library/busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70                                                    0.7s
 => => resolve docker.io/library/busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70                                                    0.0s
 => => sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70 1.86kB / 1.86kB                                                                      0.0s
 => => sha256:895ab622e92e18d6b461d671081757af7dbaa3b00e3e28e12505af7817f73649 527B / 527B                                                                          0.0s
 => => sha256:db8ee88ad75f6bdc74663f4992a185e2722fa29573abcc1a19186cc5ec09dceb 1.50kB / 1.50kB                                                                      0.0s
 => => sha256:ee153a04d6837058642958836062f20badf39f558be3e6c7c7773ef7d8301d90 762.73kB / 762.73kB                                                                  0.4s
 => => extracting sha256:ee153a04d6837058642958836062f20badf39f558be3e6c7c7773ef7d8301d90                                                                           0.1s
 => [2/2] RUN true                                                                                                                                                  0.4s
 => exporting to image                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                             0.0s
 => => writing image sha256:d5cba3a06196c715b4cb29dbcfa972b95e1f0dc3ef699b23f10a022b9dfb6d51                                                                        0.0s

Still works to build without BuildKit, too.

$ docker build .
Sending build context to Docker daemon  7.168kB
Step 1/2 : FROM busybox
latest: Pulling from library/busybox
ee153a04d683: Already exists 
Digest: sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70
Status: Downloaded newer image for busybox:latest
 ---> db8ee88ad75f
Step 2/2 : RUN true
 ---> Running in 5c338dc00426
Removing intermediate container 5c338dc00426
 ---> cbbb415368d3
Successfully built cbbb415368d3

$ docker version
Client: Docker Engine - Community
 Version:           18.09.8
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        0dd43dd87f
 Built:             Wed Jul 17 17:38:58 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.8
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       0dd43dd87f
  Built:            Wed Jul 17 17:48:49 2019
  OS/Arch:          linux/amd64
  Experimental:     false
$ cat Dockerfile
FROM busybox
RUN true

[sharifelgamal]

This should be fixed. Feel free to reopen if not.