unable to pull layered images

[philips]

Minikube version (use minikube version): v0.16.0

Environment:

• OS (e.g. from /etc/os-release): OSX
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): VMWare Fusion
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep ISO): "ISO": "/Users/philips/.minikube/machines/minikube/boot2docker.iso",

What happened:

$ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
0a8490d0dfd3: Pull complete
Digest: sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
Status: Downloaded newer image for alpine:latest
$ docker pull gcr.io/google_containers/pause-amd64:3.0
3.0: Pulling from google_containers/pause-amd64
a3ed95caeb02: Pull complete
f11233434377: Extracting 311.2 kB/311.2 kB
failed to register layer: operation not supported

What you expected to happen:

No error:

failed to register layer: operation not supported

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

[philips]

Full logs

Feb 03 21:11:56 minikube systemd[1]: Started docker.service.
Feb 03 21:11:56 minikube docker[1435]: time="2017-02-03T21:11:56.422393903Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:11:56 minikube docker[1435]: time="2017-02-03T21:11:56.421257860Z" level=info msg="API listen on [::]:2376"
Feb 03 21:11:56 minikube docker[1435]: time="2017-02-03T21:11:56.422727553Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:11:57 minikube docker[1435]: time="2017-02-03T21:11:57.380760204Z" level=debug msg="Calling GET /version"
Feb 03 21:11:57 minikube docker[1435]: time="2017-02-03T21:11:57.381392993Z" level=debug msg="Calling GET /v1.23/version"
Feb 03 21:11:57 minikube docker[1435]: time="2017-02-03T21:11:57.423946626Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:11:58 minikube docker[1435]: time="2017-02-03T21:11:58.211183005Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:11:58 minikube docker[1435]: time="2017-02-03T21:11:58.211821025Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:11:58 minikube docker[1435]: time="2017-02-03T21:11:58.328664327Z" level=debug msg="containerd connection state change: READY"
Feb 03 21:11:58 minikube docker[1435]: time="2017-02-03T21:11:58.425614377Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:11:59 minikube docker[1435]: time="2017-02-03T21:11:59.427153236Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:11:59 minikube docker[1435]: time="2017-02-03T21:11:59.570733389Z" level=debug msg="Calling GET /v1.23/version"
Feb 03 21:12:00 minikube docker[1435]: time="2017-02-03T21:12:00.211097980Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:00 minikube docker[1435]: time="2017-02-03T21:12:00.428696241Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:01 minikube docker[1435]: time="2017-02-03T21:12:01.429491111Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:01 minikube docker[1435]: time="2017-02-03T21:12:01.970807789Z" level=debug msg="Calling POST /v1.23/images/create?fromImage=gcr.io%2Fgoogle_containers%2Fpause-amd
64%3A3.0"
Feb 03 21:12:02 minikube docker[1435]: time="2017-02-03T21:12:02.100540458Z" level=debug msg="hostDir: /etc/docker/certs.d/gcr.io"
Feb 03 21:12:02 minikube docker[1435]: time="2017-02-03T21:12:02.157600889Z" level=debug msg="hostDir: /etc/docker/certs.d/gcr.io"
Feb 03 21:12:02 minikube docker[1435]: time="2017-02-03T21:12:02.157685457Z" level=debug msg="Trying to pull gcr.io/google_containers/pause-amd64 from https://gcr.io v2"
Feb 03 21:12:02 minikube docker[1435]: time="2017-02-03T21:12:02.210414277Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:02 minikube docker[1435]: time="2017-02-03T21:12:02.210914873Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:02 minikube docker[1435]: time="2017-02-03T21:12:02.430854649Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:03 minikube docker[1435]: time="2017-02-03T21:12:03.432031869Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:03 minikube docker[1435]: time="2017-02-03T21:12:03.487956240Z" level=debug msg="Pulling ref from V2 registry: gcr.io/google_containers/pause-amd64:3.0"
Feb 03 21:12:03 minikube docker[1435]: time="2017-02-03T21:12:03.488513991Z" level=debug msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b
46d4\""
Feb 03 21:12:03 minikube docker[1435]: time="2017-02-03T21:12:03.488813218Z" level=debug msg="pulling blob \"sha256:f112334343777b75be77ec1f835e3bbbe7d7bd46e27b6a2ae35c6b3cfea0
987c\""
Feb 03 21:12:03 minikube docker[1435]: time="2017-02-03T21:12:03.913147373Z" level=debug msg="Calling GET /v1.23/images/json"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.061764490Z" level=debug msg="Downloaded a3ed95caeb02 to tempfile /mnt/sda1/var/lib/docker/tmp/GetImageBlob61525
9287"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.061946071Z" level=debug msg="Start untar layer"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.077272673Z" level=debug msg="Untar time: 0.015320001000000001s"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.077332569Z" level=debug msg="Applied tar sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6e
f to 9b1e464e51d06001d28ef0af7b35f6ed5f37c76bfd517e5418009e6fcab592e2, size: 0"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.210908775Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.211648636Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.433537947Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.461184379Z" level=debug msg="Downloaded f11233434377 to tempfile /mnt/sda1/var/lib/docker/tmp/GetImageBlob00170
3178"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.465659581Z" level=debug msg="Cleaning up layer 0f13544555d30a5fa0f551da27e3ecf428547c6e20294857611fe6074335d64a
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.465659581Z" level=debug msg="Cleaning up layer 0f13544555d30a5fa0f551da27e3ecf428547c6e20294857611fe6074335d64a
: operation not supported"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.473262698Z" level=error msg="Error trying v2 registry: failed to register layer: operation not supported"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.473315439Z" level=error msg="Attempting next endpoint for pull after error: failed to register layer: operation
 not supported"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.473458378Z" level=debug msg="Skipping v1 endpoint https://gcr.io because v2 registry was detected"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.473801412Z" level=info msg="Layer sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef clean
ed up"
Feb 03 21:12:04 minikube docker[1435]: time="2017-02-03T21:12:04.572893867Z" level=debug msg="Calling GET /v1.23/version"
Feb 03 21:12:05 minikube docker[1435]: time="2017-02-03T21:12:05.435359467Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:06 minikube docker[1435]: time="2017-02-03T21:12:06.211024159Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:06 minikube docker[1435]: time="2017-02-03T21:12:06.211360440Z" level=debug msg="Calling GET /v1.23/containers/json?limit=0"
Feb 03 21:12:06 minikube docker[1435]: time="2017-02-03T21:12:06.437409965Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"
Feb 03 21:12:07 minikube docker[1435]: time="2017-02-03T21:12:07.388826470Z" level=debug msg="Calling GET /version"
Feb 03 21:12:07 minikube docker[1435]: time="2017-02-03T21:12:07.389247851Z" level=debug msg="Calling GET /v1.23/version"
Feb 03 21:12:07 minikube docker[1435]: time="2017-02-03T21:12:07.438631229Z" level=debug msg="Calling GET /v1.23/containers/json?all=1&limit=0"```

[philips]

Moving to the --storage-driver=vfs "fixes it". I had been using the overlay driver.

[dlorenc]

Hmm, maybe the new minikube ISO kernel doesn't support the overlay driver.

[philips]

Hrm, looks like the vmware fusion driver might not be using the right ISO url for some reason. This works:

$  minikube start --vm-driver=vmwarefusion  --memory=4096 --v=5 --show-libmachine-logs=true   --iso-url=https://storage.googleapis.com/minikube/iso/minikube-v1.0.6.iso

[philips]

$ docker info
Containers: 8
 Running: 8
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 1.11.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Swarm:
 NodeID:
 Is Manager: false
 Node Address:
Security Options:
Kernel Version: 4.7.2
Operating System: Buildroot 2016.08
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: minikube
ID: 34NE:VRKU:3FKM:XVPM:HPQZ:QGJ4:325W:OUDR:BOJO:XXJT:X3LU:ICBY
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=vmwarefusion
Insecure Registries:
 127.0.0.0/8

Yep, works. I will close this and open a new version.