Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

golang:1.12-alpine does not have layers for linux/arm64/v8 #269

Open
moikot opened this Issue Mar 14, 2019 · 13 comments

Comments

Projects
None yet
7 participants
@moikot
Copy link

moikot commented Mar 14, 2019

The result of running

docker run --rm weshigbee/manifest-tool inspect golang:1.12-alpine
...
4    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
4       Digest: sha256:7cf1f7ccf392bd834eb91f02892f48992d3c2ba292c2198315a4637bb9454c30
4  Mfst Length: 6981
4     Platform:
4           -      OS: linux
4           -    Arch: arm64
4           - Variant: v8
4           - Feature:
4     # Layers: 0
...

Please note that the number of layers is zero. The same for golang:1.11-alpine

@wglambert

This comment has been minimized.

Copy link

wglambert commented Mar 14, 2019

Looks like an instance of docker-library/official-images#3835

The manifest is updated now

$ docker pull arm64v8/golang:1.12-alpine
1.12-alpine: Pulling from arm64v8/golang
3b00a3925ee4: Already exists 
7809c1a4c8e2: Pull complete 
8c00b1d46f44: Pull complete 
955cc90a48f7: Pull complete 
72f16051d572: Pull complete 
Digest: sha256:05f1d1242721f0042550fcc84f35cbe87f39ef5e6f75852d0608b92f4a2d1878
Status: Downloaded newer image for arm64v8/golang:1.12-alpine
@moikot

This comment has been minimized.

Copy link
Author

moikot commented Mar 14, 2019

The interesting thing is that I could pull 1.12 and 1.11 specifying arm64v8 prefix without a problem. But I can't do my builds using buildkit because it totally relies on the published manifests.
I've just checked again, no layers for 1.12 or 1.11 for arm64 😞 (1.10 is OK)

Here is the buildkit's error:

error: failed to solve: rpc error: code = Unknown desc = failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:7cf1f7ccf392bd834eb91f02892f48992d3c2ba292c2198315a4637bb9454c30 (application/vnd.docker.distribution.manifest.v1+json) from remote: not found
@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Mar 14, 2019

After much digging. The manifest is technically fine. The item in question is a manifest v1:

4 Mfst Type: application/vnd.docker.distribution.manifest.v1+json

And works fine for docker pull. We have no control over what the hub gives us back when we push an image. The Hub gave us "500 Internal Server Error" twice while trying to push the alpine images of golang on the 7th of March. Doing an inspect on the pushed artifacts in arm64v8/golang shows that some pushed as v2 and some as v1. This makes me think that the Docker Hub was having issues and our docker client "helpfully" downgraded us to a v1 push by assuming it was an old registry.

cc @tianon, perhaps we need to somehow disable v1 pushing on our build jobs (this won't fix this now, but will prevent it in the future). edit: this would probably require custom patches to docker itself 😞

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Mar 14, 2019

This is partly a bug in manifest-tool, so I've filed that: estesp/manifest-tool#74

I'm really not very keen on patching Docker for our builders; that's kind of heinous. 😞 😱

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Mar 14, 2019

(Especially given that this was a blip in the Hub that caused these to be pushed in the first place, and docker pull can handle them just fine.)

@tonistiigi

This comment has been minimized.

Copy link

tonistiigi commented Mar 16, 2019

Could someone fix the current version of that golang:1.12-alpine image please. No official image created in the last 3 years should ever be schema1.

cc @dmcgowan For possible docker push side validation for this.

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Mar 16, 2019

Is it still? There was a bump since then that should've pushed a new image that isn't schema1.

@tonistiigi

This comment has been minimized.

Copy link

tonistiigi commented Mar 16, 2019

@tianon Yes, golang:1.12-alpine is ok now. golang:1.12.0-alpine still produces the issue. So it's not as critical anymore(not sure if you usually would bump the previous versions as well in these cases). We should still find a solution where v1 variants could never happen though, even if some error appears in the process.

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Mar 16, 2019

I'm kind of surprised the Hub still allowed a schema1 push 😅

@tonistiigi

This comment has been minimized.

Copy link

tonistiigi commented Mar 16, 2019

fwiw it seems likely to me that the configuration of the manifest list pointing to a v1 manifest will never be supported by containerd. Tracked in containerd/containerd#3100

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Mar 20, 2019

I'm kind of surprised the Hub still allowed a schema1 push 😅

@cowsrule any ideas? ^^

@cowsrule

This comment has been minimized.

Copy link

cowsrule commented Mar 21, 2019

Also surprised, we should be blocking all v1 pushes, will followup internally. we also published this today: https://engineering.docker.com/2019/03/registry-v1-api-deprecation/

@tonistiigi

This comment has been minimized.

Copy link

tonistiigi commented Mar 21, 2019

@cowsrule The issue here is in v2/schema1 image not actual v1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.