Private registry push fails (S3): Error with Blob unknown to registry

[jlandersen]

Hi,

We are running with the latest Registry 2.4.0 and trying to use S3 as the backend.
When pushing an image to the registry, it keeps retrying until ultimately ending in "Image push failed". Once the image push has failed the following errors are available:

docker logs registry | grep -E '500 |error'

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:c807ad6f343636e38ac12b5cf4cc90529a8e26e295f9f5f5a746a269b64f9d74 err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=f2ba32b9-c0d8-43ae-aa76-4f8061bc4cc8 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56405" http.request.uri="/v2/webgateway/blobs/sha256:c807ad6f343636e38ac12b5cf4cc90529a8e26e295f9f5f5a746a269b64f9d74" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.143871ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:c807ad6f343636e38ac12b5cf4cc90529a8e26e295f9f5f5a746a269b64f9d74" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:7a90e5079b15df521550cfa6ba5f2ae07a60f08fcb081560bafc6418a22c9faa err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=0d0c45d2-89d2-4580-8215-34fba6a0266a http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56407" http.request.uri="/v2/webgateway/blobs/sha256:7a90e5079b15df521550cfa6ba5f2ae07a60f08fcb081560bafc6418a22c9faa" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.924243ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:7a90e5079b15df521550cfa6ba5f2ae07a60f08fcb081560bafc6418a22c9faa" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:bfe199ea6ecf24ff769f2c9c0a65eba5f2816e6a11c7c8c2b36b7a6da0087e0d err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=6e4bad09-ce07-4811-bac6-176ed6192a54 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56409" http.request.uri="/v2/webgateway/blobs/sha256:bfe199ea6ecf24ff769f2c9c0a65eba5f2816e6a11c7c8c2b36b7a6da0087e0d" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=31.48559ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:bfe199ea6ecf24ff769f2c9c0a65eba5f2816e6a11c7c8c2b36b7a6da0087e0d" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:9f8ccdfa573d9fe166cb9f20e1d05eec768f648879c519eabe686f1f75ab6cf6 err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=e3c04a84-ed10-433d-a579-95b6a4399b75 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56412" http.request.uri="/v2/webgateway/blobs/sha256:9f8ccdfa573d9fe166cb9f20e1d05eec768f648879c519eabe686f1f75ab6cf6" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=24.663973ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:9f8ccdfa573d9fe166cb9f20e1d05eec768f648879c519eabe686f1f75ab6cf6" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:683680549799915daca7cd05676b1d96ad05e63423d074d5aefd7d9240dc9a83 err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=8c6e5366-a2d1-4294-bd03-67c98efb4674 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56416" http.request.uri="/v2/webgateway/blobs/sha256:683680549799915daca7cd05676b1d96ad05e63423d074d5aefd7d9240dc9a83" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=32.042124ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:683680549799915daca7cd05676b1d96ad05e63423d074d5aefd7d9240dc9a83" vars.name=webgateway version=v2.4.0

I can see it creates buckets in S3, so it has connection to it - the configuration is as follows:

docker run \
         -v `pwd`/certs:/certs \
         -e "REGISTRY_STORAGE=s3" \
         -e "REGISTRY_STORAGE_S3_REGION=eu-west-1" \
         -e "REGISTRY_STORAGE_S3_BUCKET=docker-registry" \
         -e "REGISTRY_STORAGE_S3_ACCESSKEY=[OWNKEY]" \
         -e "REGISTRY_STORAGE_S3_SECRETKEY=[OWNKEY]" \
         -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt" \
         -e "REGISTRY_HTTP_TLS_KEY=/certs/domain.key" \
         -d \
         -p 5000:5000 \
         registry:2.4.0

[RichardScothern]

Blob missing errors are not necessarily an indication of error: the daemon will issue Stat calls to the registry to determine if it needs to upload something and use 404 as an indication that it should.

Can you post the registry logs without filtering (up to the first failure should be sufficient).

Are you using an IAM for the registry?

Do the S3 logs indicate any errors?

[jlandersen]

I have attached the complete log from starting the registry to the client making the push failing.
I am trying to get the S3 logs, will post them here asap.
No IAM is used for the machine.

Thanks!
Attachment: out.txt

[RichardScothern]

I'm interested to see those logs.

Also, which docker engine are you using? I see UpstreamClient(Docker-Client..) in the user-agent string. The logs from the engine (run in debug mode) may shed more light onto this.

[jlandersen]

Sorry for the late reply, took some time to get ahold of it. Attached is the S3 logs.
It is currently running Docker 1.11 on both sides.

logs.zip

[RichardScothern]

I can't see anything amiss there.

Do you have the logs from the engine (run in debug mode)?

[jlandersen]

Attached are the logs from the registry engine running in debug mode

out.txt

[RichardScothern]

@jlandersen There is no reference to a docker push in those logs.

[jlandersen]

@RichardScothern apologies, I thought you wanted it for the registry - here it is for the client making the push.
I am seeing some error messages in there related to a self-signed cert - is this causing issues when using S3 as storage?

Attachment:
out.txt

[RichardScothern]

There is an outstanding bug for this in the engine. For more details:
#426 (comment)

The fix is to append your own ca.crt to a complete ca-certificates.crt and use the result as /etc/docker/certs.d/REGISTRY/ca.crt. assume ca-certificates.crt is some full CA-chain from your favorite OS

cat ca-certificates.crt ca.crt >> ca-full.crt
cp ca-full.crt /etc/docker/certs.d/REGISTRY/ca.crt

[iahmad-khan]

I am facing similiar issue , the image upload process ends normally on the client but there is no image in docker-hub repo , from the logs i see:

Aug 12 10:29:51 cmsbuild7-01 dockerd: time="2016-08-12T10:29:51.091865728+02:00" level=error msg="Upload failed, retrying: blob upload unknown"
Aug 12 10:29:57 cmsbuild7-01 kernel: XFS (dm-1): Mounting V4 Filesystem
Aug 12 10:29:57 cmsbuild7-01 kernel: XFS (dm-1): Ending clean mount
Aug 12 10:33:28 cmsbuild7-01 dockerd: time="2016-08-12T10:33:28.360053301+02:00" level=error msg="Not continuing with push after error: context canceled"

[RichardScothern]

@iahmad-khan : i'm not sure how this is the same issue. Your output says the upload was cancelled. If you are seeing a bug with the registry please file a new issue with the relevant details:

https://github.com/docker/distribution/blob/master/CONTRIBUTING.md#reporting-an-issue-properly

[shenshouer]

I met the same issue with the harbor behind nginx ingress. the registry service is registry:5000 and harbor nginx service is nginx:80,

I push image to registry:5000 and nginx:80 worked OK, and got "blob unknown to registry" from nginx ingress.

So I think there have some thing like digest in header had bean erased by nginx ingress.

[shenshouer]

It worked OK when I removed the tls from ingress config.

[leon0306]

@shenshouer I meet the same error. How do u resolve this probelm?

[hasonhai]

I encounter the same issue. Is there any solution but removing the tls from ingress config?

[ghost]

I encounter the same issue. Is there any solution but removing the tls from ingress config?

I was trying to deploy registry in Kubernetes but got the same error.

I got it to work by removing forced redirects to SSL/TLS.

  annotations:
    kubernetes.io/ingress.class: nginx # use created nginx ingress
    cert-manager.io/cluster-issuer: "letsencrypt-issuer" # user created issuer
    # nginx.ingress.kubernetes.io/ssl-redirect: "true" # force all traffic to https < ---- comment out this line
    nginx.ingress.kubernetes.io/proxy-body-size: 1000m

[inselberg]

Solved via:
proxy_set_header X-Forwarded-Proto "https";

in nginx config.

[MrNocTV]

Solved via: proxy_set_header X-Forwarded-Proto "https";

in nginx config.

and

It worked OK when I removed the tls from ingress config.

Both of these do not work :(

This is my ingress config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    kubernetes.io/tls-acme: "true"
  name: private-repository-k8s
  namespace: mlu-showroom-api-alpha
spec:
  rules:
  - host: mlu-registry-alpha.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: private-repository-k8s
            port: 
              number: 5000
  tls:
  - hosts:
    - mlu-registry-alpha.com
    secretName: private-registry-haproxy-tls

[knee-cola]

X-Forwarded-Proto "https" has solved problem for me too.

My registry is published via Traefik and an SSL offloader.

Here's a snippet from docker-compose file which configures this:

        # Add headers to request before it arrives at the registry server
        # Without this, when a push is attempted the docker client will log "unknown blob"
        # while the Docker registry server will log "blob unknown to registry"
        - traefik.http.middlewares.my-private-registry--add-headers.headers.customrequestheaders.X-Forwarded-Proto=https
        - traefik.http.routers.my-private-registry-registry.middlewares=my-private-registry--add-headers

I suspect this might have something to do with the fact that traffic is SSL offloaded before it arrived at registry server.

I also tried playing with maxRequestBodyBytes and memRequestBodyBytes which however made no difference.

[playerla]

There is an outstanding bug for this in the engine. For more details:
#426 (comment)

The fix is to append your own ca.crt to a complete ca-certificates.crt and use the result as /etc/docker/certs.d/REGISTRY/ca.crt. assume ca-certificates.crt is some full CA-chain from your favorite OS

cat ca-certificates.crt ca.crt >> ca-full.crt
cp ca-full.crt /etc/docker/certs.d/REGISTRY/ca.crt

Hi,

I suspect untrusted registry cert by docker client.

Work well without certificates. No proxy.
When I upgrade to TLS on port 5000, got "blob unknown to registry" (certificate signed by private PKI, ca.crt deployed on Debian). But pushing to Harbor with same docker client works well on https.

I switch registry to 443 and it works !

[jimmyjxiao]

Can confirm. Tried everything under the sun including the proxy header options here. Nothing worked except changing the container registry to its own subdomain and going to 443 instead of 5050.

[kubecto]

My current approach is to use the local site, store the data locally, and then I sync the data to the minio cluster every night

mc cp --recursive /home/dockerstore/ shandong/docker-registry