Can't push Windows based images to private registry >= 2.7.0

[MaGitKo]

Hi everyone,

I am using a private registry on Ubuntu 18.04.1 LTS (with insecure access from Windows Server 2016).

Uploading Windows images from Windows Server 2016 works fine with registry version <= 2.6.2.
But if I use registry version >= 2.7.0, I get following error on the Windows console during docker push:
received unexpected HTTP status: 500 Internal Server Error

When I look into the logs of the registry container, I see messages like these:
manifest blob unknown: blob unknown to registry

I tested these configurations with different registry versions:
docker run -d -p 5000:5000 --restart always --name registry registry:2
docker run -d -p 5555:5000 --restart always --name registry registry:2

I attached the logs for registry 2.6.2 (good case) and registry 2.7.1 (bad case):
Logs_Registry_2.6.2.txt
Logs_Registry_2.7.1.txt

What goes wrong here?

Tested with my own Docker image as well as with the hello-world image (both based on nanoserver).

The push of the Linux based hello-world image works fine, even with registry >= 2.7.0.

Thanks in advance for your replies

[MaGitKo]

This is the docker version output on Ubuntu:

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     true

This is the docker version output on Windows Server:

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        20b67756d0
 Built:             unknown-buildtime
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.6
  Git commit:       20b67756d0
  Built:            01/09/2019 17:09:57
  OS/Arch:          windows/amd64
  Experimental:     false

[MaGitKo]

I found out, that the problem does not occur with registry 2.7, if I set the allow-nondistributable-artifacts at the Docker client (in C:\ProgramData\docker\config\daemon.json).

Of course this is only a workaround, not a real solution.

[dozer75]

I can confirm this, for me the workaround is to stick on 2.6.2... :|

I can't deploy that setting on all computers that pushing images.

[yarglah]

I confirm issue with registry > 2.6.2 running on Debian stretch (Docker version 18.09.6, build 481bc77), from Windows server 2016 10.0.14393 (Docker version 18.09.6, build 481bc77), pushing deprecated images microsoft/windowsservercore:latest and newest images mcr.microsoft.com/windows/servercore:ltsc2016.

Works with registry:2.6.2.

[nsaboo]

do we have a solution to push windows container images to registry > 2.7.0 version?

[AndreasHassing]

@stevvooe / @dmcgowan / @aaronlehmann: Can one of you provide some hints on how to solve this issue, so someone can pick it up from the community (I'll do it if it's not too difficult 😃).

Could potentially be a GoodFirstIssue; @MaGitKo provided some good logs to get started.

[alpharameeztech]

I had the same problem, I fixed it by disabling validation.
docker run -d -p 5000:5000 --name registry -e REGISTRY_VALIDATION_DISABLED=true registry:2

[AndreasHassing]

Maybe caused by this?

[geoidesic]

Any progress on this? I'm getting the same issue with:

    - docker build -t $CI_REGISTRY/app/docker-images/appclient .
    - docker push $CI_REGISTRY/app/docker-images/appclient