Received unexpected HTTP status: 500 Internal Server Error when pushing large image from windows

[robindegen]

When pushing a particularly large image from Windows (18.5GB) I receive a 500 error after all the parts have seemingly been uploaded succesfully. I have pushed images from my linux machine (largest was ~1gb) and those have all pushed fine.

I have tried both a memory blob descriptor and a redis one. I thought that maybe that might be the cause but it isn't. I also deleted the data folder and recreated it just to test if maybe something got corrupted, but that doesn't seem to be the case.

Note that I have pushed this exact image to Dockerhub just fine.

I have have a very basic setup of the registry that I started with the following settings.

docker run -d \
        -u root \
        --net=internal \
        --ip 172.18.0.20 \
        --hostname=aeon-registry \
        --restart always \
        -v /docker/containers/registry/data:/var/lib/registry \
        -v /docker/containers/registry/config.yml:/etc/docker/registry/config.yml \
        --name registry registry:2

My config is as follows:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: redis
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :80
  secret: secret
  headers:
    X-Content-Type-Options: [nosniff]
  timeout: 500s
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
redis:
  addr: 172.18.0.22:6379

Output on the windows machine:

C:\Users\robindegen>docker push aeon-registry:80/aeon/win10_vs2019
The push refers to repository [aeon-registry:80/aeon/win10_vs2019]
fe6c60f0b272: Pushed
fc8571232b39: Pushed
62d48696a9ef: Pushed
a24ec4408d2d: Pushed
d1cc15672172: Pushed
0ac52f8f59f5: Pushed
b118e3229ea4: Pushed
f4b2822d9a53: Pushed
29c500f2d4f3: Pushed
314ba5f39117: Pushed
b8f2c26808e8: Pushed
546fcb2020b4: Pushed
2ab0b02dd7a4: Pushed
2e2f393b58c7: Pushed
b13274851650: Pushed
e484a57d56c2: Pushed
a2789f22b6ad: Pushed
a445a20ceb6c: Pushed
bdb2337ae74b: Pushed
a0b9942faa26: Pushed
a394458f5b6b: Skipped foreign layer
a7c7908171f7: Skipped foreign layer
received unexpected HTTP status: 500 Internal Server Error

C:\Users\robindegen>

I attached the full log of this happening. This is where i cleared out the data folder and started clean. Then the only thing i did was push this image from the windows machine.
log.txt

What stands out to me are the following lines at the bottom

time="2020-07-01T12:51:00.540239795Z" level=error msg="response completed with error" err.code=unknown err.message="unknown error" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="aeon-registry:80" http.request.id=5d5154ca-a8bc-4287-958a-c67cc26713ab http.request.method=PUT http.request.remoteaddr="172.18.0.1:53036" http.request.uri="/v2/aeon/win10_vs2019/manifests/latest" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/2ee0c57608 os/windows arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.220894ms http.response.status=500 http.response.written=523 vars.name="aeon/win10_vs2019" vars.reference=latest 
time="2020-07-01T12:51:00.540268343Z" level=error msg="response completed with error" err.code=unknown err.message="invalid URL on layer" go.version=go1.11.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="aeon-registry:80" http.request.id=5d5154ca-a8bc-4287-958a-c67cc26713ab http.request.method=PUT http.request.remoteaddr="172.18.0.1:53036" http.request.uri="/v2/aeon/win10_vs2019/manifests/latest" http.request.useragent="docker/19.03.5 go/go1.12.12 git-commit/2ee0c57608 os/windows arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(windows\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.249434ms http.response.status=500 http.response.written=523 vars.name="aeon/win10_vs2019" vars.reference=latest 
time="2020-07-01T12:51:00.54028954Z" level=error msg="response completed with error" err.code="manifest blob unknown" err.detail=sha256:830e00eda3f1e66421f779823d61a1e8f5529ddb20ac80716c817e2e9658d313 err.message="blob unknown to registry"

[robindegen]

Update: Disabling "validation" seems to help, but now i'm not sure what the implications of that are. Will things just corrupt over time? What is failing validation?

[AndreasHassing]

Since you have registry version 2.7.1 deployed, have a look at #2866.

[alpharameeztech]

@robindegen You set the validation through config.yml file? Can I update the running registry through config.yml file without effecting anything?

[robindegen]

@alpharameeztech No I stopped the registry, edited the file and started it up again.

[alpharameeztech]

I had the same problem, I fixed it by disabling validation.
docker run -d -p 5000:5000 --name registry -e REGISTRY_VALIDATION_DISABLED=true registry:2