clair.layer400 Bad Request: "vulnerability scanning for <image> failed: clair error: could not find layer"

[tedsluis]

I am not able to get reg working. My setup is:

• openshift 3.11
• Openshift registry
• clair 2.1.2 (running on openshift)
• reg (running on openshift in a container based on redhat ubi image)

To access the openshift registry you need a token:

$MYTOKEN=$(oc whoami -t)

Using this token I can do for example a docker login:

$ docker login -u $USER -p $MYTOKEN <registry>

Now I am able to push and pull images.

I also use this token in reg and clair to read the manifest and image layers in the examples below:

Using reg form inside the reg container I can list images in the pls-clair inside my registry, using my openshift authentication token:

sh-4.2$ ./reg ls -u ted.sluis.ocp -p $MYTOKEN -d -k  docker-registry.default.svc:5000 list
INFO[0000] domain: docker-registry.default.svc:5000
INFO[0000] server address: docker-registry.default.svc:5000
pls-clair/clair                                                v2.1.2
pls-clair/reg                                                  v1.0
pls-clair/ubi                                                  latest
(left out other images)

I run reg sever in the reg container with the follow arguments:

$ reg server -d --clair http://clair:6060 -k -u ted.sluis -p $MYTOKEN  -r docker-registry.default.svc:5000 --asset-path /tmp --port 6006 

It serves the static web page with the images in my registry and their individual tags. It is not able to service layer information.

This is what I see in the reg log when I try to view the pls-clair/clair:v2.1.2 image from the static web page:

time="2020-04-24T04:37:01Z" level=info msg="fetching vulnerabilities" URL="/repo/pls-clair%2Fclair/tag/v2.1.2/vulns" func=vulnerabilities method=GET
2020/04/24 04:37:01 registry.manifests uri=https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2 repository=pls-clair/clair ref=v2.1.2
2020/04/24 04:37:01 registry.registry resp.Status=200 OK
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:339073ee4259ec00139ce0f376829e2c265f67aabe406e82cb6e2ee559ea1ea6
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:339073ee4259ec00139ce0f376829e2c265f67aabe406e82cb6e2ee559ea1ea6
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:df12825c8c18ed0861e6dda82b3c9cdca4c00b65d57e6d04f67d3e198fab3a06
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:df12825c8c18ed0861e6dda82b3c9cdca4c00b65d57e6d04f67d3e198fab3a06
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:a9915232c639134dda946eef2ca8dda4692be91be52b12a7ffe9d53d0ec0bf81
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:a9915232c639134dda946eef2ca8dda4692be91be52b12a7ffe9d53d0ec0bf81
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:91c388d3f8dfb80ca43c81d0c424483c7c3238ad175da1a68d4bae2d44e7a238
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:91c388d3f8dfb80ca43c81d0c424483c7c3238ad175da1a68d4bae2d44e7a238
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:4a33db4f96e98ba0b227eb9476f8931f07e8c38bc9793bcec65d90000bb8e855
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:4a33db4f96e98ba0b227eb9476f8931f07e8c38bc9793bcec65d90000bb8e855
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:97ac75448aa2cb011366f12171e36234a3418e4beccf911881dfe3dfdeb37a50
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:97ac75448aa2cb011366f12171e36234a3418e4beccf911881dfe3dfdeb37a50
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:901faf5e6502c7c8de28af6c73f08053ce2f69aeed3539c1612eb63acaaf5fd0
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:901faf5e6502c7c8de28af6c73f08053ce2f69aeed3539c1612eb63acaaf5fd0
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:cddc5806bc93e98d13f39a7e7fd8dc13bf27f772f6f9b0d9c0251b962afc0448
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:cddc5806bc93e98d13f39a7e7fd8dc13bf27f772f6f9b0d9c0251b962afc0448
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:f2e593b86155abf6c8a2c50fd6e086e76b0e2a68c52b07a6218810c5ccbaa3cc
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:f2e593b86155abf6c8a2c50fd6e086e76b0e2a68c52b07a6218810c5ccbaa3cc
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b
2020/04/24 04:37:01 clair.ancestry.post name=sha256:3cad95957f1fee23e262cdd5bb084abcd827b2aba78edec194a217c09f98224e
2020/04/24 04:37:01 registry.manifests uri=https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2 repository=pls-clair/clair ref=v2.1.2
2020/04/24 04:37:01 registry.registry resp.Status=200 OK
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 clair.layers.post url=http://clair:6060/v1/layers name=sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 clair.layers.post req.Body={"Layer":{"Name":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","Path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","Format":"Docker"}}
2020/04/24 04:37:01 clair.layers.post resp.Status=400 Bad Request
time="2020-04-24T04:37:01Z" level=error msg="vulnerability scanning for pls-clair/clair:v2.1.2 failed: clair error: could not find layer" URL="/repo/pls-clair%2Fclair/tag/v2.1.2/vulns" func=vulnerabilities method=GET

And in the clair log I see this at the same time:

{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2020-04-24 04:37:00.296345","engine version":3,"format":"Docker","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","parent layer":"","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2020-04-24 04:37:00.305892","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2020-04-24 04:37:00.305950","error":"could not find layer","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2020-04-24 04:37:00.306556","elapsed time":10312151,"method":"POST","remote addr":"10.131.0.1:42472","request uri":"/v1/layers","status":"400"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2020-04-24 04:37:01.847010","engine version":3,"format":"Docker","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","parent layer":"","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2020-04-24 04:37:01.857130","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2020-04-24 04:37:01.857229","error":"could not find layer","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2020-04-24 04:37:01.858427","elapsed time":11477492,"method":"POST","remote addr":"10.131.0.1:42472","request uri":"/v1/layers","status":"400"}

In the registry I see this at the same time:

time="2020-04-24T04:36:59.942638871Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=db646aee-0caf-4f5d-b205-b6ed0a355796 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/tags/list http.request.useragent=Go-http-client/1.1 http.response.contenttype="application/json; charset=utf-8" http.response.duration=9.320875ms http.response.status=200 http.response.written=45 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:36:59.942677086Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=bf37e1d9-4b36-4890-a683-65f5eccfb272 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/tags/list http.request.useragent=Go-http-client/1.1 http.response.contenttype="application/json; charset=utf-8" http.response.duration=9.399815ms http.response.status=200 http.response.written=45 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:36:59.961723966Z" level=info msg="rewriting manifest sha256:55fb9b5af9a1862fea000da8157919f083de1cd328ce25cc1593f3321dc6ef3d in schema1 format to support old client" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=01581626-3cab-48f3-be26-2217ddb10fc1 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076 openshift.auth.user=Ted.Sluis.ocp openshift.auth.userid=28b463bc-020f-11ea-9541-005056b65da0 vars.name=pls-clair/clair vars.reference=v2.1.2
time="2020-04-24T04:36:59.969134594Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=01581626-3cab-48f3-be26-2217ddb10fc1 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.response.duration=25.945578ms http.response.status=200 http.response.written=17097 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:36:59.969294048Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=9c430cdd-d6a1-4616-94b9-d260b031d33b http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.response.duration=26.172736ms http.response.status=200 http.response.written=17097 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.083026722Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=1af53c15-dd35-4fbb-9f5e-b5094ea62e5b http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=13.614697ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.083159483Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=1c4800cd-0383-46b5-9218-d7d8e5884d8e http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=13.788153ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.283528191Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=38853b66-c344-405e-bfd7-9817f0b7b719 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=11.819507ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.28356633Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=9737e4b1-c2bf-4d6a-a2ca-b42632442b23 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=11.898217ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.640114926Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=8eb8f2ab-6036-4091-9de3-138d95f51985 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=26.200464ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.640267302Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=20beee50-2a8a-438e-8383-7c769aa3c4af http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=26.372037ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.834676388Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=53ae0dc9-fcbb-4ede-8f8c-b0e2807a6f1a http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=12.667436ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.834711297Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=a2fe45ce-254b-41c3-9862-5d494d5964fe http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=12.738622ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076

I am not sure what's going wrong. To troubleshoot I tried the following steps:

1. Getting the manifest of an image from my registry using my token.

curl -k  -H "Authorization: Bearer $MYTOKEN" -X GET -i https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2
HTTP/1.1 200 OK
Content-Length: 17097
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Docker-Content-Digest: sha256:a6d230227302affbd296d26b35293630af714c40ddd468d33ab0a92b9de25e74
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:a6d230227302affbd296d26b35293630af714c40ddd468d33ab0a92b9de25e74"
X-Registry-Supports-Signatures: 1
Date: Fri, 24 Apr 2020 06:27:10 GMT
{
   "schemaVersion": 1,
   "name": "pls-clair/clair",
   "tag": "v2.1.2",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b"

The full output can be found here: https://pastebin.com/FqCNP1fh

2. Getting the blobs from the manifest of this image from my registry using my token.

$ curl -k  -H "Authorization: Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo" -X GET -i https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2 | grep blob
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
         "blobSum": "sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b"
         "blobSum": "sha256:f2e593b86155abf6c8a2c50fd6e086e76b0e2a68c52b07a6218810c5ccbaa3cc"
         "blobSum": "sha256:cddc5806bc93e98d13f39a7e7fd8dc13bf27f772f6f9b0d9c0251b962afc0448"
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
1         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:901faf5e6502c7c8de28af6c73f08053ce2f69aeed3539c1612eb63acaaf5fd0"
          "blobSum": "sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1"
1         "blobSum": "sha256:97ac75448aa2cb011366f12171e36234a3418e4beccf911881dfe3dfdeb37a50"
7         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
9         "blobSum": "sha256:4a33db4f96e98ba0b227eb9476f8931f07e8c38bc9793bcec65d90000bb8e855"
7         "blobSum": "sha256:91c388d3f8dfb80ca43c81d0c424483c7c3238ad175da1a68d4bae2d44e7a238"
          "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          "blobSum": "sha256:a9915232c639134dda946eef2ca8dda4692be91be52b12a7ffe9d53d0ec0bf81"
1         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          "blobSum": "sha256:df12825c8c18ed0861e6dda82b3c9cdca4c00b65d57e6d04f67d3e198fab3a06"
1         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
7         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:339073ee4259ec00139ce0f376829e2c265f67aabe406e82cb6e2ee559ea1ea6"
9         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
7         "blobSum": "sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd"
          "blobSum": "sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5"
          "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          "blobSum": "sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"

3. Downloading the blob from the registry using using my token:

sh-4.2$ curl -k  -H "Authorization: Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo" -X GET -i https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 > /tmp/layer

sh-4.2$ ls -l /tmp
-rw-r--r--. 1 1016010000 root 462 Apr 24 06:37 layer

4. Post a layer to clair:

$ curl -k -X POST -i http://clair:6060/v1/layers -d '{"Layer": {"Name": "b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1", "Path": "https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1", "Headers": {  "Authorization": "Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo" }, "Format": "Docker", "ParentName": ""}}'
HTTP/1.1 201 Created
Content-Type: application/json;charset=utf-8
Server: clair
Date: Fri, 24 Apr 2020 06:54:24 GMT
Content-Length: 353

{"Layer":{"Name":"b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1","Path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1","Headers":{"Authorization":"Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo"},"Format":"Docker","IndexedByVersion":3}}

The logging in the clair container:

{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2020-04-24 06:54:24.526477","engine version":3,"format":"Docker","layer":"b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1","parent layer":"","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2020-04-24 06:54:24.578878","elapsed time":52523668,"method":"POST","remote addr":"10.131.0.1:50514","request uri":"/v1/layers","status":"201"}

A http status 201 looks okay, I think (201 = result of HTTP POST request, one or more new resources have been successfully created on server).

Now back to reg and the error I got in the first place while running reg server:

2020/04/24 04:37:01 clair.layers.post resp.Status=400 Bad Request
time="2020-04-24T04:37:01Z" level=error msg="vulnerability scanning for pls-clair/clair:v2.1.2 failed: clair error: could not find layer" URL="/repo/pls-clair%2Fclair/tag/v2.1.2/vulns" func=vulnerabilities method=GET

According the clair v1 api documentation a http status 400 means "The body of the request invalid". Does reg request to clair in the correct way?

Were it I go wrong? How can I resolve this. or troubleshoot. Any feedback is welcome.

[danielwindit]

Hitting this same issue, same circumstances

[danielwindit]

Solved it in my case by adding out custom CA to the container running REG and pointing reg (-r) to the route of the internal registry of Openshift instead of the service.