analyze-local-image could not analyze layer: Got response 404 with message

[jeremie0]

I've got clair running in a docker container:

-sh-4.2$  docker run --env-file ./proxy-env  -p 6060:6060 -p 6061:6061  -v /tmp:/tmp  -v /home/jeremie/quay-config:/config:ro quay.io/coreos/clair:latest --config=/config/config.yaml
2016-02-11 17:27:33.823292 I | database: database at /db/bolt does not exist yet, creating it
2016-02-11 17:27:33.835770 I | api: starting main API on port 6060.
2016-02-11 17:27:33.836521 I | notifier: notifier service is disabled
2016-02-11 17:27:33.836549 I | api: starting health API on port 6061.
2016-02-11 17:27:33.836819 I | updater: updater service started. lock identifier: f3a44bee-bf15-4242-8b0a-60a0c3f70bbe
2016-02-11 17:27:33.839272 I | updater: updating vulnerabilities
2016-02-11 17:27:33.839360 I | updater/fetchers: fetching Debian vulneratibilities
2016-02-11 17:27:33.839437 I | updater/fetchers: fetching Red Hat vulneratibilities
2016-02-11 17:27:33.839863 I | updater/fetchers: fetching Ubuntu vulneratibilities
2016-02-11 17:56:24.140684 N | database: Ignoring 17374 notifications
2016-02-11 17:56:24.492351 I | updater: update finished
2016-02-11 17:58:37.701983 E | worker: layer <missing>: failed to extract data from /tmp/analyze-local-image-257595031/<missing>/layer.tar: the resource cannot be found
2016-02-11 17:58:37.704748 I | api: 404 POST /v1/layers (96.482403ms)

But when I try to analyze-local-images debian:jessie, it is able to save the image, but errors during analyis:

-sh-4.2$ ./work/bin/analyze-local-images debian:jessie
Saving debian:jessie
Getting image's history
Analyzing 2 layers
- Analyzing <missing>
2016/02/11 11:58:37 - Could not analyze layer: Got response 404 with message {"Message":"the resource cannot be found"}

If I look inside the container that's running clair, I do see the images in /tmp/

-sh-4.2$ docker exec insane_bohr ls -l /tmp/analyze-local-image-257595031/
total 20
-rw-r--r-- 1 29857 5002 1395 Jan 25 22:24 9a02f494bef8d0d088ee7533aa1ba4aaa1dbf38a97192d36fa79a51279bc04de.json
drwxr-xr-x 2 29857 5002 4096 Jan 25 22:24 9c3e80c252d138c43eee4c75cdcaaea98b1d2809b9f29263a293cbcae2b30366
drwxr-xr-x 2 29857 5002 4096 Jan 25 22:24 a240a64787deebe0fbf5ac2682dd24f4f93b6275e4d61ded85f7e2a3e6771c0d
-rw-r--r-- 1 29857 5002  279 Jan  1  1970 manifest.json
-rw-r--r-- 1 29857 5002   89 Jan  1  1970 repositories

Any ideas?

[Quentin-M]

Hi,

I believe that's because you have to use the actual Image ID of the image instead of the repository name with that contrib tool. When you do docker images, use the string from the third column, called Image ID.

PS: You should probably mount /db as well to avoid loosing the Clair database each time you restart the Docker container.

[jeremie0]

Using the image ID doesn't help. But, I've dug a little bit more and I think the problem is because of changes with Docker 1.10 (moby/moby#20131 has similar symptoms).

If I try to use the image ID I get the same error:

-sh-4.2$ docker images debian:jessie
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
debian              jessie              9a02f494bef8        2 weeks ago         125.1 MB
-sh-4.2$ ./work/bin/analyze-local-images 9a02f494bef8
Saving 9a02f494bef8
Getting image's history
Analyzing 2 layers
- Analyzing <missing>
2016/02/12 15:09:03 - Could not analyze layer: Got response 404 with message {"Message":"the resource cannot be found"}

But I also see the missing image ID when I look at docker history (and this happens for all the containers I've tried)

-sh-4.2$ docker history 9a02f494bef8
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
9a02f494bef8        2 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
<missing>           2 weeks ago         /bin/sh -c #(nop) ADD file:e5a3d20748c5d3dd5f   125.1 MB

Is clair or the analyze-local-images relying on this same history info?

[Quentin-M]

Really good catch. The contrib tool does rely on docker history -q --no-trunc to determine all the layers that it needs to POST to Clair, and in which hierarchy/order.

[kuldeepb]

Clair is broken because of the content addressability inconsistencies in docker 1.10 (clair works fine till docker version 1.9.1); "docker history" command gives SHA256 layer IDs, while "docker save" command saves the layers.tar files in plain layer-UID directories.

Not sure if clair is going to fix the issue or will it be taken care of when distribution/distribution#727 feature is supported by Docker.

[Quentin-M]

Clair is not broken per se. The contrib tool that helps analyzing local images is.

[kuldeepb]

Yes, only the contrib tool is affected. As a work around, would it work if the contrib tool for local image scanning is run inside a container with docker version 1.9.1 installed in the container and by mounting /var/lib/docker directory of host in the container? I guess not as the migration tools of docker 1.10 talks only of converting 1.9.x images to 1.10 compatible images and not the other way round.

[Quentin-M]

It won't. I believe that the solution would be to make the contrib tool read the manifests instead as they orderly point to the layers / blobs. I'll try to experiment and see how it goes asap. Just a lot of work with the new major version of Clair that will be released really soon. Any contributions are very welcomed.

[liangchenye]

The saved manifest uses the struct in docker/image/tarexport/tarexport.go.
The patch fixed it.

[liangchenye]

@kuldeepb I cannot find the place to reply your last comment in github :(
I still think manifest[0] is enough, even if we use a loop for the range of manifest, we need to break/return when we catch the first item.

[kuldeepb]

Sorry, I deleted my comment after seeing your reply to the first one. I
will try out one such image later and see if their is any issue. Right now
analyse-local-image tool gives CVE details from topmost layer, in case it
starts supporting reports for all the layers of given image it would be
required I believe.
On Feb 14, 2016 13:05, "梁辰晔 (Liang Chenye)" notifications@github.com
wrote:

@kuldeepb https://github.com/kuldeepb I cannot find the place to reply
your last comment in github :(
I still think manifest[0] is enough, even if we use a loop for the range
of manifest, we need to break/return when we catch the first item.

—
Reply to this email directly or view it on GitHub
#69 (comment).

[prashant-wipro]

We have installed the latest version of Clair and still getting this issue. Do we still have to use Docker 1.9.1? Is this the conclusion of this issue?