trivy on EKS v1.22 running containerd does not scan local images

[youwalther65]

Description

trivy throws errors when running trivy image against local container image when using containerd runtime. This should work with Trivy version => 0.29.. See closed issue https://github.com/aquasecurity/trivy/issues/851

What did you expect to happen?

trivy image to scan image

What happened instead?

# export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
# crictl images | grep kube-proxy
602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy v1.21.2-minimal-eksbuild.1 31efd177236c8 24.5MB
602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy v1.22.6-eksbuild.1 c8c9982c9d037 35.9MB
602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy v1.22.6-minimal-eksbuild.1 c10db2ac3d2d9 25.5MB

# containerd --version
containerd github.com/containerd/containerd 1.4.13 9cc61520f4cd876b86e77edfeb88fbcd536d1f9d

# trivy image 602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1
2022-07-18T14:04:16.338Z INFO Vulnerability scanning is enabled
2022-07-18T14:04:16.339Z INFO Secret scanning is enabled
2022-07-18T14:04:16.339Z INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-07-18T14:04:16.339Z INFO Please see also https://aquasecurity.github.io/trivy/0.30.0/docs/secret/scanning/#recommendation for faster secret detection
2022-07-18T14:04:16.364Z FATAL image scan error: scan error: unable to initialize a scanner: unable to initialize a docker scanner: 4 errors occurred:
* unable to inspect the image (602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the dockerdaemon running?
* unable to initialize Podman client: no podman socket found: stat /run/user/0/podman/podman.sock: no such file or directory
* failed to get 602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1: image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1": not found
* GET https://602401143452.dkr.ecr.eu-west-1.amazonaws.com/v2/eks/kube-proxy/manifests/v1.22.6-minimal-eksbuild.1: unexpected status code 401 Unauthorized: Not Authorized

Output of run with -debug:

# trivy image --debug 602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1
2022-07-18T14:57:38.804Z        DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-07-18T14:57:38.806Z        DEBUG   cache dir:  /root/.cache/trivy
2022-07-18T14:57:38.806Z        DEBUG   DB update was skipped because the local DB is the latest
2022-07-18T14:57:38.806Z        DEBUG   DB Schema: 2, UpdatedAt: 2022-07-18 12:06:56.099185305 +0000 UTC, NextUpdate: 2022-07-18 18:06:56.099184905 +0000 UTC, DownloadedAt: 2022-07-18 13:56:12.228315379 +0000 UTC
2022-07-18T14:57:38.806Z        INFO    Vulnerability scanning is enabled
2022-07-18T14:57:38.806Z        DEBUG   Vulnerability type:  [os library]
2022-07-18T14:57:38.806Z        INFO    Secret scanning is enabled
2022-07-18T14:57:38.806Z        INFO    If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-07-18T14:57:38.806Z        INFO    Please see also https://aquasecurity.github.io/trivy/0.30.0/docs/secret/scanning/#recommendation for faster secret detection
2022-07-18T14:57:38.809Z        DEBUG   failed to get authorization token: MissingRegion: could not find region configuration
2022-07-18T14:57:38.834Z        FATAL   image scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:367
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:227
  - unable to initialize a scanner:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:522
  - unable to initialize a docker scanner:
    github.com/aquasecurity/trivy/pkg/commands/artifact.imageStandaloneScanner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/scanner.go:22
  - 4 errors occurred:
        * unable to inspect the image (602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the dockerdaemon running?
        * unable to initialize Podman client: no podman socket found: stat /run/user/0/podman/podman.sock: no such file or directory
        * failed to get 602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1: image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.22.6-minimal-eksbuild.1": not found
        * GET https://602401143452.dkr.ecr.eu-west-1.amazonaws.com/v2/eks/kube-proxy/manifests/v1.22.6-minimal-eksbuild.1: unexpected status code 401 Unauthorized: Not Authorized

Output of trivy -v:

# trivy -v
Version: 0.30.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-07-18 12:06:56.099185305 +0000 UTC
  NextUpdate: 2022-07-18 18:06:56.099184905 +0000 UTC
  DownloadedAt: 2022-07-18 13:56:12.228315379 +0000 UTC

Additional details (base image name, container registry info...):

[github-actions]

This issue is stale because it has been labeled with inactivity.

[pmengelbert]

@youwalther65 I just spent all day looking at this. There's really no good way to do this with the current CLI as far as I can tell. the relevant code is in pkg/fanal/image/image.go, but I don't think you can Disable... options from the CLI flags.

After monkeying around with the code, I was able to force it to use containerd by setting the proper flags in that file. This relates to my work so I may submit a pull request in the coming days, assuming the maintainers are game.

[pmengelbert]

@youwalther65 Looking at your log, you'll notice that it doesn't give you an error that it couldn't connect to the containerd socket. That means it connected to the socket but couldn't find your image. My guess is that those kube-proxy images are in the k8s.io namespace. Do you have CONTAINERD_NAMESPACE set in your environment for that first command?

trivy currently only looks at the default namespace, default. It's a hardcoded const value in

trivy/pkg/fanal/image/daemon/containerd.go

Line 31 in 9f5113a

defaultContainerdNamespace = "default"

. I'm going to submit a PR that allows you to override the default namespace using the CONTAINERD_NAMESPACE env var.

[youwalther65]

@youwalther65 Looking at your log, you'll notice that it doesn't give you an error that it couldn't connect to the containerd socket. That means it connected to the socket but couldn't find your image. My guess is that those kube-proxy images are in the k8s.io namespace. Do you have CONTAINERD_NAMESPACE set in your environment for that first command?

trivy currently only looks at the default namespace, default. It's a hardcoded const value in

trivy/pkg/fanal/image/daemon/containerd.go

Line 31 in 9f5113a

defaultContainerdNamespace = "default"

. I'm going to submit a PR that allows you to override the default namespace using the CONTAINERD_NAMESPACE env var.

Thanks a lot, will take a look after my business travel

[youwalther65]

I just had to point crictl to the right socket using:

export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock

then crictl image and other commands just work without using CONTAINERD_NAMESPACE env variable.

Otherwise I 'll get a warning like:

# unset CONTAINER_RUNTIME_ENDPOINT

# crictl images
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
IMAGE                                                              TAG                          IMAGE ID            SIZE
xxx.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni-init   v1.10.4-eksbuild.1           ba94d361f4e70       111MB
...

[github-actions]

This issue is stale because it has been labeled with inactivity.