QuestionHello, recently trivy is having issues updating the database before scanning container images. To isolate the problem, I have introduced a separate update task which runs the update-only commands. Here is the update commands output:
From my point of view, it looks like it's a problem related to the Github container registry. I guess there is some kind of DDoS protection in place. Has anyone already faced the same issue? Or already found a solution? Best regards, TargetContainer Image ScannerNone Output FormatNone ModeStandalone Operating SystemUbuntu Version0.54.1 |
Replies: 13 comments 58 replies
|
Hi @gwitsch ! We have encountered the same thing at CI and are still looking into the problem. |
|
I have seen this as well |
|
Please see https://aquasecurity.github.io/trivy/v0.55/docs/references/troubleshooting/#github-rate-limiting |
|
For those that my have a on site registry that can act as a cache for remote content, the following environment variables can prevent the problem by making Trivy pull from the cache on not hit the ghcr pull limit: where |
|
Adding a Github token seems to be working. At least the last couple of days, the update process worked as expected. |
|
I added a token and 1 out of 2 scans still failed. Haven’t seen any failures since Thursday though without using token. |
|
Sadly, Github tokens don't fix the issue. |
|
Same here. |
|
We published trivy-db in Amazon ECR Public Gallery as well as GHCR. The following should work now. We'll update the documentation tomorrow. |
|
@Connor-Wood i'm sorry i don't know, i just saw it in the docs |
|
I would guess that you used a slightly wrong command line syntax. Not sure how Because the way you did it, trivy won't understand that the second repo's URI belongs to the |
|
@knqyf263 Could you please populate the trivy-chekcs repo too on ecr? https://gallery.ecr.aws/aquasecurity/trivy-checks |
|
@ktzsolt I'm not an owner of that. It's being discussed here. aquasecurity/trivy-checks#262 |
|
it works for me. Thanks! |
|
I've not been able to get a successful Trivy run in GitLab for the last few hours due to this issue. We've seen it on and off in the past but all of our installs fail 100% now. This is causing our pipelines to fail so we can no longer merge code. |
that makes sense, thank you for providing the context |
The long term solution doesn't seem to be working for us as we had this in place long before these issues have been popping up. For the short term solution being using ECR how frequently is this image updated as I don't want to be pointing to an image that will only be updated short term? Is there a long-term solution we can use or are we good to permanently use the ECR image from now on? |
@knqyf263 If you need any help with Gitlab configuration, let me know. This is what our Gitlab job looks like: We're still facing issues with the TOOMANYREQUESTS error though. As you can see we're using a cache:key: value here. This defines when the cache is re-used in another job. In this case I'm using a hardcoded string value, so each pipeline job in the same repository will use the same cache. However, we have a lot of different projects and repository and it's not possible to share cache across repositories in Gitlab (due to security reasons and cache poisoning). |
Does this work for you? Because (on GitHub Actions btw) with I'm getting |
It works fine for me on all of our Gitlab projects. |
|
I found an option to create a local cache by setting the environment variable TRIVY_CACHE_DIR= and using the command trivy --download-db-only. It downloads the database to the specified path, and when scanning images, it won't connect to https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2. I think this could solve the issue. Doc : https://aquasecurity.github.io/trivy/v0.56/docs/advanced/air-gap/#populating-the-trivy-cache |
|
Thank you, @bryanhermass, |
|
@kapi-caz I'm using AZ DevOps, but only specifying cache folder is not a solution, didn't try to add '--download-db-only' but here is the question, why I need this flag, and what will it affect if I'm using like this in the future ? |
|
It’s not working with me on Az DevOps even when specifying both flags TRIVY_CACHE_DIR= + --download-db-only. |
|
@bryanhermass @kapi-caz does anyone bypassed the error on az devops ? how ? I tried as mentioned adding two flags but still see the errors |
|
@karl18 yes, I managed to use cache in azure devops pipelines and it solves the issue (at least in the meantime).
|
|
But when you need to update your database the issue keep happen. I use github actions here and the solution that I had to use was cache for the db and config file changing the repository to new one that they sent here and the issues is happens much less now. |
|
I tried all of the suggestions above. Running on Gitlab, enabled artifacts for the I am scanning ~6-7 images in seperate pipelines so it makes sense that I am hitting some rate-limit, but I cannot figure out how I am supposed to be configuring this cache to work properly... I think the real issue is that Am I missing a CLI flag or something? |
|
Update: I ended up cloning the docker images to a local docker registry and using that in my CI/CD pipelines and I was able to resolve the issue. The problem is the cloning of the To solve it I used skopeo to copy the image (since it is OCI docker cannot copy it) into our artifactory instance, and then pointed my pipeline to my company's artifactory instance. This is my pipeline: container_scan:
stage: containerize
# https://aquasecurity.github.io/trivy/v0.18.3/usage/image/
image:
name: docker.io/aquasec/trivy:latest
entrypoint: [ "" ]
variables:
# No need to clone the repo
GIT_STRATEGY: none
# Variables to authenticate to our private registry
TRIVY_USERNAME: "$JFROG_CI_USER"
TRIVY_PASSWORD: "$JFROG_CI_PASSWORD"
TRIVY_AUTH_URL: "$ARTIFACTORY_REGISTRY"
# Misc.
TRIVY_NO_PROGRESS: "true"
TRIVY_CACHE_DIR: ".trivycache/"
# Configure Trivy to pull the database images from our local private images instead of ghcr.io
TRIVY_DB_REPOSITORY: artifactory.mydomain.com/docker-mirror/aquasecurity/trivy-db:2
TRIVY_JAVA_DB_REPOSITORY: artifactory.mydomain.com/docker-mirror/aquasecurity/trivy-java-db:1
# The image I want scanned
FULL_IMAGE_NAME: ${ARTIFACTORY_REGISTRY}/${CONTAINER_NAME}:${CI_COMMIT_TAG:-$CI_COMMIT_SHA}
script:
# Dump version.
- trivy --version
# Clear out the previous scan history from the cache
- time trivy clean --scan-cache
# update databases
- time trivy image --download-db-only
- time trivy image --download-java-db-only
# Builds report and puts it in the default workdir $CI_PROJECT_DIR, so `artifacts:` can take it from there
- time trivy image --exit-code 0 --format template --template "@/contrib/gitlab.tpl" --output "$CI_PROJECT_DIR/gl-container-scanning-report.json" "$FULL_IMAGE_NAME"
# Prints full report to the console
- time trivy image --exit-code 0 "$FULL_IMAGE_NAME"
# Fail on critical vulnerabilities
- time trivy image --exit-code 1 --severity CRITICAL "$FULL_IMAGE_NAME"
cache:
key: trivy-cache
paths:
- .trivycache/
artifacts:
when: always
reports:
container_scanning: gl-container-scanning-report.jsonHopefully this helps some future souls. Though it's a bit rough for people who don't have any way to mirror docker images to some local cache/registry...
Turns out this was happening because I have 10 CI/CD runners and the trivy cache only gets placed onto individual runners. So the trivy cache was runner dependent, and getting a different CI/CD runner would caus the image to get pulled again. Something to keep in mind if you are using multiple runners... |
|
Anyone trying to run it locally as part of pre-commit checks when pushing the code upstream? default_stages:
- pre-commit
repos:
- repo: local
hooks:
- id: trivy-fs
stages:
- pre-push
name: trivy-fs
language: docker_image
entry: -v /tmp:/.cache aquasec/trivy:latest fs -c .config/trivy.yaml .
pass_filenames: false |
|
We found that |
|
It works like a charm. I don't find any rate limits on mirror.gcr.io. |
|
One concern is that the old artifact is cached for a long time, and clients cannot get the latest DBs. |
|
From my experiment, the cached artifact seems invalidated after 1 hour. I'm not sure exactly 1 hour, but it doesn't look like 1 day or 1 week. |
|
I consider a delay of up to one hour in updating the DBs to be acceptable. I'd set |
|
Tried @knqyf263 's answer and it worked for the trivy db but was still getting request errors for the Java DB, tried with AWS ECR gallery and that worked. |
|
looking good so far, thanks! |
|
i get this: 2024-11-19T10:47:01Z ERROR [vulndb] Failed to download artifact repo="ghcr.io/aquasecurity/trivy-db:2" err="oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:d8f527b0dde6241ad3cfa63f82e6b0a8c9a64ba55571a06063dad700b3a15390: TOOMANYREQUESTS: retry-after: 426.501µs, allowed: 44000/minute" after flipping to |
|
for trivy-action check this: aquasecurity/trivy-action#389 (comment) |
|
Currrently getting an error, even with the new GCR mirror on Trivy |
|
Is it still persisting? |
|
It started working again after a little while. |
|
Can confirm, it fails sometimes with BLOB_UNKNOWN as @baksetercx wrote. Much better than with ghcr but still fails sometimes. |
and others
I consider a delay of up to one hour in updating the DBs to be acceptable. I'd set
mirror.gcr.ioas primary and setghcr.ioas secondary.