Improving Performance - Maven Artifact Caching

[goneall]

Question

I'm running some very large Trivy scans with several hundred Maven POM files being analyzed.

It is taking over 11 hours to complete.

From analysis, almost all of the time is waiting for Maven artifact downloads.

I was wondering if I could configure Trivy to use a local artifact repository which would essentially be a very large cache or mirror of Maven to speed things up. Something like Sonatype Nexus.

Is it possible to configure Trivy to utilize such a repository?

I did find an old article on implementing a caching forward proxy to solve a slightly different problem. This is something I could try if you think it would help.

Target

Filesystem

Scanner

License

Output Format

SPDX

Mode

Standalone

Operating System

Ubuntu

Version

No response

[knqyf263]

@DmitriyLewen Can you please take a look? You already have many tasks, so anytime you have a minute is fine.

[DmitriyLewen]

I will try to check that on this weak.

[DmitriyLewen]

Hello @goneall
I looked at your case.

I did find an old article on implementing a caching forward proxy to solve a slightly different problem

That instruction was for jar files.
No, it doesn't work now because we use trivy-java-db.

About pom.xml files:
Trivy doesn't have an env (like MAVEN_CENTRAL_URL) to set repositories to look for pom files.
Trivy only checks repositories from pom.xml files (so you can temporarily add your local repository to the scanned pom file (but I'm not sure if that's ok for you)).
I think the best solution is to use the local repository from the settings file. We already have #7807 with a request to add this functionality.

[goneall]

Just a quick update - I've been running a couple of experiments. I've added some logging to narrow down which POM file fetches are slow and experimented with forcing the fetching of the POM files to go through a Nexus cache.

If I limit the fetches to only use Maven Central and go through Nexus, the time goes from around 11 hours to approximately 90 minutes. I did verify that if I change the following line to go through the Nexus cache, it works and the cache utilized:

trivy/pkg/dependency/parser/java/pom/parse.go

Line 30 in 02ebb4c

centralURL = "https://repo.maven.apache.org/maven2/"

Watching the log files real-time, it looks like the Apache staging repo is rate limiting the responses. Once I identify the slow repos, I'll see if I can configure them to go through the Nexus cache. If that speeds things up significantly, I'll do a bit more looking at the code to see if there is an easy way to configure the Nexus cache.

[goneall]

I finished my performance experiments and have a solution that I've hacked together on my fork of Trivy: https://github.com/goneall/trivy/blob/usenexus/pkg/dependency/parser/java/pom/parse.go

The biggest performance gain was not repeatedly "trying" to fetch a POM file from a repository which did not respond (either an error or timeout). I didn't implement this directly in the Trivy code, but redirected the repository requests through the Nexus server which has a feature to respond immediately if a previous attempted fetch of the upstream failed. This is something that could be implemented directly in Trivy without having to implement the caching / proxy implementation.

The second biggest performance gain was going through the Nexus server which caches the POM file locally. The gains from this was not nearly as large as the gains from avoiding multiple bad fetches, but it was more than measurable.

My "hack" was to change the hard coded URL for the Maven repo to use the Nexus server. I also implemented a map of proxy URLs and when fetching from a remote URL matching the key, it fetched it from the Nexus server. I had to also configure the Nexus server.

If we wanted to update Trivy to such an approach, you can fetch the globally configured proxy server from the Maven settings configuration file (same approach used by Maven).

My recommendation, based on the effort and return, would be to implement a feature which doesn't do multiple bad calls. The caching would be nice, but it would be more difficult and less impact.

If you want to test this out, the bad repo URL I found in more than one LF projects are:

https://repository.apache.org/content/repositories/releases/

and

https://repository.apache.org/content/repositories/staging/

One example project is the opendaylight project. I think (but I'm not completely sure) the yangtools repo can reproduce the problem.

If I get some time, I may change my hack to a more supportable version which just skips the bad repos as recommended - but I have a few higher priority non Trivy related performance problems to chase down.

[DmitriyLewen]

hmm...
I like the idea of ​​"disabling" broken repositories (we can only leave 404 errors).
At first I thought we need a switch (I think TRIVY_EXPERIMENTAL_* will be enough for now) so that the user will see that Trivy couldn't access the repository, but then I realized that the user either looks at the logs or not (I mean that if the user doesn't look at the logs - he will miss the error and it doesn't matter how many errors there are - one or for each artifact).

I can think of another case - what if the repository "turns off" for a short time
for example, the scan lasts 30 minutes. After 10 minutes, the repository stops responding for 1-2 minutes.
should we exclude this repository?
I think this is a rare case, so I suggest the following logic:
if the repository did not return a response on the first attempt - skip this repository, otherwise continue trying (thinking that this is a temporary failure)
although this may be redundant.

[goneall]

You could vary the policy based on the response from the server - e.g. have a different policy for 502 responses vs. timeouts.

I wish I would have recorded the response code for the Apache calls.

Having some kind of timeout on the exclusion seems reasonable. This is the approach Nexus takes - they exclude the repo for one day then try again. A timeout of 1 hour would not impact the performance of my scans.

Another note - in my experiments, there was never any warnings logged. It looks like the code currently just moves onto the next repository without logging an error. I'm thinking you only need to report an error if you can't resolve the dependency at all.

[DmitriyLewen]

I'm thinking you only need to report an error if you can't resolve the dependency at all.

it make sense 👍