OWASP gradle check failed

[emartynov]

Describe the bug
We got a crash on CI.

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
	Failed to request component-reports: null
	at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:699)
	at org.owasp.dependencycheck.Engine$analyzeDependencies$3.call(Unknown Source)
	at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:89)

** Version of dependency-check used **
The problem occurs using version 5.1.0 gradle plugin.

** Log file **
Full stacktrace is here.

Nothing special but here is also build log

To Reproduce
No steps to reproduce. Looks like some race condition or gradle cache issues.

Expected behavior
OWASP report doesn't crash.

Additional context
It is an android project that uses AGP 3.4.2 and Gralde 5.5.1. It is a multimodule project and we use gradle cache.

[gabrielbieules]

Hi,

I have the same issue using 5.2.0. It does not occur using 5.0.0 but I cannot use it as it got that other issue #2073.

[jeremylong]

I have added additional logging to try and figure out what is going on - this will be included in the next release (point release hopefully this weekend).

[wraschke]

I'm wondering if this is a somewhat different symptom of same underlying cause?

Caused by: java.net.ConnectException: Server Error (500) at net.ossindex.common.request.OssIndexHttpClient.performPostRequest(OssIndexHttpClient.java:122) at net.ossindex.common.request.PackageRequest.run(PackageRequest.java:121) at net.ossindex.gradle.audit.DependencyAuditor.runAudit(DependencyAuditor.java:69) ... 157 more

The source for that appears to be at https://github.com/OSSIndex/java-api/blob/master/src/main/java/net/ossindex/common/request/OssIndexHttpClient.java (the plugin version we're using is not the most current and we're testing with the latest version right now).

I am using the OSSIndex Gradle plugin in my build when I see that. It started happening around July 28.

I am not seeing any other information (besides the rest of the long exception stack that I don't think you'd find meaningful), and definitely not any info about why the server returns 500. I'm presuming the POST request is being made to https://ossindex.sonatype.org/v2.0/<uri>.

[jeremylong]

Any chance someone facing this issue can provide the an example project that causes this behavior?

[emartynov]

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
	Failed to request component-reports
	at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:705)
	at org.owasp.dependencycheck.Engine$analyzeDependencies$4.call(Unknown Source)
	at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:89)
	... 182 more

[jeremylong]

We need a specific example project or dependency that demonstrates this issue to try to resolve. It does appear that this affects golang projects. However, we need additional information to help resolve this specific case with the android project.

[emartynov]

This is not a permanent error. It happens from time to time. It might be also hard to create a test project for this. We have 60+ modules in the project now that could be the key factor with concurrency.

[emartynov]

If we could reduce the number of nested exceptions level then we could probably find the root cause exception. Right now gradle with --stacktrace option doesn't print it whole in the log.

[emartynov]

It happens almost every night build for us. Morning job re-run just pass.

How to troubleshoot? Would a retry mechanism help?

This is frustrating.

[emartynov]

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
	Failed to request component-reports
	at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:705)
	at org.owasp.dependencycheck.Engine$analyzeDependencies$3.call(Unknown Source)
	at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:89)
	... 89 more

... 89 more how to make it shorter or how to see that full stack trace?

[jeremylong]

Can you run this with the debug logging enabled and look for the entries:
oss index adding package...

[jdillon]

@wraschke

The source for that appears to be at https://github.com/OSSIndex/java-api/blob/master/src/main/java/net/ossindex/common/request/OssIndexHttpClient.java (the plugin version we're using is not the most current and we're testing with the latest version right now).

This is the OLD client and should not have anything to do with the integration ODC has for OSS Index. The client impl is here:

https://github.com/sonatype/ossindex-public/tree/master/client/src/main/java/org/sonatype/ossindex/service/client

and ODC integration is here:

https://github.com/jeremylong/DependencyCheck/blob/master/core/src/main/java/org/owasp/dependencycheck/analyzer/OssIndexAnalyzer.java
https://github.com/jeremylong/DependencyCheck/tree/master/core/src/main/java/org/owasp/dependencycheck/data/ossindex

[jdillon]

@emartynov is there any more detailed logs from the ODC usage? The partial stack-trace here is not informative and chances are a full version of that stack wouldn't be any more helpful. What would be helpful would be the equivalent of the dependency-check.sh --log <file> which may have some more details before the exception was thrown as to what was going on that might have caused a failure to fetch the component-report. I have never personally used the ODC gradle plugin so i'm not sure what it supports in terms of logging.

A rough guess based on what was said regarding "happens from time to time" and "almost every night build for us. Morning job re-run just pass" though has me wondering if rate-limiting and/or some other environmental factor is causing this fail.

I am happy to look closer if more specifics can be provided so that I can see the failure and reproduce it. But presently there is not enough information in this issue to do so.

[emartynov]

People, finally I tried to collect some logging - I can not, I have to get more support from our SRE team.

Adding --debug makes our build longer 45 mins timeout. I can not imagine how huge is the log file will be.

Any other suggestion until I will ask SRE team to lift the timeout (I don't know what would be sage one) and I will have xx GB log file to analyse and pass by?

[emartynov]

One more error from today:

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
	Unable to connect to the database - if this error persists it may be due to a corrupt database. Consider running `purge` to delete the existing database

[emartynov]

And one more today:

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
	Unable to obtain an exclusive lock on the H2 database to perform updates
	No documents exist

[jeremylong]

These last two reported errors have nothing to do with the original component-report errors and indicate that the H2 database may have been corrupted. As the error above states - if the error persists consider running the purge command.

[emartynov]

Jeremy, sorry for posting all findings in this thread. But I just hopeless the problem is becoming clear. What purge command should I run? Can plugin do it by itself and forcing refetching of the data later?

[jeremylong]

gradle dependencyCheckPurge will remove the local H2 database. See https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration-purge.html

[emartynov]

It is happening at one of the CI machines. So going and running this command there is not really convenient, I event don't know (without investigation) which machine is affected.

[emartynov]

People, this night we saw another cause exception:

Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: One or more exceptions occurred during analysis:
	Unable to download meta file: https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta
	No documents exist

[ghost]

Yes, see #2222.

[LarsKaulen]

Hello,
we regulary (at least 3-4 times a week) got the same crash on CI as described in this ticket with different versions of the dependency check tool, but without being able to detect a pattern or provide a test project where it happens deterministically.

Around 3 weeks ago we changed our build process to use a local mirror within our company network, using a nightly cron job and curl to download the necessary gzip and meta files from the NIST website. Since this change, not a single CI job failed due to this error message anymore, without us changing anything else regarding the dependency check plugin.

So the error seems to be connected to issues/downtimes of the NIST website and switching to (a more reliable) local mirror "solved" it for us.

[jeremylong]

Closing this thread as it has been a mashup of different issues. If someone has a specific issue please create a new ticket.