Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538

dmitry-weirdo · 2022-05-25T08:32:04Z

Starting today (25.05.2022), multiple errors started to fail on each execution. No plugin version changed (was and remains the 7.1.0 version).

Error examples (an error is failing for each of the dependencies):

[WARNING] An error occurred while analyzing '/tmp/dctemp2224d3d1-bfef-48de-b1f5-0e97d0983265/check9478530389331146964tmp/208/META-INF/resources/webjars/shepherd.js/8.2.3/.jsdoc.js' (Sonatype OSS Index Analyzer).

[ERROR] 	AnalysisException: OSS Index rate limit exceeded
[ERROR] 		caused by TransportException: Unexpected response; status: 429

There are actually the NPEs that are probably the root cause:

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:7.1.0:check (default-cli) on project ins-app: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR] 	AnalysisException: Failed to request component-reports
[ERROR] 		caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR] 	AnalysisException: Failed to request component-reports
[ERROR] 		caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR] 	AnalysisException: Failed to request component-reports
[ERROR] 		caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR] 	AnalysisException: Failed to request component-reports
[ERROR] 		caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR] 	AnalysisException: Failed to request component-reports

etc.

The text was updated successfully, but these errors were encountered:

flash-me · 2022-05-25T08:53:02Z

We also have these issues. In our pipelines and can also be reproduced locally.

There was a change recently, maybe related to that?
https://ossindex.sonatype.org/updates-notice

Also tried out the same POST (with the same data) calls on their API manually: https://ossindex.sonatype.org/rest
Works fine

Also the curl to the API from a local machine works fine.

cheers
flash ⚡

ankurga · 2022-05-25T08:53:04Z

Happening for me as well. Also, all the plugin versions have started giving false positives for multiple jar files:

spring-boot-2.6.8.jar (pkg:maven/org.springframework.boot/spring-boot@2.6.8, cpe:2.3:a:vmware:spring_boot:2.6.8:*:*:*:*:*:*:*, cpe:2.3:a:vmware:spring_framework:2.6.8:*:*:*:*:*:*:*) : CVE-2013-4152, CVE-2013-7315, CVE-2014-0054, CVE-2016-1000027, CVE-2022-22965, CVE-2022-22968
spring-core-5.3.20.jar (pkg:maven/org.springframework/spring-core@5.3.20, cpe:2.3:a:pivotal_software:spring_framework:5.3.20:*:*:*:*:*:*:*, cpe:2.3:a:springsource:spring_framework:5.3.20:*:*:*:*:*:*:*, cpe:2.3:a:vmware:spring_framework:5.3.20:*:*:*:*:*:*:*, cpe:2.3:a:vmware:springsource_spring_framework:5.3.20:*:*:*:*:*:*:*) : CVE-2016-1000027

This has now blocked all our maven builds. Can we please have a solution to this?

danielbraeutigam · 2022-05-25T08:57:09Z

@ankurga The NVD database was updated, see e.g. spring-projects/spring-framework#24434 (comment) - you have to suppress it by yourself.

ankurga · 2022-05-25T09:00:55Z

@danielbraeutigam thanks, but we have many projects dependent on that. So, we need to update for all of them?

danielbraeutigam · 2022-05-25T09:07:18Z

@danielbraeutigam thanks, but we have many projects dependent on that. So, we need to update for all of them?

No, you just have to add it to your suppression file as described in https://jeremylong.github.io/DependencyCheck/general/suppression.html

mlemmens · 2022-05-25T09:12:05Z

We are experiencing almost the same issue, only the check fails on a HTTP 500 or NullPointerException from OSS Index.

HTTP 500:

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:6.5.3:aggregate (default-cli) on project xxx: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR] AnalysisException: Failed to request component-reports
[ERROR] caused by TransportException: Unexpected response; status: 500

NullPointerException:

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:6.5.3:aggregate (default-cli) on project xxx: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR] AnalysisException: Failed to request component-reports
[ERROR] caused by NullPointerException: null

I think this is also related to the recent improvements Sonatype applied to OSS Index: https://ossindex.sonatype.org/updates-notice.

May 23: OSS Index will start using the new data pipeline and you will see the improvements listed above.

jwwallin · 2022-05-25T09:12:15Z

We are also experiencing this same issue. I did some investigating and this seems to be caused by handling of the response failing. That in turn results in the same request being sent on every dependency until rate limit is reached.

[DEBUG] Error requesting component reports
java.lang.NullPointerException
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryKey (DirectoryCache.java:149)
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryFile (DirectoryCache.java:157)
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.putAll (DirectoryCache.java:134)
    at org.sonatype.ossindex.service.client.internal.OssindexClientImpl.requestComponentReports (OssindexClientImpl.java:171)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.requestReports (OssIndexAnalyzer.java:219)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency (OssIndexAnalyzer.java:134)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:131)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
    at java.lang.Thread.run (Thread.java:829)
[WARNING] An error occurred while analyzing '/home/user/.m2/repository/com/amazonaws/aws-java-sdk-mediaconnect/1.11.863/aws-java-sdk-mediaconnect-1.11.863.jar' (Sonatype OSS Index Analyzer).
[DEBUG] 
org.owasp.dependencycheck.analyzer.exception.AnalysisException: Failed to request component-reports
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency (OssIndexAnalyzer.java:157)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:131)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
    at java.lang.Thread.run (Thread.java:829)
Caused by: java.lang.NullPointerException
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryKey (DirectoryCache.java:149)
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryFile (DirectoryCache.java:157)
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.putAll (DirectoryCache.java:134)
    at org.sonatype.ossindex.service.client.internal.OssindexClientImpl.requestComponentReports (OssindexClientImpl.java:171)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.requestReports (OssIndexAnalyzer.java:219)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency (OssIndexAnalyzer.java:134)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:131)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
    at java.lang.Thread.run (Thread.java:829)

There is a large amount of these failed requests until eventually only rate-limit is logged.

flash-me · 2022-05-25T09:14:08Z

@ankurga The NVD database was updated, see e.g. spring-projects/spring-framework#24434 (comment) - you have to suppress it by yourself.

I'm not sure if suppression is related to this. These are not "false positives", they are errors:

cheers
flash ⚡

achintSatsangi · 2022-05-25T09:16:58Z

@danielbraeutigam @ankurga : The problem you are discussing is not related to the issue reported so lets take it separately. As of now all projects using owasp-dependency-check plugin with version 6.X.X and 7.1.0 don't build. Something has recently changed at SONATYPE. I can reproduce the same locally as well on our CI platforms.

[ERROR] 	AnalysisException: Failed to request component-reports
[ERROR] 		caused by NullPointerException: null
[ERROR] 	AnalysisException: OSS Index rate limit exceeded
[ERROR] 		caused by TransportException: Unexpected response; status: 429

On debug we see:

[DEBUG] Connecting to: https://ossindex.sonatype.org/api/v3/component-report
[DEBUG] Error requesting component reports
java.lang.NullPointerException
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryKey (DirectoryCache.java:149)
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.entryFile (DirectoryCache.java:157)
    at org.sonatype.ossindex.service.client.cache.DirectoryCache.putAll (DirectoryCache.java:134)

The URL https://ossindex.sonatype.org/api/v3/component-report is throwing 429 (Rate limiting)

petergphillips · 2022-05-25T09:20:38Z

It looks like sonatype have started rate limiting more aggressively for anonymous accounts. One solution is to register for an account with sonatype and supply a username and password / api key e.g. for gradle users

dependencyCheck {
  analyzers.ossIndex.username = "<username>"
  analyzers.ossIndex.password = "<password>"
}

rsolci · 2022-05-25T09:21:28Z

@achintSatsangi https://ossindex.sonatype.org/updates-notice
They published an update notice regarding breaking changes

flash-me · 2022-05-25T09:23:06Z

It looks like sonatype have started rate limiting more aggressively for anonymous accounts. One solution is to register for an account with sonatype and supply a username and password / api key e.g. for gradle users
dependencyCheck {
  analyzers.ossIndex.username = "<username>"
  analyzers.ossIndex.password = "<password>"
}

I'm not sure if this is the real issue.
The first calls do not start with 429, but with NullPointerException, as already mentioned

Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538 (comment)
Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538 (comment)

cheers
flash ⚡

petergphillips · 2022-05-25T09:25:24Z

It looks like sonatype have started rate limiting more aggressively for anonymous accounts. One solution is to register for an account with sonatype and supply a username and password / api key e.g. for gradle users
dependencyCheck {
  analyzers.ossIndex.username = "<username>"
  analyzers.ossIndex.password = "<password>"
}
I'm not sure if this is the real issue. The first calls do not start with 429, but with NullPointerException, as already mentioned

Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538 (comment)

Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538 (comment)

cheers flash ⚡

Agreed, it didn't look like that to me either, since I was getting exactly the same errors as you. However I can confirm that for me supplying credentials to the oss index then meant that I stopped getting the null pointer exception and rate limited exceeded errors and the check then succeeded.

jwwallin · 2022-05-25T09:25:52Z

This problem looks to me like the Sonatype provided client isn't handling the responses correctly.

An issue should probably be raised there as-well.

sailro · 2022-05-25T09:29:22Z

When using dependency-check directly:

[INFO] Analysis Started
[INFO] Finished Archive Analyzer (0 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Jar Analyzer (0 seconds)
[INFO] Finished Central Analyzer (1 seconds)
[INFO] Finished Dependency Merging Analyzer (0 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Created CPE Index (0 seconds)
[INFO] Finished CPE Analyzer (1 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (0 seconds)
[WARN] An error occurred while analyzing '[redacted]' (Sonatype OSS Index Analyzer).
[INFO] Finished Sonatype OSS Index Analyzer (1 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
[INFO] Finished Dependency Bundling Analyzer (0 seconds)
[INFO] Analysis Complete (6 seconds)
[INFO] Writing report to: [redacted]
[ERROR] Failed to request component-reports

lzuyderhoff · 2022-05-25T09:41:08Z

I am experiencing the same issue since this morning with the plugin.
Note that from the same machine I could make a request the OSS rest API without issuing the 429 error using a rest client and this :
`POST https://ossindex.sonatype.org/api/v3/component-report
Content-Type: application/json

{
"coordinates": [
"pkg:maven/commons-beanutils/commons-beanutils@1.9.3"
]
}`
So I am thinking this is due to the recent update of the API

kashiuno · 2022-05-25T09:54:02Z

As I can see mvn dependency-check:update-only works for me in local environment, after that I don't have this issue.

lzuyderhoff · 2022-05-25T09:58:20Z

According to the doc https://ossindex.sonatype.org/doc/rest
"Use POST when requesting vulnerability reports for multiple components. There is a limit of 128 components per HTTP request."
Maybe the 128 limit is new and the plugin don't chunk the call.

Nriver · 2022-05-25T09:58:24Z

Is it possible to disable it? Or use a local nexus server instead?
It really slows down the scan process.

jwwallin · 2022-05-25T09:58:43Z

According to the doc https://ossindex.sonatype.org/doc/rest "Use POST when requesting vulnerability reports for multiple components. There is a limit of 128 components per HTTP request." Maybe the 128 limit is new and the plugin don't chunk the call.

Nope. That is not the issue here.

dmitry-weirdo · 2022-05-25T09:59:43Z

An additional thing: when I run the same check locally (not on GitLab), I am getting the following NPEs:

UPDATE: these NPEs are also present in the GitLab pipeline.

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:7.1.0:check (default-cli) on project ins-schema: One or more exceptions occurred during dependency-check analysis: One or more exceptions occ
urred during analysis:
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR]         AnalysisException: Failed to request component-reports
[ERROR]                 caused by NullPointerException: Cannot invoke "org.sonatype.goodies.packageurl.PackageUrl.toString()" because "coordinates" is null
[ERROR] -> [Help 1]

jwwallin · 2022-05-25T10:00:12Z

Is it possible to disable it? Or use a local nexus server instead? It really slows down the scan process.

You can disable the OSS Index Analyzer entirely. Check the docs for how.

lzuyderhoff · 2022-05-25T10:00:27Z

Is it possible to disable it? Or use a local nexus server instead? It really slows down the scan process.

Yes you can disable the analyzer using
<ossindexAnalyzerEnabled >false</ossindexAnalyzerEnabled>

petergphillips · 2022-05-25T10:02:47Z

I wouldn't recommend disabling the oss indexer though, we are currently getting four failures with the indexer enabled, but none with it disabled. I would instead recommend registering with oss index and setting a username and password / api key.

flash-me · 2022-05-25T10:03:52Z

I wouldn't recommend disabling the oss indexer though, we are currently getting four failures with the indexer enabled, but none with it disabled. I would instead recommend registering with oss index and setting a username and password / api key.

How to accomplish this with maven? (which properties to set / provide)

cheers
flash ⚡

garyc0221 · 2022-05-25T10:07:18Z

I wouldn't recommend disabling the oss indexer though, we are currently getting four failures with the indexer enabled, but none with it disabled. I would instead recommend registering with oss index and setting a username and password / api key.

Looks like currently you cannot register on ossindex.sonatype getting a 500 error

NorthernKgalagadi · 2022-05-25T10:08:10Z

I wouldn't recommend disabling the oss indexer though, we are currently getting four failures with the indexer enabled, but none with it disabled. I would instead recommend registering with oss index and setting a username and password / api key.

Looks like currently you cannot register on ossindex.sonatype getting a 500 error

registering works; you should receive an email with a confirmation link ....

flash-me · 2022-05-25T10:14:44Z

I wouldn't recommend disabling the oss indexer though, we are currently getting four failures with the indexer enabled, but none with it disabled. I would instead recommend registering with oss index and setting a username and password / api key.

How to accomplish this with maven? (which properties to set / provide)

cheers flash ⚡

got it working. Provide ossIndexServerId with a value to the id in the settings.xml file

e.g.
-DossIndexServerId=owasp-oss-index

and in your settings.xml

<settings>
  <servers>
    <server>
      <id>owasp-oss-index</id>
      <username>foo</username>
      <password>bla</password>
    </server>
  </servers>
</settings>

More info: https://jeremylong.github.io/DependencyCheck/dependency-check-maven/aggregate-mojo.html#ossIndexServerId

@petergphillips can confirm, that with a registered user account it works Thanks for the hints!

And yes, it seems like the rate limit to 128 modules per request is new

cheers
flash ⚡

NorthernKgalagadi · 2022-05-25T11:44:56Z

see #4535 (comment)

Also had to disable .net assembly analysis (through lombok bundled windows executables) explicitly to prevent analysis failure once OSS index analyzer disabled. For OSS index breaking changes see: jeremylong/DependencyCheck#4539 and probably next step to sign up Sonatype user: jeremylong/DependencyCheck#4538

echalone · 2022-05-25T11:52:49Z

We too have this issue since today. Registring with ossindex.sonatype.org and using that username/password with command line options --ossIndexUsername and --ossIndexPassword solved the issue.

ankurga · 2022-05-25T12:40:06Z

@ankurga The NVD database was updated, see e.g. spring-projects/spring-framework#24434 (comment) - you have to suppress it by yourself.

I'm not sure if suppression is related to this. These are not "false positives", they are errors:

cheers flash ⚡

But how come when I disable ossIndexAnalyzer it starts working then without any issue?

rvgiesen · 2022-05-25T14:02:40Z

If anyone comes here because their pipeline in azure devops is failing, just follow these two steps:

register a free account at https://ossindex.sonatype.org/
add your new account info to your pipeline yaml file in the inputs part, f.e.:

additionalArguments: |
--ossIndexUsername YOUREMAIL --ossIndexPassword YOURPASS

danielbraeutigam · 2022-05-25T14:31:08Z

Fix mentioned in #4535 (comment) worked for me now without changing any configuration.

lzuyderhoff · 2022-05-25T15:04:13Z

Same for me. Problem solved. Thanks.

carloreggiani · 2022-05-25T15:18:30Z

Any possibility to not use user and password in clear? Any token?

…

On Wed, May 25, 2022 at 4:02 PM rvgiesen ***@***.***> wrote: If anyone comes here because their pipeline in azure devops is failing, just follow these two steps: - register a free account at https://ossindex.sonatype.org/ - add your new account info to your pipeline yaml file in the inputs part, f.e.: additionalArguments: | --ossIndexUsername YOUREMAIL --ossIndexPassword YOURPASS — Reply to this email directly, view it on GitHub <#4538 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHXVHPTFSSJZDLFMCZNXJDVLYXI7ANCNFSM5W4JRWLA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Carlo Reggiani about.me/reggianicarlo [image: Carlo Reggiani on about.me] <http://about.me/reggianicarlo>

j-s-3 · 2022-05-25T15:25:08Z

Any possibility to not use user and password in clear? Any token?
…

OSSI supports generating an API Token https://ossindex.sonatype.org/doc/api-token

ctnelson1997 · 2022-05-25T16:24:10Z

People are looking for an immediate solution: set ossindexAnalyzerEnabled to false in your scans.

Then, work on registering a username and password to authenticate with OSS.

j-s-3 · 2022-05-25T16:51:07Z

@ctnelson1997 these issues should be resolved now

bergerst · 2022-05-25T18:31:54Z

Just so I understand correctly: The rate limiting issues are fixed on OSS Index so a user account is not needed anymore?

By the way, after I specified a user and password, a few [sonatype-xxx] popped up, which I've suppressed with <vulnerabilityName regex="true">^\[sonatype.*</vulnerabilityName>.

I assume they're only visible to logged in users, most don't have CVE numbers and quite a few are not fixed yet.

For example, https://ossindex.sonatype.org/component/pkg:maven/org.terracotta/offheap-store has the same vulnerability in all versions, https://ossindex.sonatype.org/vulnerability/sonatype-2020-0267.

Subhalakshmi1986 · 2022-05-26T02:31:20Z

We have been facing this issue too!.But the cause seems to be different.
AnalysisException: Failed to request component-reports
caused by SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
caused by ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
caused by CertPathValidatorException: validity check failed
caused by CertificateExpiredException: NotAfter: Thu May 26 09:59:59 AEST 2022

owasp gradle plugin version. - 7.1.0.1
What's the resolution for this issue

chadlwilson · 2022-05-26T02:52:58Z

@Subhalakshmi1986 Looks like it expired a few hours ago, so a new issue.

Subhalakshmi1986 · 2022-05-26T05:18:22Z

We have been facing this issue too!.But the cause seems to be different. AnalysisException: Failed to request component-reports caused by SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed caused by ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed caused by CertPathValidatorException: validity check failed caused by CertificateExpiredException: NotAfter: Thu May 26 09:59:59 AEST 2022

owasp gradle plugin version. - 7.1.0.1 What's the resolution for this issue

The issue is resolved now.We don't face this issue now.

j-s-3 · 2022-05-26T10:43:56Z

Just so I understand correctly: The rate limiting issues are fixed on OSS Index so a user account is not needed anymore?

By the way, after I specified a user and password, a few [sonatype-xxx] popped up, which I've suppressed with <vulnerabilityName regex="true">^\[sonatype.*</vulnerabilityName>.

I assume they're only visible to logged in users, most don't have CVE numbers and quite a few are not fixed yet.

For example, https://ossindex.sonatype.org/component/pkg:maven/org.terracotta/offheap-store has the same vulnerability in all versions, https://ossindex.sonatype.org/vulnerability/sonatype-2020-0267.

I highly recommend making authenticated requests so that you receive the higher rate limit but also so that you get the SONATYPE-* vulnerabilities. These are vulnerabilities discovered by our large team of human researchers. Most often these are because the public sources are incorrect (they report something has been fixed but our testing shows that it hasn't actually been fixed).

aikebah · 2022-06-14T21:02:27Z

@jeremylong @jlstephens89 I think this one can now be considered resolved with the updates/fixes made in the OSSINDEX? Or are there still pending issues related to this ticket?

Redirts · 2022-10-18T17:39:43Z

I am facing this issue now, seems to be same behaviour:

...
[DependencyCheck] [WARN] An error occurred while analyzing '/var/jenkins_home/workspace/xxx/node_modules/@blueprintjs/core/lib/cjs/components/panel-stack2/panelStack2.js' (Sonatype OSS Index Analyzer).
....
[DependencyCheck] [ERROR] Failed to request component-reports
[DependencyCheck] [ERROR] OSS Index rate limit exceeded
....

I tried providing a username and password to authenticate with OSS but the problem persists.

aikebah · 2022-10-18T22:15:29Z

@Redirts
See also #4513 (comment) and #4513 (comment): check that your Jenkins setup uses a shared persistent dataDirectory for DependencyCheck so that subsequent runs of your build as well as runs for other builds can reuse already recently requested results from OSSIndex and only needs to access OSSIndex for 'new/unknown' libraries and libraries for which the cached data has become stale.
Otherwise for a larger build systems you still have a reasonable risk to run into rate limiting at OSSIndex, even for the (unknown) rate limits they have set for authenticated users.

xtermi2 · 2022-12-21T14:51:55Z

I am facing this issue now, seems to be same behaviour:

...

[DependencyCheck] [WARN] An error occurred while analyzing '/var/jenkins_home/workspace/xxx/node_modules/@blueprintjs/core/lib/cjs/components/panel-stack2/panelStack2.js' (Sonatype OSS Index Analyzer).
....
[DependencyCheck] [ERROR] Failed to request component-reports
[DependencyCheck] [ERROR] OSS Index rate limit exceeded
....
I tried providing a username and password to authenticate with OSS but the problem persists.

We also get rate limit errors even when setting a ossIndex username/password :( This started about 2 week ago. bevore everything was fine. We use org.owasp.dependencycheck gradle plugin in version 7.4.1 / 7.3.2! Other older versions I have not tested, but it was working with 7.3.2 ~3 weeks before.

I get AnalysisException: OSS Index rate limit exceeded caused by TransportException: Unexpected response; status: 429 723 times and AnalysisException: Failed to request component-reports caused by TransportException: Unexpected response; status: 500 238 times.

aikebah · 2022-12-21T15:27:58Z

@xtermi2 For the http 500 cases it would be worthwhile to find out for which of your dependencies that happens as it apears to indicate an issue at OSSINDEX server-side (there was recently a report for an old version of Jgroups that triggers a 500 at OSSINDEX API (#5154) which has already been raised with Sonatype, but maybe your cases concern other libraries that have a server-side issue on OSSINDEX side)

Regarding the 429-cases like mentioned before in this thread: double-check your caching configuration on the build server. ODC uses OSSINDEX's apiclient that has built-in caching features to avoid flooding their API, but it requires you to ensure that different invocations of ODC use the same datadirectory (which among others contains the OSSINDEX cached data and the CVE database)

xtermi2 · 2022-12-21T16:04:35Z

@aikebah

The HTTP 500 occurs 238 times... how can i get the details which exception belongs to which dependency? The gradle plugin just logs e.g An error occurred while analyzing '/cache/.gradle/caches/modules-2/files-2.1/javax.validation/validation-api/1.1.0.Final/8613ae82954779d518631e05daa73a6a954817d5/validation-api-1.1.0.Final.jar' (Sonatype OSS Index Analyzer). But at this point I don't know the details (HTTP 500 or 429) and later in the collected exceptions I don't see the dependency causing the exception.
we cache the CVEs with a local Database mirror and update it once a day (in the data{} gradle configuration). Thought there is also persistet the oss data from sonatype. But thanks, I will try it out.

aikebah · 2022-12-21T18:00:57Z

Don't know the amount of logging that gradle plugin can do. The Jgroups case I referred to I was able to quickly reproduce and see all the detail by enabling debug of the maven plugin (mvn ... -X which, among others logs the full detail of the OSSIndex library calls towards their API. I would expect the same to happen for you when you enable gradle debug logging (capture the verbose output and search for the reports on a 500 error. Shortly before those I expect a log of the POST payload towards OSSINDEX).
In the maven plugin these debug log statements look like:

[DEBUG] OSS Index Analyzer submitting: [pkg:maven/org.jgroups/jgroups@2.6.21.Final]
[DEBUG] Requesting 1 component-reports
[DEBUG] Requesting 1 un-cached component-reports
[DEBUG] POST https://ossindex.sonatype.org/api/v3/component-report; payload: {"coordinates":["pkg:maven/org.jgroups/jgroups@2.6.21.Final"]} (application/vnd.ossindex.component-report-request.v1+json); accept: application/vnd.ossindex.component-report.v1+json
[DEBUG] Connecting to: https://ossindex.sonatype.org/api/v3/component-report
[DEBUG] Error requesting component reports
org.sonatype.ossindex.service.client.transport.Transport$TransportException: Unexpected response; status: 500
    at org.sonatype.ossindex.service.client.transport.HttpUrlConnectionTransport.post (HttpUrlConnectionTransport.java:106)
    at org.sonatype.ossindex.service.client.internal.OssindexClientImpl.doRequestComponentReports (OssindexClientImpl.java:204)
    at org.sonatype.ossindex.service.client.internal.OssindexClientImpl.requestComponentReports (OssindexClientImpl.java:170)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.requestReports (OssIndexAnalyzer.java:217)
    at org.owasp.dependencycheck.analyzer.OssIndexAnalyzer.analyzeDependency (OssIndexAnalyzer.java:134)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:131)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
    at java.lang.Thread.run (Thread.java:833)

jeremylong · 2022-12-24T13:29:38Z

@aikebah I wonder if we should add rate limiting to the client - to slow down the requests a bit to avoid this issue? I've been experimenting with options for the NVD API when we convert (see vuln-tools/.../RateLimitedClient.java and a slightly modified version at cdn-jscan/.../RateLimitedClient.java). Thoughts?

aikebah · 2022-12-24T13:45:30Z

My gut feel is that something's weird in Node analysis... I saw dependency counts in the tenthousands range when I trialled a dummy project created from https://github.com/facebook/create-react-app

NPM lists package counts in the thousands, but ODC counts in the tenthousands (somewhere around 26k, with 16k unique)

I can imagine that such huge counts if they leak to the OSSindex inquiry will overload in any case (with 128 deps per request you'd need an excessive amount of calls).

In general rate-limited client is a good idea (although I don't know if we could plug it into their API-client to perform the requests), but my gut feel is that we should first plug the dependencycheck-hole, because I cannot reasonably believe that a react app has 26k depended-on NPM packages

jeremylong · 2022-12-24T14:27:33Z

Just your typical node developer going to work:

(https://photographyofchina.com/author/alain-delorme)

I agree there is likely something wrong (I did "recently" fix a duplication problem in npm - see #3983). Also, the recent fixes around the node_modules directory might have corrected some of the problem... But there is likely still an underlying issue - but some npm projects due have 10k+ dependencies.

jeremylong · 2022-12-24T15:27:05Z

Just had a thought - is the retireJS analyzer causing this issue on npm projects?

…

On Sat, Dec 24, 2022, 8:45 AM Hans Aikema ***@***.***> wrote: My gut feel is that something's weird in Node analysis... I saw dependency counts in the tenthousands range when I trialled a dummy project created from https://github.com/facebook/create-react-app NPM lists package counts in the thousands, but ODC counts in the tenthousands (somewhere around 26k, with 16k unique) I can imagine that such huge counts if they leak to the OSSindex inquiry will overload in any case (with 128 deps per request you'd need an excessive amount of calls). In general rate-limited client is a good idea (although I don't know if we could plug it into their API-client to perform the requests), but my gut feel is that we should first plug the dependencycheck-hole, because I cannot reasonably believe that a react app has 26k depended-on NPM packages — Reply to this email directly, view it on GitHub <#4538 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGSVQVPIGEHGBIPYTJE3W3WO35ALANCNFSM5W4JRWLA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

msillence · 2023-01-04T14:13:44Z

~~I've tried registering an account and I still get the same 500/429 errors~~
~~The plugin continues making a lot of requests all of which seem to fail~~
~~It doesn't seem to make any obvious progress running it again~~

~~Could the plugin be less aggressive in how many requests it's making a second or retry/back off when there's an error?~~

~~Or is it simply that this is all too resource consuming and the plugin now needs commercial contract for a project with many dependencies?~~

msillence · 2023-01-04T15:22:13Z

I've just tested this again and I can reproduce it with simply the request

POST https://ossindex.sonatype.org/api/v3/component-report

{"coordinates":["pkg:maven/org.codehaus.xfire/xfire-core@1.2.6"]}

1.2.7 is OK

https://issues.sonatype.org/browse/OSSIPUB-37

So I think there are 2 issues,
1/ oss index returns 500 for this artifact
2/ the plugin retries hard when it gets a 500 and results in a blocked IP address

j-s-3 · 2023-01-05T18:34:32Z

I've just tested this again and I can reproduce it with simply the request

POST https://ossindex.sonatype.org/api/v3/component-report
{"coordinates":["pkg:maven/org.codehaus.xfire/xfire-core@1.2.6"]}
1.2.7 is OK

https://issues.sonatype.org/browse/OSSIPUB-37

So I think there are 2 issues, 1/ oss index returns 500 for this artifact 2/ the plugin retries hard when it gets a 500 and results in a blocked IP address

@msillence I've reproduced your issue and have passed it on to the team that maintains OSSI at Sonatype. Thanks for reporting this.

dmitry-weirdo added the bug label May 25, 2022

jwwallin mentioned this issue May 25, 2022

Client as used by dependency-check-maven fails with NullPointerException sonatype/ossindex-public#35

Closed

Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538

Multiple "AnalysisException: OSS Index rate limit exceeded" errors on plugin execution #4538

Comments

dmitry-weirdo commented May 25, 2022 • edited

flash-me commented May 25, 2022 • edited

ankurga commented May 25, 2022

danielbraeutigam commented May 25, 2022 • edited

ankurga commented May 25, 2022

danielbraeutigam commented May 25, 2022 • edited

mlemmens commented May 25, 2022 • edited

jwwallin commented May 25, 2022

flash-me commented May 25, 2022

achintSatsangi commented May 25, 2022 • edited

petergphillips commented May 25, 2022

rsolci commented May 25, 2022

flash-me commented May 25, 2022

petergphillips commented May 25, 2022

jwwallin commented May 25, 2022 • edited

sailro commented May 25, 2022 • edited

lzuyderhoff commented May 25, 2022

kashiuno commented May 25, 2022

lzuyderhoff commented May 25, 2022

Nriver commented May 25, 2022

jwwallin commented May 25, 2022

dmitry-weirdo commented May 25, 2022 • edited

jwwallin commented May 25, 2022

lzuyderhoff commented May 25, 2022 • edited

petergphillips commented May 25, 2022

flash-me commented May 25, 2022

garyc0221 commented May 25, 2022

NorthernKgalagadi commented May 25, 2022

flash-me commented May 25, 2022 • edited

NorthernKgalagadi commented May 25, 2022

echalone commented May 25, 2022

ankurga commented May 25, 2022

rvgiesen commented May 25, 2022

danielbraeutigam commented May 25, 2022

lzuyderhoff commented May 25, 2022

carloreggiani commented May 25, 2022 via email

j-s-3 commented May 25, 2022

ctnelson1997 commented May 25, 2022 • edited

j-s-3 commented May 25, 2022

bergerst commented May 25, 2022

Subhalakshmi1986 commented May 26, 2022 • edited

chadlwilson commented May 26, 2022

Subhalakshmi1986 commented May 26, 2022

j-s-3 commented May 26, 2022

aikebah commented Jun 14, 2022

Redirts commented Oct 18, 2022 • edited

aikebah commented Oct 18, 2022

xtermi2 commented Dec 21, 2022

...

aikebah commented Dec 21, 2022 • edited

xtermi2 commented Dec 21, 2022 • edited

aikebah commented Dec 21, 2022

jeremylong commented Dec 24, 2022

aikebah commented Dec 24, 2022

jeremylong commented Dec 24, 2022

jeremylong commented Dec 24, 2022 via email

msillence commented Jan 4, 2023 • edited

msillence commented Jan 4, 2023 • edited

j-s-3 commented Jan 5, 2023

dmitry-weirdo commented May 25, 2022 •

edited

flash-me commented May 25, 2022 •

edited

danielbraeutigam commented May 25, 2022 •

edited

danielbraeutigam commented May 25, 2022 •

edited

mlemmens commented May 25, 2022 •

edited

achintSatsangi commented May 25, 2022 •

edited

jwwallin commented May 25, 2022 •

edited

sailro commented May 25, 2022 •

edited

dmitry-weirdo commented May 25, 2022 •

edited

lzuyderhoff commented May 25, 2022 •

edited

flash-me commented May 25, 2022 •

edited

ctnelson1997 commented May 25, 2022 •

edited

Subhalakshmi1986 commented May 26, 2022 •

edited

Redirts commented Oct 18, 2022 •

edited

aikebah commented Dec 21, 2022 •

edited

xtermi2 commented Dec 21, 2022 •

edited

msillence commented Jan 4, 2023 •

edited

msillence commented Jan 4, 2023 •

edited