NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. #6547

inerbus · 2024-03-25T19:04:48Z

Describe the bug
NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times.

Version of dependency-check used
The problem occurs using version 9.0.10 of the Maven

Pom configuration

     <build>
	<plugins>
		<plugin>
			<groupId>org.owasp</groupId>
			<artifactId>dependency-check-maven</artifactId>
			<version>9.0.10</version>

			<configuration>
				<nvdApiKey>{{apiKey}}</nvdApiKey>
				<nvdApiDelay>16000</nvdApiDelay>
				<nvdMaxRetryCount>20</nvdMaxRetryCount>
				<nvdValidForHours>8</nvdValidForHours>
				<connectionTimeout>15000</connectionTimeout>
			</configuration>
			<executions>
				<execution>
					<goals>
						<goal>check</goal>
					</goals>
				</execution>
			</executions>

		</plugin>
	</plugins>
</build>

Log file
C:\Users\amolina\Documents\workspace-serfinsa\org.serfinsa.starter.model>mvn dependency-check:check
[INFO] Scanning for projects...
[INFO]
[INFO] ----------< org.serfinsa.starter:org.serfinsa.starter.model >-----------
[INFO] Building org.serfinsa.starter.model 1.0.0-RELEASE
[INFO] from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- dependency-check:9.0.10:check (default-cli) @ org.serfinsa.starter.model ---
[INFO] Checking for updates
[INFO] NVD API has 242,601 records in this update
[INFO] Downloaded 10,000/242,601 (4%)
[INFO] Downloaded 20,000/242,601 (8%)
[INFO] Downloaded 30,000/242,601 (12%)
[INFO] Downloaded 40,000/242,601 (16%)
[INFO] Downloaded 50,000/242,601 (21%)
[INFO] Downloaded 60,000/242,601 (25%)
[INFO] Downloaded 70,000/242,601 (29%)
[INFO] Downloaded 80,000/242,601 (33%)
[INFO] Downloaded 90,000/242,601 (37%)
[INFO] Downloaded 100,000/242,601 (41%)
[INFO] Downloaded 110,000/242,601 (45%)
[INFO] Downloaded 120,000/242,601 (49%)
[INFO] Downloaded 130,000/242,601 (54%)
[INFO] Downloaded 140,000/242,601 (58%)
[INFO] Downloaded 150,000/242,601 (62%)
[INFO] Downloaded 160,000/242,601 (66%)
[INFO] Downloaded 170,000/242,601 (70%)
[INFO] Downloaded 180,000/242,601 (74%)
[INFO] Downloaded 190,000/242,601 (78%)
[INFO] Downloaded 200,000/242,601 (82%)
[INFO] Downloaded 210,000/242,601 (87%)
[INFO] Downloaded 220,000/242,601 (91%)
[INFO] Downloaded 230,000/242,601 (95%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:389)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1953)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1136)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:568)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times.
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful (NvdCveClient.java:422)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext (NvdCveClient.java:300)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:323)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:324)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:341)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:349)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1953)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1136)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:568)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours.
[WARNING] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:02 min
[INFO] Finished at: 2024-03-25T12:54:40-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.owasp:dependency-check-maven:9.0.10:check (default-cli) on project org.serfinsa.starter.model: Fatal exception(s) analyzing org.serfinsa.starter.model: One or more exceptions occurred during analysis:
[ERROR] UpdateException: Error updating the NVD Data
[ERROR] caused by NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times.
[ERROR] NoDataException: No documents exist
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

The text was updated successfully, but these errors were encountered:

nicollasteixeira · 2024-03-25T20:36:03Z

This is happening practically daily and apparently no solution has been made available yet.

defaultbranch · 2024-03-25T21:27:02Z

I see something similar; I have been trying to bootstrap the plugin (mvn dependency-check:update-only) and am experimenting with the available options; in all my last tries, I got up to 95%:

...
[INFO] Downloaded 210,000/242,617 (87%)
[INFO] Downloaded 220,000/242,617 (91%)
[INFO] Downloaded 230,000/242,617 (95%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:389)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
...

What is wrong? Latest, I provided -DnvdApiDelay=10000, and the time-to-failure was still ~600 seconds (10 minutes) like before with -DnvdApiDelay=5000. Was the value ignored? I'll keep trying...

defaultbranch · 2024-03-25T21:30:14Z

this looks similar to #6546 ?

inerbus · 2024-03-25T21:36:54Z

Yes, its similar

knoxg · 2024-03-26T01:54:25Z

I'm seeing this as well at the 95% downloaded mark

#6 1005.0 [INFO] Downloaded 220,000/242,633 (91%)
#6 1052.7 [INFO] Downloaded 230,000/242,633 (95%)
#6 1341.5 [ERROR] Error updating the NVD Data
#6 1341.5 org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
#6 1341.5     at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:389)
#6 1341.5     at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
[...]
#6 1341.5 Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times.
[...]

joywaves · 2024-03-26T02:46:11Z

like @knoxg mentioned choking at the 95% download mark.

[INFO] Downloaded 220,000/242,634 (91%)
[INFO] Downloaded 230,000/242,634 (95%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
        at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389)
        at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times.
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.queueUnsuccessful(NvdCveClient.java:422)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.hasNext(NvdCveClient.java:300)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:323)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:324)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
        at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349)
        ... 7 common frames omitted
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[INFO] End database defrag (4897 ms)
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.
[ERROR] One or more fatal errors occurred
[ERROR] Error updating the NVD Data
[ERROR] No documents exist

knoxg · 2024-03-26T04:51:58Z

For what it's worth, trying to get the last page of results via curl fails with a 503 error most of the time, but works every now and then.

 curl --header "apiKey: ********-****-****-****-************" \
  -X GET 'https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=242000'

I'll try increasing the nvdMaxRetryCount

tarrinho · 2024-03-26T11:08:45Z

Same here! Still happening!

jeremylong · 2024-03-26T11:10:27Z

see #6546

inerbus added the bug label Mar 25, 2024

defaultbranch mentioned this issue Mar 25, 2024

Update Errors #6546

Open

jeremylong added the duplicate label Mar 26, 2024

jeremylong closed this as completed Mar 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. #6547

NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. #6547

inerbus commented Mar 25, 2024 •

edited

nicollasteixeira commented Mar 25, 2024

defaultbranch commented Mar 25, 2024 •

edited

defaultbranch commented Mar 25, 2024

inerbus commented Mar 25, 2024

knoxg commented Mar 26, 2024

joywaves commented Mar 26, 2024

knoxg commented Mar 26, 2024

tarrinho commented Mar 26, 2024

jeremylong commented Mar 26, 2024

NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. #6547

NVD Update Failed: attempted to retrieve starting index 242000 from the NVD unsuccessfully five times. #6547

Comments

inerbus commented Mar 25, 2024 • edited

nicollasteixeira commented Mar 25, 2024

defaultbranch commented Mar 25, 2024 • edited

defaultbranch commented Mar 25, 2024

inerbus commented Mar 25, 2024

knoxg commented Mar 26, 2024

joywaves commented Mar 26, 2024

knoxg commented Mar 26, 2024

tarrinho commented Mar 26, 2024

jeremylong commented Mar 26, 2024

inerbus commented Mar 25, 2024 •

edited

defaultbranch commented Mar 25, 2024 •

edited