Dependency-Track NVD API mirroring for Dependency-Check

[tomkuipers]

Current Behavior

Previously I used the NVD mirroring feature to speed up dependency-check using parameters -DcveUrlModified=https://my.dependency-track.host/mirror/nvd/nvdcve-1.1-modified.json.gz -DcveUrlBase=https://my.dependency-track.host/mirror/nvd/nvdcve-1.1-%d.json.gz.
Using dependency-check parameter nvdDatafeedUrl results in error [ERROR] Unable to download the NVD API cache.properties.

On what endpoint is the NVD API mirroring made available? How can a third party tool use the NVD API mirroring functionality?

Steps to Reproduce

1. In DT settings set Enable National Vulnerability Database mirroring, Enable mirroring via API, Additionally download feeds and provide an API key.
2. Check logs to if NistMirrorTask and NistApiMirrorTask have run successfully:

  [NistMirrorTask] NIST mirroring complete
  [NistMirrorTask] Time spent (d/l):   3767ms
  [NistMirrorTask] Time spent (total): 4411ms 
  [NistApiMirrorTask] Mirroring of 39 CVEs completed in PT6.87017985S

3. Run mvn org.owasp:dependency-check-maven:9.0.4:aggregate -DnvdDatafeedUrl=https://my.dependency-track.host/mirror/nvd -Dformats=ALL
4. Results in [ERROR] Unable to download the NVD API cache.properties org.owasp.dependencycheck.data.update.exception.UpdateException: Unable to download the NVD API cache.properties

Expected Behavior

I expected to be able to use the mirror functionality in DT in dependency-check just like the formerly downloaded feeds. How should I integrate both OWASP software projects?

What endpoint should be used for NVD mirroring via API for consumption in third party tools (dependency-check)?
I could not find any documentation apart from a tooltip in the settings Feeds will not be parsed, but made available to other clients at /mirror/nvd.

Dependency-Track Version

4.10.0

Dependency-Track Distribution

Container Image

Database Server

PostgreSQL

Database Server Version

No response

Browser

N/A

Checklist

• I have read and understand the contributing guidelines
• I have checked the existing issues for whether this defect was already reported

[aggeboe]

@tomkuipers To my understanding, dependency-check 9.x will not work with dependencytrack's NVD mirror functionality, because only NVD API 1.1 (data feeds) are made available for other tools.

I suspect I'm struggling with the same scenario

• dependencytrack 4.9.x
• dependencycheck 8.x getting NVD API 1.1 updates from /mirror/nvd

and now want to switch to (not working or haven't found a solution yet)

• dependencytrack 4.10.0
• dependency-check 9.x and get NVD API 2.0 updates from dependencytrack

This issue maybe relevant for you as well: jeremylong/DependencyCheck#6277

[tomkuipers]

@aggeboe Thank you for referencing the dependency-check issue. We have the same scenario indeed.

[nscuro]

It might be best that DT adopts to working with the cache created by vulnz, which is what DC is already doing or will be doing. That way you could host your cache centrally, and serve both DT and DC with it.

[schwichtgit]

If DT is concerned to avoid EOL-ing the NVD API 1.1 (data feeds) at https://my.dependency-track.host/mirror/nvd
for existing downstream consumers, it may be a good feature to have an additional url
https://my.dependency-track.host/mirror/nvd-v2 (or equiv),
so legacy downstream consumers are not being disrupted and downstream consumers such as dependency-check V9+
could use a different path mirror/nvd-v2 to have vulnz/dependency-check 9+ compatible feeds.

CI implementations of dependency-check V9+ have to change its use anyway, as dependency-check V9+'s use of command line arguments changed with the support of NIST NVD apiv2 support,
so having a new endpoint URL in the local DT server would be a good compromise IMHO.

[tomkuipers]

It might be best that DT adopts to working with the cache created by vulnz, which is what DC is already doing or will be doing. That way you could host your cache centrally, and serve both DT and DC with it.

But that would mean the introduction of another piece of infrastructure. I would rather like DT to function as this central cache, just like before with dependency-track 4.9.x and dependency-check 8.x.
DT claims to feature NVD 2.0 API mirroring, although it is not well-documented how to consume the cache. From what I can tell the DT codebase it is already relying on vulnz client library. The vulnz cli tool uses the same library (also by @jeremylong):

<dependency>
   <groupId>io.github.jeremylong</groupId>
   <artifactId>open-vulnerability-clients</artifactId>
   <version>5.1.0</version>
</dependency>

Class org/dependencytrack/tasks/NistApiMirrorTask.java (https://github.com/DependencyTrack/dependency-track/blob/master/src/main/java/org/dependencytrack/tasks/NistApiMirrorTask.java) has dependencies on io.github.jeremylong.openvulnerability.client.*

[aggeboe]

@tomkuipers +1
We used to have a central caching mechanism in our organisation (H2 pre-built NVD image) to speed up dependency-check in CI/CD. When DT was introduced in our organisation with the NVD 1.1 data feed mirror, we immediately switched to the DT cache to utilize the mirror/cache already in place here.

Thus, we really hoped that a similiar mechanism would be in place for the NVD API 2.0 in DT 4.10.0, for it to continue to function as a central cache.

[nscuro]

For those not following the linked DependencyCheck thread, what we'll do on the DT side is to provide an optional feature which can be used to emulate the NVD's REST API: jeremylong/DependencyCheck#6277 (comment)

Dependency-Check 9.0.5 now supports customization of the REST API URL, so once this is implemented in DT, you'll be able to point DC to DT's emulated API endpoint.

[aggeboe]

@nscuro Thanks for all your hard work, really looking forward to release 4.10.1. Any idea when it migth be released?

[tomkuipers]

I'm also looking forward to version v.4.10.1, thank you @nscuro for implementing the feature and collaborating with dependency-check's @jeremylong to find a solution.

[Kretikus]

On Moday I installed 4.10.0 and set up the new NIST Rest API.
As far as i see, DT has not set a Last Modification Time yet and therefore pulls every day the complete list.
I see this in my logs:

dependency-track.log:2023-12-14 09:27:32,686 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 2000/233111 CVEs (0%)
dependency-track.log:2023-12-14 09:27:49,277 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 4000/233111 CVEs (1%)
dependency-track.log:2023-12-14 09:28:04,956 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 6000/233111 CVEs (2%)
dependency-track.log:2023-12-14 09:28:21,147 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 8000/233111 CVEs (3%)

is this related to the OP issue?

[officerNordberg]

Yesterday I installed 4.10.0 and set up the new NIST Rest API.
As far as i see, DT has not set a Last Modification Time yet NOR does my log contain any NistApiMirrorTask lines or any apparent errors. I've changed the Task Schedule from 24 hours to 1 hour, waited and no dice. I've disabled and reenabled a few times. I've been tempted to set the Last Mod variables but have heretofore headed the Configuration warning. Not sure what to do next.

[nscuro]

@Kretikus Does it ever complete successfully? There should be a log line stating:

Mirroring of CVEs completed in

Unfortunately due to the way the API was designed, we cannot store intermediate last modification timestamps, it's only possible at the very end of the mirror operation...

@officerNordberg Can you share your configuration in the UI? What toggles did you enable / disable? I've gotten some feedback via Slack already that the way it's structured is confusing so I'll definitely get that improved.

As an FYI we did get confirmation from the NVD via email that feeds will not be retired as planned on 18th. I asked about a public statement but haven't yet gotten feedback. But I want you to know that, as per my current knowledge, you will not be cut off from NVD updates if the REST API integration doesn't work for you right now.

[officerNordberg]

@nscuro got the same heads up on the predictable further delay of the feed retirement through yocto/linux foundation this morning.

I have all 3 toggles enabled, I have tried just enabling API as well as just the first two without the "Additionally..." toggle.

[aggeboe]

@nscuro @officerNordberg Thanks for the heads up, regarding the feed retirement delay! Much appreciated!

[officerNordberg]

@nscuro I should have realized that configuring it requires a restart afterward. I don't know why I didn't expect that but, yup, that did it.

[nscuro]

The NVD setting should not require a restart. The decision of which source to use is happening dynamically at runtime.

... but I think change of schedule does, because tasks are scheduled on startup based on what is configured at that moment in time. So yeah that doesn't make testing this any easier because impact can't be observed immediately. Yet another reason for adding the ability to manually trigger tasks.

[officerNordberg]

@Kretikus try restarting, not only did the restart fix my missing EventSubscription for the ApiTask but once my mirror completed, the Last updated fields remained empty until I restarted yet again.

[nscuro]

An official update has been posted here: https://groups.google.com/a/list.nist.gov/g/nvd-news/c/aofnAd3HP2g

Due to feedback received from many different downstream data consumer groups after our previous reminder, we will again be extending the retirement date for the Legacy Data Feed files. However, we will still be retiring the 1.0 APIs.
Going forward, we will be improving capabilities to allow for bulk download of the NVD dataset. The legacy data feeds will remain available until this effort is completed.

For everyone experiencing issues with the new REST API integration, you can safely switch back to the previous feed-based way of mirroring.

[valentijnscholten]

Sounds like (I hope), they're gonna reinstate some kind of batch download feeds.

[Kretikus]

@Kretikus Does it ever complete successfully? There should be a log line stating:

Mirroring of CVEs completed in

Yes, here is the log from today:

2023-12-18 09:55:30,067 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 196000/233564 CVEs (83%)
2023-12-18 09:55:44,206 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 198000/233564 CVEs (84%)
2023-12-18 09:55:59,579 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 200000/233564 CVEs (85%)
2023-12-18 09:56:13,136 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 202000/233564 CVEs (86%)
2023-12-18 09:56:23,415 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 204000/233564 CVEs (87%)
2023-12-18 09:56:37,305 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 206000/233564 CVEs (88%)
2023-12-18 09:56:54,020 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 208000/233564 CVEs (89%)
2023-12-18 09:57:08,287 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 210000/233564 CVEs (89%)
2023-12-18 09:57:20,495 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 212000/233564 CVEs (90%)
2023-12-18 09:57:36,939 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 214000/233564 CVEs (91%)
2023-12-18 09:57:51,644 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 216000/233564 CVEs (92%)
2023-12-18 09:58:07,079 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 218000/233564 CVEs (93%)
2023-12-18 09:58:23,707 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 220000/233564 CVEs (94%)
2023-12-18 09:58:38,319 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 222000/233564 CVEs (95%)
2023-12-18 09:58:53,594 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 224000/233564 CVEs (95%)
2023-12-18 09:59:06,151 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 226000/233564 CVEs (96%)
2023-12-18 09:59:22,012 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 228000/233564 CVEs (97%)
2023-12-18 09:59:35,948 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 230000/233564 CVEs (98%)
2023-12-18 09:59:50,941 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 232000/233564 CVEs (99%)
2023-12-18 10:00:01,439 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirroring of 233564 CVEs completed in PT32M46.652815377S

A restart showed now a date, but with a wrong time. The times above are CET (UTC+1) and the screenshot in the front-end shows:

[officerNordberg]

My last update seems to only update on restarts read from the Config loaded at restart.

2023-12-15 17:16:59,679 INFO [NistApiMirrorTask] Mirroring of 233501 CVEs completed in PT24M51.368955359S
2023-12-15 19:56:29,793 INFO [NistApiMirrorTask] Mirroring CVEs that were modified since 2023-12-15T16:53:57Z
2023-12-15 19:56:33,445 INFO [NistApiMirrorTask] Mirroring of 49 CVEs completed in PT3.655197639S
2023-12-16 19:56:29,784 INFO [NistApiMirrorTask] Mirroring CVEs that were modified since 2023-12-15T16:53:57Z
2023-12-16 19:56:34,960 INFO [NistApiMirrorTask] Mirroring of 219 CVEs completed in PT5.17599173S
2023-12-17 19:56:29,784 INFO [NistApiMirrorTask] Mirroring CVEs that were modified since 2023-12-15T16:53:57Z
2023-12-17 19:56:35,536 INFO [NistApiMirrorTask] Mirroring of 247 CVEs completed in PT5.752890702S

[nscuro]

My last update seems to only update on restarts read from the Config loaded at restart.

Checking. Looks like the property in question is loaded from the ORMs cache even though it shouldn't, and cache has been explicitly disabled for it...

@Kretikus A restart showed now a date, but with a wrong time. The times above are CET (UTC+1) and the screenshot in the front-end shows: [...]

The time in the frontend does not represent the time when Mirroring of 233564 CVEs completed was logged, but the latest "last modification" timestamp of all mirrored CVEs in that run. So even though your mirroring operation completed "now", the frontend may show a time from (way) earlier in the day.

[nscuro]

Yup, the last modification timestamp requiring a restart to take effect is caused by inconsistencies of the ORM's L2 cache. Fixed in #3322 and backported to 4.10.1 in #3323.

If anyone is feeling adventurous, you can disable the L2 cache system-wide via alpine.datanucleus.cache.level2.type=none / ALPINE_DATANUCLEUS_CACHE_LEVEL2_TYPE=none. Other than resolving this particular issue, it also improves memory footprint quite a bit. We disabled it in Hyades and only saw benefits. I am planning to disable it per default for DT v4.x as well eventually, but would love some input whether others are running into issues with L2 disabled.

[nscuro]

Sorry folks, the cache implementation did not make it to 4.10.1. I'll assign this to 4.11; We can still decide to pre-pone the release to 4.10.x once it's fully implemented.

[aggeboe]

@nscuro I guess this means, it will not make it to 4.11. Really looking forward to this feature! 🙏