Recommended way to deal with failed downloads?

[ghost]

In the light of issues like #38: What is the recommended way to use this tool when downloads can fail?

Is it safe to use this tool without risking to corrupt an existing mirror directory? If I have used the tool successfully to mirror the JSON files into a directory nist/, will this directory still contain the data in a usable way even if running this tool again fails?

Edit: If I look at the code

nist-data-mirror/src/main/java/us/springett/nistdatamirror/NistDataMirror.java

Line 196 in 5e5ef4a

file = new File(outputDir, filename);

the target file is directly opened for writing. This can potentially corrupt it. Maybe a temporary file could be used instead?

[ghost]

I've opened a PR #45 that implements downloading to a temporary directory.

[nigredo-tori]

+1 to this issue. I've stumbled upon this project when issues with nvd.nist.gov broke CI in our projects. I intended to use the mirror to avoid this issue in the future. However, if connection issues can, indeed, break the mirror, it defeats the point.

[lbreuss]

+1 We suffer the same issue with 1.5.3, and now upgraded to docker image nvd-mirror 1.6.0. But I expect the problem to show again in a few days. This is quite a problem for our CI system, as the maven dependency-check plugin caches the files itself but does not handle corrupt .json.gz files very well, i.e. it does not try to immediately fetch again when I've already fixed the nvd-mirror manually...

IMHO, nist-data-mirror should quarantine downloaded corrupt files. Or -- as suggested by @ghost -- download to a temporary file, before moving the integrity-checked file to the output directory, i.e. /tmp/nvd.

• The .meta needs a sanity check after download, e.g. last line must match "^sha256:[0-9A-F]{64}$
• The the .json and .json.gz have to be checked according to the length and checksum in the .meta file.
• As a result, the nvd-mirror will never update the htdocs folder with corrupt files from /tmp/nvd