NVD Mirroring Connection Hangs, needs timeout

[officerNordberg]

Mirroring started having issues for us on February 25th. Initially a NistMirrorTask hung after logging "Downloading... " This task was still hung March 2nd.
When restarting the containers from docker-compose, a handful of years hung just after the previous log "Initiating download of {...}"

HttpClient should specify a timeout

dependency-track/src/main/java/org/dependencytrack/tasks/NistMirrorTask.java

Lines 206 to 239 in 10d9446

	LOGGER.info("Initiating download of " + url.toExternalForm());
	final HttpUriRequest request = new HttpGet(urlString);
	try (final CloseableHttpResponse response = HttpClientPool.getClient().execute(request)) {
	final StatusLine status = response.getStatusLine();
	final long end = System.currentTimeMillis();
	metricDownloadTime += end - start;
	if (status.getStatusCode() == 200) {
	LOGGER.info("Downloading...");
	try (InputStream in = response.getEntity().getContent()) {
	file = new File(outputDir, filename);
	FileUtils.copyInputStreamToFile(in, file);
	if (ResourceType.CVE_YEAR_DATA == resourceType || ResourceType.CVE_MODIFIED_DATA == resourceType) {
	// Sets the last modified date to 0. Upon a successful parse, it will be set back to its original date.
	file.setLastModified(0);
	}
	if (file.getName().endsWith(".gz")) {
	uncompress(file, resourceType);
	}
	}
	} else if (response.getStatusLine().getStatusCode() == 403) {
	mirroredWithoutErrors = false;
	final String detailMessage = "This may occur if the NVD is throttling connections due to excessive load or repeated " +
	"connections from the same IP address or as a result of firewall or proxy authentication failures";
	LOGGER.warn("Unable to download - HTTP Response 403: " + status.getReasonPhrase());
	LOGGER.warn(detailMessage);
	Notification.dispatch(new Notification()
	.scope(NotificationScope.SYSTEM)
	.group(NotificationGroup.DATASOURCE_MIRRORING)
	.title(NotificationConstants.Title.NVD_MIRROR)
	.content("An error occurred mirroring the contents of the National Vulnerability Database. Check log for details. HTTP Response: " + status.getStatusCode() + ". " + detailMessage)
	.level(NotificationLevel.ERROR)
	);
	} else {
	mirroredWithoutErrors = false;

[stevespringett]

Thanks for the report. I should be able to include support for user configurable timeout settings globally for all HTTP client connections.

[stevespringett]

I have an important defect that's been fixed, so I'll be releasing 4.4.1 shortly. I'll try to get this in there.

[stevespringett]

In Dependency-Track v4.4.2, you'll be able to set these properties using:

ALPINE_HTTP_TIMEOUT_CONNECTION
ALPINE_HTTP_TIMEOUT_SOCKET
ALPINE_HTTP_TIMEOUT_POOL

The first two are likely the ones you'll care about. The last one is a timeout for the pool of available HttpClient objects, which can be tweaked if the server is under heavy load doing a lot of inbound/outbound tasks.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.