Implement the hackage and nixpkgs meta analyzers

[MangoIV]

Description

This solves part 2 of #3545

Addressed Issue

#3545

Additional Details

this depends on #3546

the apache http client utils version 4 don't implement brotli decompression; I redid some of the common http client code to keep this scoped; a future refactoring could remove the extraneous code.

Checklist

• I have read and understand the contributing guidelines
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[MangoIV]

what's missing:

• invoke the refreshing job every now and then
• make Brotli decompression not error
• the usual chores.

[MangoIV]

so if my theory is true, then this is some truly cursed shit:

• at runtime, the http client library looks for presence of the brotli dec class
• if it doesn't exist, it doesn't try to decode brotli encoded files but still removes the decoding header and goes on
• if it does exist, it actually converts it into a decompressing entity and then goes on decoding it

[MangoIV]

Hi, I would like some input on where to best invoke the update of the nixpkgs meta analyzer singleton. Thanks in advance!

[emil-wire]

@nscuro I don't want to pester you and understand you have a lot on your mind. We'd be grateful if you could provide some guidance at your earliest convenience to make this PR a success and land this feature in DT. Thank you!

[nscuro]

@MangoIV @emil-wire Hey, will have a look at this ASAP and provide feedback. Thank you for your contributions! :)

[nscuro]

@MangoIV Instead of using a singleton of NixpkgsMetaAnalyzer, and updating it every now and then, you could use a read-through cache with "expire-after-write" eviction. The Caffeine cache library is already available in DT.

So the first time a task tries to access nix versions, it will fetch and parse packages.json, and put it in the cache. Other tasks trying to access it are blocked until the cache population finishes. After that, they can be quickly served from said cache. The cache will expire after a given interval (e.g. 1h), and the whole thing repeats.

The benefit of this is that you won't have to deal with concurrency controls such as locking on your own. Also refreshing happens implicitly and as such doesn't require a dedicated scheduled task.

Because we'd want to refresh the entire package index at once, the entire index would be a single cache entry:

class NixPkgMetaAnalyzer {

    private static final Cache<String, Map<String, String>> CACHE = Caffeine.newBuilder()
            .expireAfterWrite(60, TimeUnit.MINUTES)
            .maximumSize(1)
            .build();

    MetaModel analyze(Component component) {
        Map<String, String> latestVersions = CACHE.get("nixpkgs", cacheKey -> {
            final var versions = new HashMap<String, String>();
            // TODO: Download and parse package index
            return versions;
        });

        // TODO: Do things with latestVersions
    }

More details here: https://github.com/ben-manes/caffeine/wiki/Population#manual

[MangoIV]

Thank you for the answer, that looks very nice!

It looks like a fairly large hammer for the tiny nail in trying to get into the wall but if it’s already used across the codebase, this looks like a feasible option; I’ll adjust accordingly.

[MangoIV]

package-url/packageurl-java@73b2f51

btw; this got merged, i.e. theoretically I could do this minor cleanup; however upstream didn't do a release yet so idk if this is worth the effort.

[MangoIV]

new tests pass locally, let's see if CI is merciful today 🥺

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.43% (target: -1.00%)	✅ 72.22% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (f30bef6)	21514	15924	74.02%
Head commit (2e71242)	21739 (+225)	16185 (+261)	74.45% (+0.43%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3549)	90	65	72.22%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{You may notice some variations in coverage metrics with the latest Coverage engine update. For more details, visit the documentation}

[MangoIV]

@nscuro is this how you'd imagine it? what can I do to make this PR progress?

[nscuro]

Thanks @MangoIV!

As a future enhancement, we'd want to improve the parsing of the nixpkgs catalogue. At the moment, the entire thing is deserialized at once, which causes quite a large amount of memory allocations:

A more efficient way would be to leverage a streaming API such as the one provided by Jackson. That allows us to only deserialize fields we actually care about. We make use of that when parsing CVE data from the NVD already:

dependency-track/src/main/java/org/dependencytrack/parser/nvd/NvdParser.java

Lines 82 to 107 in 3e8f93f

	try (final InputStream in = Files.newInputStream(file.toPath());
	final JsonParser jsonParser = objectMapper.createParser(in)) {
	jsonParser.nextToken(); // Position cursor at first token
	// Due to JSON feeds being rather large, do not parse them completely,
	// but "stream" through them. Parsing individual CVE items
	// one-by-one allows for garbage collection to kick in sooner,
	// keeping the overall memory footprint low.
	JsonToken currentToken;
	while (jsonParser.nextToken() != JsonToken.END_OBJECT) {
	final String fieldName = jsonParser.getCurrentName();
	currentToken = jsonParser.nextToken();
	if ("CVE_Items".equals(fieldName)) {
	if (currentToken == JsonToken.START_ARRAY) {
	while (jsonParser.nextToken() != JsonToken.END_ARRAY) {
	final ObjectNode cveItem = jsonParser.readValueAsTree();
	parseCveItem(cveItem);
	}
	} else {
	jsonParser.skipChildren();
	}
	} else {
	jsonParser.skipChildren();
	}
	}
	} catch (Exception e) {

[MangoIV]

It’s a tradeoff. I would be reluctant to make a guess on whether it’s wise to use streaming here without thorough benchmarks.