Optimize domainscrawl domain matching

[CorentinB]

This pull request introduces significant improvements to the domainscrawl postprocessor package, focusing on more efficient domain matching. It doesn't touch the URLs/regex matching part.

We now use an Adaptive Radix Tree (ART) in addition to the map. The map is still used in priority for O(1) lookup of domains, but when we go to the second step to check for subdomain match we now use the ART with prefix-based search on the reversed hostname to allow for O(k) lookup where k is the domain length.

It also adds simple e2e testing for the domains crawl feature.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.72%. Comparing base (7ba600c) to head (1bddc50).
⚠️ Report is 85 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #452      +/-   ##
==========================================
+ Coverage   55.86%   56.72%   +0.85%     
==========================================
  Files         128      130       +2     
  Lines        8053     8136      +83     
==========================================
+ Hits         4499     4615     +116     
+ Misses       3182     3153      -29     
+ Partials      372      368       -4     

Flag	Coverage Δ
e2etests	39.88% <53.46%> (+0.84%)	⬆️
unittests	30.23% <100.00%> (+0.91%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This pull request optimizes domain matching in the domainscrawl postprocessor by introducing an Adaptive Radix Tree (ART) for more efficient subdomain lookups. The optimization maintains O(1) exact domain matches via a map while providing O(k) subdomain matching where k is the domain length, replacing the previous O(n) iteration over all stored domains.

• Replaces map-based domain storage with a hybrid ART + map approach
• Implements efficient subdomain matching using reversed hostnames for prefix searches
• Adds comprehensive test coverage and benchmarks for the new functionality

Reviewed Changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
internal/pkg/postprocessor/domainscrawl/tree.go	Core ART implementation with domain insertion and matching logic
internal/pkg/postprocessor/domainscrawl/reversehost.go	Host reversal utility for efficient prefix matching in the ART
internal/pkg/postprocessor/domainscrawl/domainscrawl.go	Updated to use ART instead of map for domain storage and matching
internal/pkg/postprocessor/domainscrawl/*_test.go	Comprehensive test coverage for ART, host reversal, and integration
e2e/test/domainscrawl/	End-to-end testing for domains crawl functionality
go.mod	Added dependency on go-adaptive-radix-tree library

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[equals215]

LGTM

[NGTmeaty]

Looks good to me. Thanks for the improvements here!

[NGTmeaty]

so SURT.... 😆

[CorentinB]

when I wrote that, I was like "wait, that seems familiar" 🤣