Add faster domain lookup for domainscrawl#445
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #445 +/- ##
==========================================
- Coverage 54.77% 54.69% -0.09%
==========================================
Files 120 120
Lines 7326 7345 +19
==========================================
+ Hits 4013 4017 +4
- Misses 2990 3004 +14
- Partials 323 324 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull Request Overview
This PR optimizes domain lookup performance by replacing a slice-based domain storage with a map-based approach for faster O(1) lookups instead of O(n) iterations. It also adds support for loading domains from files.
- Changed domain storage from
[]stringtomap[string]struct{}for faster lookup performance - Added file support to load domains from external files via new
DomainsCrawlFileconfiguration option - Updated the
AddElementsfunction signature to accept both inline domains and file paths
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| internal/pkg/postprocessor/domainscrawl/domainscrawl.go | Converts domain storage from slice to map and adds file reading capability |
| internal/pkg/postprocessor/domainscrawl/domainscrawl_test.go | Updates test to handle map-based domain storage by converting to slice for comparison |
| internal/pkg/config/config.go | Adds DomainsCrawlFile configuration option and updates AddElements call |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
NGTmeaty
left a comment
There was a problem hiding this comment.
Looks good! Thank you!
|
Thanks for the quick review, 🐐 |
Use map instead of slice for domain matching.