Skip to content

Add faster domain lookup for domainscrawl#445

Merged
NGTmeaty merged 7 commits intomainfrom
faster-domainscrawl-domain-lookup
Aug 22, 2025
Merged

Add faster domain lookup for domainscrawl#445
NGTmeaty merged 7 commits intomainfrom
faster-domainscrawl-domain-lookup

Conversation

@CorentinB
Copy link
Copy Markdown
Collaborator

Use map instead of slice for domain matching.

@CorentinB CorentinB self-assigned this Aug 22, 2025
@CorentinB CorentinB added the enhancement New feature or request label Aug 22, 2025

This comment was marked as outdated.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Aug 22, 2025

Codecov Report

❌ Patch coverage is 34.61538% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.69%. Comparing base (b56f8b4) to head (85a0f72).

Files with missing lines Patch % Lines
...nal/pkg/postprocessor/domainscrawl/domainscrawl.go 34.78% 14 Missing and 1 partial ⚠️
internal/pkg/config/config.go 0.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #445      +/-   ##
==========================================
- Coverage   54.77%   54.69%   -0.09%     
==========================================
  Files         120      120              
  Lines        7326     7345      +19     
==========================================
+ Hits         4013     4017       +4     
- Misses       2990     3004      +14     
- Partials      323      324       +1     
Flag Coverage Δ
e2etests 37.17% <3.84%> (-0.09%) ⬇️
unittests 30.89% <30.76%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@NGTmeaty NGTmeaty requested a review from Copilot August 22, 2025 08:47
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes domain lookup performance by replacing a slice-based domain storage with a map-based approach for faster O(1) lookups instead of O(n) iterations. It also adds support for loading domains from files.

  • Changed domain storage from []string to map[string]struct{} for faster lookup performance
  • Added file support to load domains from external files via new DomainsCrawlFile configuration option
  • Updated the AddElements function signature to accept both inline domains and file paths

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
internal/pkg/postprocessor/domainscrawl/domainscrawl.go Converts domain storage from slice to map and adds file reading capability
internal/pkg/postprocessor/domainscrawl/domainscrawl_test.go Updates test to handle map-based domain storage by converting to slice for comparison
internal/pkg/config/config.go Adds DomainsCrawlFile configuration option and updates AddElements call

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread internal/pkg/postprocessor/domainscrawl/domainscrawl.go
Comment thread internal/pkg/postprocessor/domainscrawl/domainscrawl.go
Copy link
Copy Markdown
Collaborator

@NGTmeaty NGTmeaty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!

@NGTmeaty NGTmeaty merged commit c8a6c40 into main Aug 22, 2025
5 checks passed
@CorentinB
Copy link
Copy Markdown
Collaborator Author

Thanks for the quick review, 🐐

@CorentinB CorentinB deleted the faster-domainscrawl-domain-lookup branch August 22, 2025 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants