Improve GitHub archiving

[yzqzss]

Improved the archiving quality of GitHub Issue&PR pages.

26e6dd2, f94197f fix:
I noticed that some "data:" URLs were being queued that should have been filtered => We forgot to call filterURLsByProtocol() for assets.

e543bb3 fix:
GitHub has rewritten the issue pages in react in recent months. Now they embed many useful (images, videos, user avatars...) assets into HTML <script> tags as JSON for issue pages.

These assets are in JSON strings.

and JSON extractor currently only considers an asset if the entire string is a valid URL.

49de5c7 improved:
These GitHub assets have no file suffix, so a sitespecific/github.go patch was added to JSON extractor to treat them as assets.

daa9ee8 fix:
hasFileExtension() wrongly considers Protocol and domain only, no path links like https://example.com to have a suffix.

92241c7: make isValidURL() check more robust

fe3f888 Changed: Renamed the original LinkRegex to QuotedLinkRegex to reflect its actual purpose. The new LinkRegex is copyied and adapted from heritrix3. :)

[yzqzss]

Before (left window), after (right window):

2025-06-21.23-56-01.mp4

You can see that the avatars, pictures, and video attachment have been captured correctly.

Example issue url: https://github[.]com/webrecorder/browsertrix-behaviors/issues/58

[Copilot]

Pull Request Overview

This PR enhances URL filtering and extraction to improve archiving quality, especially for GitHub Issue/PR pages.

• Introduced filterURLsByProtocol to exclude unwanted protocols (e.g., data:, file:).
• Expanded JSON extractor to detect URLs in text fields and treat GitHub-specific asset patterns as assets.
• Updated tests and core logic for file-extension detection, URL validation, and GitHub asset recognition.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/pkg/postprocessor/utils.go	Added skipProtocolsRe and filterURLsByProtocol for protocol-based URL filtering.
internal/pkg/postprocessor/sitespecific/github/github_test.go	Added unit tests for ShouldConsiderAsAsset covering various GitHub URL patterns.
internal/pkg/postprocessor/sitespecific/github/github.go	Implemented ShouldConsiderAsAsset with regex matchers for GitHub asset URLs.
internal/pkg/postprocessor/outlinks.go	Removed duplicate filterURLsByProtocol and skipProtocolsRe, centralizing filtering logic.
internal/pkg/postprocessor/extractor/xml_test.go	Adjusted expected indices in XML feed test after parsing changes.
internal/pkg/postprocessor/extractor/utils_test.go	Added tests ensuring domain-only URLs are not considered as having file extensions.
internal/pkg/postprocessor/extractor/utils.go	Enhanced hasFileExtension to strip query/fragment and exclude protocol-only URLs.
internal/pkg/postprocessor/extractor/json_test.go	Refactored JSON tests to separate assets/outlinks and added text-link extraction cases.
internal/pkg/postprocessor/extractor/json.go	Extended JSON extraction: added text-link detection and GitHub asset handling in GetURLsFromJSON.
internal/pkg/postprocessor/extractor/html.go	Replaced commented logger call with structured logger.Debug for script-JSON errors.
internal/pkg/postprocessor/assets.go	Applied filterURLsByProtocol to both assets and outlinks before finalizing extraction.

Comments suppressed due to low confidence (1)

internal/pkg/postprocessor/sitespecific/github/github.go:24

• [nitpick] The variable name regexGithubAsset is inconsistent with the githubAvatar, githubAssets, etc. prefix. Consider renaming it to something like githubRepoAssetsRe to match the naming style of other regex variables.

var regexGithubAsset = regexp.MustCompile(`(?i)https://github\.com/[^/]+/[^/]+/assets/`)

[yzqzss]

Ready for review :)

[NGTmeaty]

Thank you! These improvements look super solid and very helpful! I had a couple of small questions in my review:

[yzqzss]

Also renamed the original LinkRegex to QuotedLinkRegex to reflect its actual purpose.
The new LinkRegex is copied and adapted from heritrix3. :)

fe3f888

[NGTmeaty]

Thanks! Changes look great!