Skip to content

Replacing the regex-based CSS extractor with a standard CSS parser - #324

Merged
NGTmeaty merged 2 commits into
mainfrom
css
Jun 3, 2025
Merged

NGTmeaty merged 2 commits into
mainfrom
css

Conversation

@yzqzss

@yzqzss yzqzss commented Jun 1, 2025 •

Copy link
Copy Markdown
Collaborator

#323

The most complete golang css parsing library I've found is https://github.com/tdewolff/parse/, but it returns the entire token as is, for example, if you let it to parse uRl( "http://a\"b\e9 c" ), it will just return a data structure like:
{TokenType: URLToken, Data: []byte(`uRl( "http://a\"b\e9 c" )`) }
and won't parse and return the actual value (http://a"b茅c) for you.

Therefore, I wrote a small parser function parseStringOrURLTokenData() to extract value from token data.

@yzqzss yzqzss mentioned this pull request Jun 1, 2025
3 tasks done
Comment thread internal/pkg/postprocessor/extractor/css.go Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the URL extraction from in-html CSS by replacing regex-based extraction with a dedicated CSS parser, and it adds comprehensive tests for various CSS URL patterns.

  • Replaces regex extraction in html.go with a call to a new CSS() function.
  • Adds a new css.go file implementing the CSS parsing logic.
  • Introduces css_test.go to thoroughly validate CSS URL extraction behavior.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

File Description
internal/pkg/postprocessor/extractor/html.go Removed regex extraction and replaced with CSS() function call
internal/pkg/postprocessor/extractor/css_test.go Added tests covering various CSS URL extraction scenarios
internal/pkg/postprocessor/extractor/css.go Introduces the new CSS parser functions and helper methods
go.mod Updated dependency to github.com/tdewolff/parse/v2 v2.8.1
Comments suppressed due to low confidence (1)

internal/pkg/postprocessor/extractor/css.go:226

  • [nitpick] The empty else block can be removed as it does not add any functionality and only clutters the code.
else { }

Comment thread internal/pkg/postprocessor/extractor/css.go
@yzqzss yzqzss self-assigned this Jun 2, 2025
@yzqzss
yzqzss requested review from CorentinB and NGTmeaty June 2, 2025 17:13
@yzqzss yzqzss changed the title [GSoC] Use CSS parser to get URLs from in-html CSS [GSoC] Replacing the regex-based CSS extractor with a standard CSS parser Jun 2, 2025
@yzqzss yzqzss changed the title [GSoC] Replacing the regex-based CSS extractor with a standard CSS parser Replacing the regex-based CSS extractor with a standard CSS parser Jun 2, 2025
@yzqzss yzqzss added the GSoC label Jun 2, 2025
NGTmeaty
NGTmeaty previously approved these changes Jun 3, 2025

@NGTmeaty NGTmeaty left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thank you for your contribution! This should massive improve our extraction!

@NGTmeaty NGTmeaty left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Thank you for the improvements here and great job! I can't wait to see the rest of the CSS PRs come through! They will be extremely impactful for Zeno!

@NGTmeaty
NGTmeaty merged commit adf43d4 into main Jun 3, 2025
@yzqzss
yzqzss deleted the css branch June 14, 2025 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants