Skip to content

refactor(preprocessor): remove url normalizer with cpp bindings - #374

Closed
otkd wants to merge 7 commits into
internetarchive:mainfrom
otkd:no-cpp-url-normalizer
Closed

otkd wants to merge 7 commits into
internetarchive:mainfrom
otkd:no-cpp-url-normalizer

Conversation

@otkd

@otkd otkd commented Jul 7, 2025

Copy link
Copy Markdown

Step towards multiplatform, in this case simpler build on mac

#245

@otkd otkd mentioned this pull request Jul 7, 2025
@NGTmeaty
NGTmeaty requested a review from willmhowes July 7, 2025 08:41
Comment thread internal/pkg/preprocessor/url.go Resolved
Comment thread internal/pkg/preprocessor/url.go Resolved
Comment thread internal/pkg/preprocessor/url.go Resolved
Comment thread internal/pkg/preprocessor/url.go Resolved
@yzqzss

yzqzss commented Jul 7, 2025

Copy link
Copy Markdown
Collaborator

Could you please append these test cases to url_test.go?
:)

		{
			name:        "URL with fragment",
			rawURL:      "https://example.com/path#fragment",
			wantErr:     false,
			expectedURL: "https://example.com/path",
		},
		{
			name:        "Unicode URL",
			rawURL:      "https://通知.收到了/Zeno/崛起.html?query=了",
			wantErr:     false,
			expectedURL: `https://xn--k7y792c.xn--ykq12govw/Zeno/%E5%B4%9B%E8%B5%B7.html?query=%E4%BA%86`,
		},
		{
			name:        "Uppercase Scheme and Domain URL",
			rawURL:      "HTTPS://EXAMPLE.COM/PATH",
			wantErr:     false,
			expectedURL: "https://example.com/PATH",
		},
		{
			name:        "URL with dots directory in path",
			rawURL:      "https://example.com/../../1/2/../3/./4/5",
			wantErr:     false,
			expectedURL: "https://example.com/1/3/4/5",
		},
		{
			name:        "Localhost URL",
			rawURL:      "http://localhost/path",
			wantErr:     true,
			expectedURL: "",
		},
		{
			name:        "URL with semicolon separator in query",
			rawURL:      "https://example.com/path?q1=a;q2=b",
			wantErr:     false,
			expectedURL: "https://example.com/path?q1=a;q2=b",
		},

@yzqzss

yzqzss commented Jul 7, 2025

Copy link
Copy Markdown
Collaborator

Hmm, we ran out of actions quota?

@willmhowes willmhowes left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why Zeno uses goada is explained here. BUT, I'm happy to move forward without goada when

  • your pull request can pass the tests proposed by @yzqzss
  • you can confirm that nlnwa/whatwg-url has at least similar performance to goada. I'm expecting this to be the case, but want to be sure

Comment thread internal/pkg/preprocessor/url.go Resolved
@yzqzss

yzqzss commented Jul 7, 2025 •

Copy link
Copy Markdown
Collaborator
func BenchmarkNormalizeURL(b *testing.B) {
        tests ... // copy from TestNormalizeURL()
	for _, tc := range tests {
		b.Run(tc.name, func(b *testing.B) {
			var parentURL *models.URL
			if tc.parentURL != "" {
				parentURL = &models.URL{Raw: tc.parentURL}
				parentURL.Parse()
			}

			b.ResetTimer()
			for i := 0; i < b.N; i++ {
				url := &models.URL{Raw: tc.rawURL}
				err := NormalizeURL(url, parentURL)
				if tc.wantErr && err == nil {
					b.Errorf("NormalizeURL() expected error but got none")
				} else if !tc.wantErr && err != nil {
					b.Errorf("NormalizeURL() error = %v", err)
				}
			}
		})
		b.Run(tc.name+"(OLD)", func(b *testing.B) {
			var parentURL *models.URL
			if tc.parentURL != "" {
				parentURL = &models.URL{Raw: tc.parentURL}
				parentURL.Parse()
			}

			b.ResetTimer()
			for i := 0; i < b.N; i++ {
				url := &models.URL{Raw: tc.rawURL}
				err := NormalizeURLOld(url, parentURL) // copy old NormalizeURL()
				if tc.wantErr && err == nil {
					b.Errorf("NormalizeURL() expected error but got none")
				} else if !tc.wantErr && err != nil {
					b.Errorf("NormalizeURL() error = %v", err)
				}
			}
		})
	}
}
go test -run='^$' -benchmem -benchtime=100000x -bench 'BenchmarkNormalizeURL' github.com/internetarchive/Zeno/internal/pkg/preprocessor
BenchmarkNormalizeURL/valid_absolute_URL-12                                 100000              2303 ns/op             960 B/op         40 allocs/op
BenchmarkNormalizeURL/valid_absolute_URL(OLD)-12                            100000              1347 ns/op             440 B/op          8 allocs/op
BenchmarkNormalizeURL/valid_relative_URL_with_parent-12                     100000              2847 ns/op            1616 B/op         52 allocs/op
BenchmarkNormalizeURL/valid_relative_URL_with_parent(OLD)-12                100000              1280 ns/op             368 B/op          7 allocs/op
BenchmarkNormalizeURL/invalid_URL-12                                        100000               362.7 ns/op           312 B/op          6 allocs/op
BenchmarkNormalizeURL/invalid_URL(OLD)-12                                   100000               120.2 ns/op           208 B/op          3 allocs/op
BenchmarkNormalizeURL/valid_URL_without_scheme-12                           100000              3203 ns/op            1920 B/op         69 allocs/op
BenchmarkNormalizeURL/valid_URL_without_scheme(OLD)-12                      100000              1125 ns/op             421 B/op          8 allocs/op
BenchmarkNormalizeURL/FTP_url-12                                            100000              1679 ns/op             632 B/op         25 allocs/op
BenchmarkNormalizeURL/FTP_url(OLD)-12                                       100000              1965 ns/op             236 B/op          5 allocs/op
BenchmarkNormalizeURL/valid_URL_with_path_without_scheme-12                 100000              3537 ns/op            2016 B/op         76 allocs/op
BenchmarkNormalizeURL/valid_URL_with_path_without_scheme(OLD)-12            100000              1202 ns/op             445 B/op          8 allocs/op
BenchmarkNormalizeURL/URL_with_leading_and_trailing_quotes-12               100000              2272 ns/op             960 B/op         40 allocs/op
BenchmarkNormalizeURL/URL_with_leading_and_trailing_quotes(OLD)-12          100000              1269 ns/op             440 B/op          8 allocs/op
BenchmarkNormalizeURL/relative_URL_with_leading_and_trailing_quotes-12      100000              2795 ns/op            1616 B/op         52 allocs/op
BenchmarkNormalizeURL/relative_URL_with_leading_and_trailing_quotes(OLD)-12 100000              1287 ns/op             368 B/op          7 allocs/op
BenchmarkNormalizeURL/relative_URL_without_parent-12                        100000              1351 ns/op             944 B/op         30 allocs/op
BenchmarkNormalizeURL/relative_URL_without_parent(OLD)-12                   100000              2263 ns/op             240 B/op          6 allocs/op

Unfortunately, in general, whatwg-url runs at 40-60% of the speed of goada on my i5-12450H laptap. (applied the code changes from my review.)

@yzqzss

yzqzss commented Jul 7, 2025

Copy link
Copy Markdown
Collaborator
BenchmarkNormalizeURL/valid_absolute_URL-12                                 100000              1252 ns/op             728 B/op         27 allocs/op
BenchmarkNormalizeURL/valid_absolute_URL(OLD)-12                            100000              1264 ns/op             440 B/op          8 allocs/op
BenchmarkNormalizeURL/valid_relative_URL_with_parent-12                     100000              1740 ns/op            1432 B/op         41 allocs/op
BenchmarkNormalizeURL/valid_relative_URL_with_parent(OLD)-12                100000              1154 ns/op             368 B/op          7 allocs/op
BenchmarkNormalizeURL/invalid_URL-12                                        100000               274.1 ns/op           368 B/op          7 allocs/op
BenchmarkNormalizeURL/invalid_URL(OLD)-12                                   100000                95.44 ns/op          208 B/op          3 allocs/op
BenchmarkNormalizeURL/valid_URL_without_scheme-12                           100000              1733 ns/op            1120 B/op         31 allocs/op
BenchmarkNormalizeURL/valid_URL_without_scheme(OLD)-12                      100000              1153 ns/op             421 B/op          8 allocs/op
BenchmarkNormalizeURL/FTP_url-12                                            100000               878.4 ns/op           496 B/op         16 allocs/op
BenchmarkNormalizeURL/FTP_url(OLD)-12                                       100000              1922 ns/op             236 B/op          5 allocs/op
BenchmarkNormalizeURL/valid_URL_with_path_without_scheme-12                 100000              1838 ns/op            1216 B/op         37 allocs/op
BenchmarkNormalizeURL/valid_URL_with_path_without_scheme(OLD)-12            100000              1181 ns/op             445 B/op          8 allocs/op
BenchmarkNormalizeURL/URL_with_leading_and_trailing_quotes-12               100000              1347 ns/op             728 B/op         27 allocs/op
BenchmarkNormalizeURL/URL_with_leading_and_trailing_quotes(OLD)-12          100000              1322 ns/op             440 B/op          8 allocs/op
BenchmarkNormalizeURL/relative_URL_with_leading_and_trailing_quotes-12      100000              2041 ns/op            1432 B/op         41 allocs/op
BenchmarkNormalizeURL/relative_URL_with_leading_and_trailing_quotes(OLD)-12 100000              1210 ns/op             368 B/op          7 allocs/op
BenchmarkNormalizeURL/relative_URL_without_parent-12                        100000               916.8 ns/op           840 B/op         23 allocs/op
BenchmarkNormalizeURL/relative_URL_without_parent(OLD)-12                   100000              2320 ns/op             240 B/op          6 allocs/op

I'm working on optimizing whatwg-url.
valid_absolute_URL: 60% (before optimizing) -> 100% (after)
valid_relative_URL_with_parent: 45% -> 65%
valid_URL_without_scheme: 35% -> 65%
relative_URL_without_parent: 165% -> 250%

It seems that as long as whatwg-url is optimized properly, it can catch up with goada's performance.

@yzqzss

yzqzss commented Jul 8, 2025 •

Copy link
Copy Markdown
Collaborator

I created a performance improvement PR for whatwg-url. NationalLibraryOfNorway/whatwg-url#44

After the optimization, the overall performance of whatwg-url should be 60%~75% of goada (the benchmark on https://github.com/ada-url/url-various-datasets/blob/main/top100/top100.txt is 67%), which is still not as good as goada, but not very bad either.

@anonrig

anonrig commented Jul 11, 2025

Copy link
Copy Markdown

Would you mind updating your benchmarks for Ada? I’m curious how is the new version…

Conrad Bhuiyan-Volkoff and others added 5 commits July 10, 2025 23:25
Co-authored-by: yzqzss <30341059+yzqzss@users.noreply.github.com>
Co-authored-by: yzqzss <30341059+yzqzss@users.noreply.github.com>
Co-authored-by: yzqzss <30341059+yzqzss@users.noreply.github.com>
@otkd
otkd force-pushed the no-cpp-url-normalizer branch from 454abf7 to 3abc22b Compare July 11, 2025 04:02
@otkd

otkd commented Jul 11, 2025

Copy link
Copy Markdown
Author

@yzqzss great points throughout, thank you!

@yzqzss

yzqzss commented Jul 13, 2025

Copy link
Copy Markdown
Collaborator

Would you mind updating your benchmarks for Ada? I’m curious how is the new version…

Hmm, it appears to have no performance change.

const (
httpPrefix = "http://"
httpsPrefix = "https://"
ftpPrefix = "ftp://"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zeno don't support ftp yet, remove it?

@yzqzss

yzqzss commented Jul 13, 2025

Copy link
Copy Markdown
Collaborator

oh, whatwg-url allow us to use non-utf8 charsets to encode/decode URL path and query.
This is exactly what #370 needs!

@yzqzss

yzqzss commented Jul 14, 2025

Copy link
Copy Markdown
Collaborator

oh, whatwg-url allow us to use non-utf8 charsets to encode/decode URL path and query. This is exactly what #370 needs!

It does support custom charsets, but only 8bit charsets...
and you can't use different custom encodings for path and query.

@yzqzss

yzqzss commented Jul 14, 2025

Copy link
Copy Markdown
Collaborator
{
    // "https://a.com/你好.html?q=世界".encode("gbk")
    name:        "GBK encoded bare URL",
    rawURL:      "https://a.com/\xc4\xe3\xba\xc3.html?q=\xca\xc0\xbd\xe7",
    wantErr:     false,
    expectedURL: "https://a.com/%C4%E3%BA%C3.html?q=%CA%C0%BD%E7", // <- goada got
    // whatwg-url got: "https://a.com/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD.html?q=%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD"
    // (https://a.com/����.html?q=����)
},

I found out that whatwg-url will replace invalid runes with � instead of hard encoding them.

Since we cannot guarantee that all URLs in Zeno are valid utf8 strings before normalizing them, this � replacement behavior is the biggest issue preventing the PR from being merged, IMO.

@yzqzss

yzqzss commented Jul 15, 2025

Copy link
Copy Markdown
Collaborator

Thanks for your efforts on this PR @otkd.

But I have to close this PR due to this � replacement issue, sorry :(

@yzqzss yzqzss closed this Jul 15, 2025
@otkd

otkd commented Jul 16, 2025

Copy link
Copy Markdown
Author

Thanks for your efforts on this PR @otkd.

But I have to close this PR due to this � replacement issue, sorry :(

@yzqzss all good, appreciate you trying to make it happen!

@CorentinB CorentinB reopened this Sep 4, 2025
@CorentinB

CorentinB commented Sep 4, 2025 •

Copy link
Copy Markdown
Collaborator

This is interesting, @otkd would you mind reporting that whatwg-url runes bug to their repo? If you still want that PR to move forward of course, all good if you don't care anymore 😄
It seems to be the "only" (but big) blocker, no?

@yzqzss

yzqzss commented Sep 4, 2025

Copy link
Copy Markdown
Collaborator

I just made a CGO-free goada by using wazero.
https://github.com/yzqzss/goada-wasm/

it works!

image

Got 1x slow (compare to goada) on single-core, but almost no performence diffrence on multi-core.

@willmhowes

willmhowes commented Sep 4, 2025 •

Copy link
Copy Markdown
Collaborator

I just made a CGO-free goada by using wazero. https://github.com/yzqzss/goada-wasm/

it works!
image

Got 1x slow (compare to goada) on single-core, but almost no performence diffrence on multi-core.

@yzqzss Wow, incredibly interesting! As the resident goada defender, this is amazing!

However, I ask what problem we are trying to solve by removing cgo bindings at this point? Originally it was to close #245, but now I'm not sure?

(CC @CorentinB )

@yzqzss

yzqzss commented Sep 5, 2025

Copy link
Copy Markdown
Collaborator

Originally it was to close #245, but now I'm not sure?

I have no idea, I saw this PR being reopened, so I tried packaging ada as WASM lib.

@anonrig

anonrig commented Sep 5, 2025

Copy link
Copy Markdown

As one of the co-authors of Ada, I had an emotional roller coaster with this pull request lol

WASM will always be slower because of the cost of serializing string. I wrote the first version in Rust and compiled to WASM to be used by Node.js and saw similar performance characteristics. Gave a talk about this in case, it helps someone: https://youtu.be/v1lQK1jHZNQ

@CorentinB

Copy link
Copy Markdown
Collaborator

My 2 cts is that any requirement of CGO in this project is more pain than needed, see: #462

@willmhowes quite convince me yesterday by telling me that WASM is better than using the pure Go whatwg-url lib (if it was ever fixed) because we could be more up to date with the WASM route. So I think it would be a fine choice, even if my heart would still prefer to have as many things are possible in pure Go

@CorentinB

Copy link
Copy Markdown
Collaborator

As one of the co-authors of Ada, I had an emotional roller coaster with this pull request lol

Thanks a lot for your work!!! Ada is awesome.

@willmhowes

Copy link
Copy Markdown
Collaborator

Seems like we can put this PR back to rest 🙂

Thanks as always, @anonrig !

@willmhowes willmhowes closed this Sep 11, 2025
@anonrig

anonrig commented Sep 11, 2025

Copy link
Copy Markdown

Thank you for using it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants