Repository navigation
Conversation
|
Could you please append these test cases to url_test.go? |
|
Hmm, we ran out of actions quota? |
func BenchmarkNormalizeURL(b *testing.B) {
tests ... // copy from TestNormalizeURL()
for _, tc := range tests {
b.Run(tc.name, func(b *testing.B) {
var parentURL *models.URL
if tc.parentURL != "" {
parentURL = &models.URL{Raw: tc.parentURL}
parentURL.Parse()
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
url := &models.URL{Raw: tc.rawURL}
err := NormalizeURL(url, parentURL)
if tc.wantErr && err == nil {
b.Errorf("NormalizeURL() expected error but got none")
} else if !tc.wantErr && err != nil {
b.Errorf("NormalizeURL() error = %v", err)
}
}
})
b.Run(tc.name+"(OLD)", func(b *testing.B) {
var parentURL *models.URL
if tc.parentURL != "" {
parentURL = &models.URL{Raw: tc.parentURL}
parentURL.Parse()
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
url := &models.URL{Raw: tc.rawURL}
err := NormalizeURLOld(url, parentURL) // copy old NormalizeURL()
if tc.wantErr && err == nil {
b.Errorf("NormalizeURL() expected error but got none")
} else if !tc.wantErr && err != nil {
b.Errorf("NormalizeURL() error = %v", err)
}
}
})
}
}go test -run='^$' -benchmem -benchtime=100000x -bench 'BenchmarkNormalizeURL' github.com/internetarchive/Zeno/internal/pkg/preprocessorUnfortunately, in general, whatwg-url runs at 40-60% of the speed of goada on my i5-12450H laptap. (applied the code changes from my review.) |
I'm working on optimizing whatwg-url. It seems that as long as whatwg-url is optimized properly, it can catch up with goada's performance. |
|
I created a performance improvement PR for whatwg-url. NationalLibraryOfNorway/whatwg-url#44 After the optimization, the overall performance of whatwg-url should be 60%~75% of goada (the benchmark on https://github.com/ada-url/url-various-datasets/blob/main/top100/top100.txt is 67%), which is still not as good as goada, but not very bad either. |
|
Would you mind updating your benchmarks for Ada? I’m curious how is the new version… |
Co-authored-by: yzqzss <30341059+yzqzss@users.noreply.github.com>
Co-authored-by: yzqzss <30341059+yzqzss@users.noreply.github.com>
Co-authored-by: yzqzss <30341059+yzqzss@users.noreply.github.com>
454abf7 to
3abc22b
Compare
|
@yzqzss great points throughout, thank you! |
Hmm, it appears to have no performance change. |
| const ( | ||
| httpPrefix = "http://" | ||
| httpsPrefix = "https://" | ||
| ftpPrefix = "ftp://" |
There was a problem hiding this comment.
Zeno don't support ftp yet, remove it?
|
oh, whatwg-url allow us to use non-utf8 charsets to encode/decode URL path and query. |
It does support custom charsets, but only 8bit charsets... |
{
// "https://a.com/你好.html?q=世界".encode("gbk")
name: "GBK encoded bare URL",
rawURL: "https://a.com/\xc4\xe3\xba\xc3.html?q=\xca\xc0\xbd\xe7",
wantErr: false,
expectedURL: "https://a.com/%C4%E3%BA%C3.html?q=%CA%C0%BD%E7", // <- goada got
// whatwg-url got: "https://a.com/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD.html?q=%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD"
// (https://a.com/����.html?q=����)
},I found out that whatwg-url will replace invalid runes with � instead of hard encoding them. Since we cannot guarantee that all URLs in Zeno are valid utf8 strings before normalizing them, this � replacement behavior is the biggest issue preventing the PR from being merged, IMO. |
|
Thanks for your efforts on this PR @otkd. But I have to close this PR due to this � replacement issue, sorry :( |
|
This is interesting, @otkd would you mind reporting that whatwg-url runes bug to their repo? If you still want that PR to move forward of course, all good if you don't care anymore 😄 |
|
I just made a CGO-free goada by using wazero. it works!
Got 1x slow (compare to goada) on single-core, but almost no performence diffrence on multi-core. |
@yzqzss Wow, incredibly interesting! As the resident goada defender, this is amazing! However, I ask what problem we are trying to solve by removing cgo bindings at this point? Originally it was to close #245, but now I'm not sure? (CC @CorentinB ) |
I have no idea, I saw this PR being reopened, so I tried packaging ada as WASM lib. |
|
As one of the co-authors of Ada, I had an emotional roller coaster with this pull request lol WASM will always be slower because of the cost of serializing string. I wrote the first version in Rust and compiled to WASM to be used by Node.js and saw similar performance characteristics. Gave a talk about this in case, it helps someone: https://youtu.be/v1lQK1jHZNQ |
|
My 2 cts is that any requirement of CGO in this project is more pain than needed, see: #462 @willmhowes quite convince me yesterday by telling me that WASM is better than using the pure Go whatwg-url lib (if it was ever fixed) because we could be more up to date with the WASM route. So I think it would be a fine choice, even if my heart would still prefer to have as many things are possible in pure Go |
Thanks a lot for your work!!! Ada is awesome. |
|
Seems like we can put this PR back to rest 🙂 Thanks as always, @anonrig ! |
|
Thank you for using it! |


Step towards multiplatform, in this case simpler build on mac
#245