Repository navigation
Replies: 1 comment
|
I checked the current dependency state rather than the benchmark itself, and this is still actionable on current Katana currently pins
For anyone hitting this before Katana changes its dependency graph, the source-build workaround is therefore: git clone https://github.com/projectdiscovery/katana
cd katana
go get github.com/odvcencio/gotreesitter@v0.52.0
go mod tidy
go build ./cmd/katanaI did not independently reproduce the memory/RSS numbers in the report, so I would keep those attributed to the reproduction above. What I can confirm from current source is that the old parser version is still selected and there is no open Katana PR for this bump at the moment. A minimal upstream fix can stay dependency-only if the project's tests/real crawl comparison confirm the reporter's result. The suggested input-size guard looks like useful defense-in-depth, but I would keep that separate from the dependency fix so the parser regression and hardening are independently reviewable. |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Since PR #1570 (cgo-free jsluice, shipped from v1.6.0)
-jsluiceparses inline<script>content and JS responses withgithub.com/Mzack9999/jsluice@30114a312f98, which depends on a March 2026 snapshot of the pure-Go tree-sitter port,github.com/odvcencio/gotreesitter v0.6.1-0.20260306002001-fbe5983c6f41. That snapshot's JavaScript parsing is super-linear on string literals that contain escaped quotes. A single inline script with a few thousand\"sequences (theself.__next_f.push([1,"..."])flight payload every Next.js page embeds) takes katana past 6 GiB in about 20 s and it is OOM-killed after emitting one line.Bumping the transitive dependency to
gotreesitter v0.52.0fixes it with no code change: same crawl, 599 MiB peak, 5042 lines, 427 unique endpoints in 30 s, and the extracted URL set is identical to the original cgo jsluice on the same page.Reproduction (jsluice alone, no network)
Measured (Ubuntu 24.04, x86_64, Go 1.24 / 1.27, 6 GiB cgroup cap):
Reproduction (katana)
Under
systemd-run --scope -p MemoryMax=6G: release build v1.7.0 is killed at 6 GiB after 1 output line; the same source built withgo get github.com/odvcencio/gotreesitter@v0.52.0peaks at 599 MiB and completes (5042 lines, 427 unique endpoints). The memory grows while the first response (the index page) is parsed, before any depth-2 request.Plain
-js-crawlwithout-jsluiceon the same target stays at 43 MiB, which is how we bisected it to the parser.Suggested fix
One-line dependency bump in
go.mod(go get github.com/odvcencio/gotreesitter@v0.52.0; the current pin is only reachable as// indirectthrough Mzack9999/jsluice, but MVS honours the higher requirement). The fork builds unchanged against v0.52.0. Optionally also implement the size guard that the comment onpkg/utils/jsluice.go::ExtractJsluiceEndpointspromises ("skip lines that are too long"), which is not implemented today, so a future parser regression cannot take the whole crawler down.Happy to open the PR if you prefer; katana v1.7.0,
go version -mshowsgithub.com/odvcencio/gotreesitter v0.6.1-0.20260306002001-fbe5983c6f41.All reactions