v1.0.6
Fixed
map()/map_urls()no longer materialize the entire sitemap tree before
applyingmap_limit. The limit previously bounded only the returned slice,
not peak memory: a large sitemap-index host could drive the process into
multiple GB and be OOM-killed even with a smallmap_limitset.map_limit
and theexclude_paths/map_searchfilters are now compiled once and
threaded through the sitemap fetch loop — entries are filtered as they are
parsed, and both child-sitemap fetching and per-child parsing stop once the
limit is reached. Peak memory is bounded to roughly the limit plus a single
child sitemap. (crates/crawlberg/src/map.rs,crates/crawlberg/src/sitemap.rs)
Closes #33.
Build
- Refreshed in-major dependencies (
deno_core0.408,uuid1.24) and lock files. - Internal maintenance: pruned stale TODO markers, closed remaining todo gaps,
and added the ai-rulez Poly commit hooks.
Zig
Add to your build.zig.zon:
.dependencies = .{
.crawlberg-zig = .{\n .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.6/crawlberg-zig-v1.0.6.tar.gz\",\n .hash = \"crawlberg-1.0.6-IoY1Cgv2pyBxPz32zlPF4jel3VK8XU8gdoF-VzfidtIv\",\n },\n},\n```\n