Skip to content

v1.0.6

Choose a tag to compare

@Goldziher Goldziher released this 19 Jul 12:19

Fixed

  • map() / map_urls() no longer materialize the entire sitemap tree before
    applying map_limit.
    The limit previously bounded only the returned slice,
    not peak memory: a large sitemap-index host could drive the process into
    multiple GB and be OOM-killed even with a small map_limit set. map_limit
    and the exclude_paths / map_search filters are now compiled once and
    threaded through the sitemap fetch loop — entries are filtered as they are
    parsed, and both child-sitemap fetching and per-child parsing stop once the
    limit is reached. Peak memory is bounded to roughly the limit plus a single
    child sitemap. (crates/crawlberg/src/map.rs, crates/crawlberg/src/sitemap.rs)
    Closes #33.

Build

  • Refreshed in-major dependencies (deno_core 0.408, uuid 1.24) and lock files.
  • Internal maintenance: pruned stale TODO markers, closed remaining todo gaps,
    and added the ai-rulez Poly commit hooks.

Zig

Add to your build.zig.zon:

.dependencies = .{
    .crawlberg-zig = .{\n        .url = \"https://github.com/xberg-io/crawlberg/releases/download/v1.0.6/crawlberg-zig-v1.0.6.tar.gz\",\n        .hash = \"crawlberg-1.0.6-IoY1Cgv2pyBxPz32zlPF4jel3VK8XU8gdoF-VzfidtIv\",\n    },\n},\n```\n