Skip to content

v0.11.0

Latest

Choose a tag to compare

@jacovdbergh jacovdbergh released this 09 Jun 06:23

Fixed

  • Inputs whose dense data is spread across a very large extent (e.g. a few
    small dense regions scattered across a whole country) no longer OOM during
    the build step. The counting grid has a fixed resolution, so over a huge
    extent a single grid cell can cover kilometres of ground and collect far
    more than the chunk target — the planner then emitted one enormous chunk per
    such cell, and the build loaded it whole into memory. The chunk planner now
    sub-splits an over-target finest-grid cell into deeper octree sub-chunks
    (routed during distribute via a deeper descent), and the build step gained a
    spill path that sub-divides any chunk whose actual point distribution still
    exceeds the per-chunk memory budget. The build and merge steps now also bound
    how many chunks they hold in memory at once — concurrency scales with the
    memory budget (and cores) instead of running one chunk per core regardless of
    budget, so peak build memory tracks --memory-limit from ~1 GB up to large
    multi-core hosts. Together these guarantee bounded build memory regardless of
    how points concentrate, letting large, geographically sparse datasets convert
    within a small --memory-limit.
  • Such inputs no longer produce a wildly over-fragmented octree (millions of
    tiny, nearly-empty deep nodes). When the build had to split a region deep for
    memory reasons, those deep voxels became permanent output nodes even where the
    data was sparse, bloating the node count ~100× and, in turn, the writer's
    memory. The bottom-up build now collapses any subtree whose points fit a
    single leaf into one node, so output node depth tracks point density rather
    than the build's memory-driven split depth. The writer's per-node bookkeeping
    was also reworked to keep a single key+count list instead of several parallel
    copies, so it scales to large node counts without holding gigabytes of index.
  • The writer no longer OOMs when producing very large outputs. It used to
    encode a whole memory-budget-sized batch of nodes before compressing, holding
    the entire batch's encoded bytes (which could be ~half the budget) resident
    alongside the compressor's working set. It now streams a fixed-size window of
    points at a time — encode in parallel, compress in order, free, advance — so
    the writer's peak memory is a small constant independent of the output size or
    --memory-limit.

Full Changelog: v0.10.1...v0.11.0