v0.3.0

ppgrid 0.3.0

Turbo mode: continent-scale grids in minutes, not hours.

What's new

• --turbo — opt-in multithreaded pipeline sized to your RAM cap.
  Presets --turbo 16|32|64|128 (GB); bare --turbo = auto
  min(physical, cgroup) - 8 GB. Also --max-ram <GB> / --ram-gb <GB>
  (imply turbo) and --turbo-strict (error + exit 3 instead of warn when
  the shared path is infeasible).
• RAM budget + regimes — planning budget Bt = 0.85 * C - 4 GB
  (C = effective cap) picks a regime per run: A full in-RAM shared,
  B input in RAM + banded val/sup memmap, C shared all-memmap
  banded descent (input + val/sup file-backed memmap, 8 GB page-cache
  floor in its peak accounting), per_box per-box warp/write fallback
  taken when regime C has < 2 in-flight units (the only genuine low-RAM
  mode). A pre-check after gridding prints the cap/budget/regime actually
  used. Note --ram-gb / --max-ram cap the shared-field budget, not
  process RSS (measured: 39.4 GB RSS under a 20 GB budget).
• Byte-exact parallel writer — parallel tile reproject + parallel
  per-tile ZSTD write, gated by a compression oracle; any mismatch falls
  back to the byte-exact stock-codec parallel write (the serial
  per-band write is the last resort only if that writer's layout guards
  fail — on the reference GDAL 3.12.4 / libtiff 6.2 stack the oracle
  mismatches, so turbo runs take the stock parallel write). Output is
  sha256-identical to the serial path.
• MT read side — Pipeline(n_threads=N) (default 1 = the old serial
  path): threaded 2048px warp tiles + parallel CSV ingest for large files.
• cgroup cap fix — auto cap now walks the process cgroup hierarchy to
  the smallest finite memory.max, so capped systemd/user slices no longer
  plan a regime that OOMs the slice.

Benchmark

Full-AU, 16M points, --res 100 on marzuki-hydrogen (CPU-only):

pre-speed-pass --turbo 64 (regime A, same code): 30:45 (quiet) to 50:46 (loaded)
post --turbo 64 (regime A):                       2:33

~12-20x wall (the pre-PR number is load-dependent; best-case pre-PR is
~12x), peak RSS 43 GB under a 64 GB cap (50.4 GB sizing budget), and the
outputs are sha256-identical to the pre-PR anchors (value.tif and
support_km.tif). Also measured: per-box 3:20 at 8.5 GB RSS (the only
genuine low-RAM mode); regime C under --ram-gb 20: 6:25 pre-PR, 3:08
post, RSS 39.4 GB (--ram-gb caps the shared-field budget, not RSS).

Install

pip install ppgrid

Breaking / flag notes

• No breaking changes to the serial default, with one exception: a
  correctness fix in the banded multi-level descent. The support blend
  computed (1-a)*pv*ps instead of (1-a)*ps (buffer reuse).
  support_km.tif for multi-level grids may change vs 0.2.2 (now
  correct); value.tif is unchanged.
• New CLI flags: --turbo [16|32|64|128], --max-ram <GB>,
  --ram-gb <GB>, --turbo-strict.
• New kwarg: Pipeline(n_threads=...) (default 1, serial).
• Turbo infeasibility: default is [warn] + continue on the per-box path
  (exit 0, correct output); --turbo-strict makes it an error (exit 3).