v3.7.0

The crawler keeps up with the storage instead of re-copying it.

A periodic reconcile now writes only what changed. Crawl states stopped lying. Two failures that only appear at production scale were found under rehearsal and fixed before release.

Upgrading

Schema changes are additive and helm rollback to the 3.6.0 chart works against the migrated database.

The first boot on a 3.6.0 index repairs every bucket's search index, one bucket at a time (FTS_REBUILD_CONCURRENCY, default 1), so a large deployment becomes searchable bucket by bucket over the first minutes. Full reconciles of seeded buckets are spread across their interval rather than all falling due at once.

Crawls do not start while the index volume has under MIN_FREE_DISK_PCT (10) percent free — check your PVC before upgrading.

Helm: the default persistence.size moved from 5Gi to 30Gi. A 1M-object index costs 1.0-1.4 GB including the search index, so 15.5M deep keys measures about 21 GB, and the disk guard needs headroom on top. An existing release's PVC is patched in place, which only works when the StorageClass allows volume expansion — otherwise pass your current size (--set persistence.size=5Gi). The PVC now carries helm.sh/resource-policy: keep, so helm uninstall no longer deletes the index.

What changed

Reconciles stopped re-copying the index. A recrawl of an unchanged 1M-object bucket writes 31 MB instead of 1,101 MB and rewrites 0 rows instead of 1,000,000. A 9.9M-object reconcile writes 36 MB with a 326 MB memory peak, where 3.6.0 was OOM-killed at 1 GiB. Both figures are single-bucket measurements.

Crawl states became truthful. No false complete, no faked freshness. A restart mid-crawl no longer leaves a partial index reporting itself complete: completed prefixes are recorded per generation and an unfinished crawl is resumed rather than seeded as fully crawled. New interrupted and degraded states, plus last_error and last_attempt_at.

First crawls got faster. Single-prefix buckets are split into sub-prefixes so every listing thread works: 2.0× on a 9.9M-object bucket, and 63× on a pathological deep-then-wide layout that previously stalled.

Listing concurrency is bounded across all buckets, not per bucket. The pools multiplied — twelve concurrent bucket tasks each opening sixteen listing threads is 192 listers in flight, and the memory lives in their buffers. A 19-bucket, 15.5M-object deployment on a 1 GiB pod was OOM-killed with repeated restarts and a persistently overflowing connection pool. One budget (LIST_CONCURRENCY, default 16, clamped to S3_MAX_POOL_CONNECTIONS) now covers every background-crawler listing path. Interactive browsing stays outside it, so the UI is unaffected while a crawl runs.

Delta discovery can page past 2,000 children. A folder was abandoned as oversized after two 1,000-key pages, so a bucket whose datasources hold 2,500 interval prefixes truncated its discovery walk on every delta and never advanced its freshness timestamp — deterministically, not intermittently. DELTA_NODE_MAX_PAGES is now 3.

Fixes: image previews load again (#27 — the CSP did not name the S3 endpoint, so presigned URLs were blocked); LDAP is selectable on the login form (#35); local accounts can change their own password (#29), and a wrong current password shows the error instead of signing you out; the password option is hidden from S3-key sessions, which have no local account; Cloudflare R2 setup names the Admin token tier that ListBuckets requires (#44); S3-key login and endpoint pinning gained regression coverage (#1, #8).

CI: backend, frontend, MCP, CLI and a browser critical path now run on every pull request.

Validation

Rehearsed on a production-shaped lab: 19 buckets, 15.5M objects, 1 GiB pod, 40 ms list latency, in-place upgrade from 3.6.0.

	Pre-fix candidate	3.7.0
Container restarts	11	0
OOM kills	yes	none
Peak anonymous memory	888 MiB	605 MiB
Connection-pool overflow	sustained	0
Objects vs provider truth	—	15,500,000 / 15,500,000
Upgrade rollout	—	11 s

Zero lock errors, zero stalled crawls, zero crawl retries. Delta certification after the paging fix: three consecutive clean cycles on the 9.9M-object bucket, all 19 buckets certified, peak anonymous memory 234 MiB. A brand-new partition beyond page two was confirmed discovered and indexed by a delta, not merely certified.

Known limitations

• This validates recovery from a damaged 3.6.0 index, not an upgrade from a fully indexed one. The 3.6.0 baseline in the rehearsal OOM-killed itself and left the largest bucket marked complete holding 1.32M of 9.9M objects — the exact failure this release fixes. 3.7.0 repaired it to 9.9M. 3.6.0 could not build a complete index on that workload.
• Peak working set reached 987 MiB against an 800 MiB target. It is dominated by reclaimable page cache from reading 15.5M rows: the cgroup reported oom=0, oom_kill=0 and about 6.4 s of cumulative memory-pressure stall. Anonymous memory, which is what an OOM kill turns on, peaked at 605 MiB.
• No live Cloudflare R2 test was performed. The R2 token-tier guidance is verified against Cloudflare's published permission table and deterministic tests, not against a real Admin-tier token.
• Freshness on wide layouts is governed by FULL_CRAWL_INTERVAL, not the delta cadence. Delta discovery follows the newest DELTA_NEWEST_K (2) branches at any level wider than DELTA_BRANCH_FANOUT (8), so new data outside those branches is picked up by the full reconcile.

Full lab methodology and per-run numbers are in benchmark/prodlab/RESULTS.md (Run 12).