v1.6.0

pg_hardstorage v1.6.0

Backup, restore and repo gc stay within bounded memory on 100 TB databases, repo gc leaves a clean repository, and a validation pass closed leaks and security gaps. Includes v1.5.1. No on-disk or on-the-wire changes.

Highlights

• Bounded memory at 100 TB: streaming manifests, disk-spilling sets, gc fence checkpoints, a restore resume log, bounded backup compare, audit, wal list/audit and logical sinks; the PostgreSQL backup_manifest ceiling is now 1 GiB.
• Clean repo gc: empty directories pruned, crash leftovers reaped; no way found to delete a referenced chunk.
• Security: replayed manifests refused, repo-URL credentials scrubbed from errors/events, the restore boot test no longer archives into the production repo, key-file mode checks.
• Leaks: abandoned backup lease, scheduler relaunch of wedged tasks, zstd encoder footprint.
• Issues: #70, #72, #73, #75, #76, #77, #79, #80, #81 (and #69 in 1.5.1).

Known limits: WAL segment manifests are unsigned; audit export-bundle and the header-embedded PG backup_manifest still scale with size; sftp/scp keep empty directories after gc.

Release notes: https://docs.pghardstorage.org/release-notes/v1.6/ · Full details: CHANGELOG.md

Changelog

• 4eb90d2: Merge PR #74 (integration/v1.5.1, includes #67): fixes #69, #70 (@postgresql007)
• cb7da44: Merge fixes_20261001: bound sftp/scp data-plane reads (#72) (@postgresql007)
• f19f5f5: Merge origin/dev_20261017 into large_manifest_20261021 (@postgresql007)
• 20e09ff: Merge origin/quality_improvements into large_manifest_20261021 (@postgresql007)
• 38a6ed1: docs(changelog): #69 and #70 (@postgresql007)
• 8395db3: docs(changelog): note the sshd fixture nofile cap (@postgresql007)
• ba19bcf: feat(#79): --tablespace-mapping-all for restore and recovery drill (@postgresql007)
• ecac5f8: feat(backup): streaming manifest reader/verifier and streamed restore (@postgresql007)
• 1be2f24: feat(repo): bounded-memory reference sets and streamed manifest walks (@postgresql007)
• 4dff128: feat: spilled adoption set and GC fence pins, backup --chunk-size, docs (@postgresql007)
• ab577cd: feat: spooled manifest writes, bounded-memory verify/integrity/list/hints (@postgresql007)
• 93055e7: feat: stream manifests through every consumer; S3 multipart put (@postgresql007)
• 8d563b8: fix(#80): schedule reports what the running agent actually loaded (@postgresql007)
• a8ce0f0: fix(#80): write agent-loaded-schedule.json with fsutil.WriteFileAtomic (@postgresql007)
• 557d7ac: fix(#81): manifests over 1 GiB work in repair, bundle, kms shred, tarsink and compare (@postgresql007)
• d230072: fix(backup): bound memory on the backup, retention and verify paths for 100 TB (@postgresql007)
• 03d8060: fix(backup): duplicate-key walk hung on malformed JSON; stream reader rejects trailing data (@postgresql007)
• 4072b00: fix(backup): stream attestation hash, KEK rotation and per-file chunk refs (@postgresql007)
• 834c6d0: fix(bundle): export --include-wal derives the WAL a backup needs and carries its chunks (#77) (@postgresql007)
• ef2f90d: fix(bundle): keep MaxEntryBytes in the oversized-entry error (@postgresql007)
• 7c694a6: fix(bundle): stream manifests on export and import (@postgresql007)
• 48559ed: fix(cli): cap scrub mismatch lists and drive heal from a spill set; docs and scale test (@postgresql007)
• 44357be: fix(cli,compliance): stop loading whole manifests where headers suffice (@postgresql007)
• dafa34c: fix(events): wal stream and agent task events name their deployment (#70) (@postgresql007)
• 207f5f8: fix(integrity): O(1)-memory chunk sampling (@postgresql007)
• d60138c: fix(leaks): release abandoned backup lease, stop relaunching wedged tasks, cap zstd state; add leak census tests (@postgresql007)
• d7b5509: fix(memory): bound compare, audit, wal gap lists, record lists and logical sinks on huge manifests (@postgresql007)
• 2ee732e: fix(release): publish the Homebrew cask through a PR to the tap (@postgresql007)
• 5e3a85f: fix(repo gc): prune empty dirs, honour spill dir, reap crash leftovers; add property tests (@postgresql007)
• 04f808f: fix(repo): bound memory in gc, sweep, wal-prune, wipe, replicate and checks for 100 TB (@postgresql007)
• 8049772: fix(repo): one fence checkpoint per sweep window; stream replica sidecars; bounded scrub heal (@postgresql007)
• 6552b9a: fix(restore): bound memory on the restore and PITR paths for 100 TB (@postgresql007)
• 5ca92da: fix(restore): bounded resume state and streaming partial restore (@postgresql007)
• 7523ac2: fix(security): refuse replayed manifests, scrub repo-URL secrets, harden key files and restore (@postgresql007)
• 81f3741: fix(sftp): bound the subsystem handshake in dialLocked (#73) (@postgresql007)
• ca7e98a: fix(sftp): single reconnect gate so the no_reconnect mutation covers read-guarded paths (@postgresql007)
• 0e3d495: fix(spill): disk-space preflight, tmpfs warning and hot cache for re-adds (@postgresql007)
• b21ed4e: fix(spill,storage): bound Bloom filter size; stream fs and scp listings (@postgresql007)
• a0b942b: fix(testkit): cap RLIMIT_NOFILE for the sshd fixtures (sftp/ssh-exec) (@postgresql007)
• 382101b: fix(wal): detect the shutdown keepalive spin over slow links (#69) (@postgresql007)
• 689a4b5: fix(wal,audit): O(1)-per-segment wal list/audit, bounded audit search and head (@postgresql007)
• 05461a7: fix(walprune): read the WAL frontier from a manifest as a token stream (@postgresql007)
• 2504a71: fix: bound uint32->int32 timeline conversions (CodeQL) (@postgresql007)
• c071179: fix: cap unbounded reads of repo objects, HTTP bodies, tool output and server messages (@postgresql007)
• 671b1e9: fix: harden repository mutations and CLI safety (@postgresql007)
• 1fb19ae: fix: per-deployment backup options (#75), per-URL s3 credentials (#76), docs (@postgresql007)
• 2e4cf16: fix: read manifest headers only where Files is not needed (@postgresql007)
• b4325c2: fix: spill-set hardening found by the pre-release review (@postgresql007)
• e33d110: perf(spill): pool the fence-block buffer; add opt-in huge-repo builder test (@postgresql007)
• b60deac: release: cut 1.6.0 (changelog, release notes, nav) (@postgresql007)
• 0abd430: release: v1.5.1 — cut the changelog (@postgresql007)
• aadcdb4: storage/sftp,scp: bound no-deadline data-plane reads (P2) (@postgresql007)
• a5dafcf: storage/sftp,scp: end-to-end stress tests for the live-peer wedged read (@postgresql007)
• 970a918: storage/sftp,scp: release the read-guard timer on the healthy path (@postgresql007)
• dc88b3f: storage/sftp: release the read-guard timer in guardedOpen too (@postgresql007)
• b695fd2: test(backup): large-manifest scale proof (opt-in) (@postgresql007)
• 1dec1df: test(backup): scale proof adds disk guard and GC mark over the huge manifest (@postgresql007)
• 0b8582e: test(fsutil): exempt spill run writer and leakcheck heap dump from close-check guard (@postgresql007)
• 4a39cbf: test(repo): opt-in gc scale proof over a huge manifest (@postgresql007)

All artefacts are cosign-signed (keyless / Sigstore). Verify with:

cosign verify-blob \
  --certificate-identity-regexp 'https://github.com/cybertec-postgresql/pg_hardstorage' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate <artefact>.pem --signature <artefact>.sig \
  <artefact>