v1.7.0

pg_hardstorage v1.7.0

Distilled for operators; the full commit-level detail is in the
CHANGELOG.

This is a minor release. Nothing on disk or on the wire changes; the
24-month schema promise is untouched, and a 1.6.0 binary reads every
repository this release writes (and the other way round). It carries every
fix reported against 1.6.0 (#87, #89, #90, #91, #95, #97, #98) and the
findings of a pre-release code review.

Throughput follows the storage, not the CPU

Backup, wal stream, restore and verify used to size their concurrency
from GOMAXPROCS, which since Go 1.25 follows the cgroup CPU limit. A pod
limited to one core uploaded two chunks at a time and read one; against an
object store that is one round trip per chunk, which capped a backup near
20 GB/h and a 2.5 TiB restore at about a day with the host idle (#90, #97).

Both directions now keep 16 operations in flight regardless of core count,
also within a single file, and the pool spans file boundaries so the
thousands of one-chunk files in a PGDATA no longer cost a round trip each.
Restores write files on a pool of 16 as well, and verify --full's scratch
restore no longer fsyncs (1,212 s → 55 s on a 68k-file backup). wal stream reads wal_sender_timeout from the walsender and keeps its status
replies inside it, so CloudNativePG's 5 s default no longer kills the
walsender under load.

Measured: S3 backup on two cores 40 → 82 MB/s (276 MB/s unconstrained);
restore of 1.3 GB through a 25 ms-per-request store 630 s → 41 s;
verify 34 → 408 chunks/s.

New: --upload-concurrency on backup and wal stream, --fetch-concurrency
on restore and verify (both 1..64, default 16). Memory per in-flight
read is one chunk.

Works on more stores

• S3-compatible stores that reject If-None-Match: * — Alibaba Cloud
  OSS answers the conditional PutObject with 400 NotImplemented, so
  repo init failed (#98). An endpoint that is not vouched for with
  ?conditional_put=native now takes that answer as the store's capability
  statement and emulates the conditional write with a read before it. The
  backup lease still has to be waived explicitly there (--allow-concurrent
  or allow_unenforceable_lease: true); see
  S3 repositories.
• WORM buckets shared between wal stream and base backups — a chunk
  already locked longer than the backup's deadline failed the backup, after
  hours (#95). The lock is read back and an equal-or-stronger lock is
  accepted. The writer additionally needs s3:GetObjectRetention. On Azure,
  re-locking in governance mode no longer shortens an existing lock.
• Lossy links — a chunk body that breaks off mid-read is fetched again
  (five attempts, with backoff), and a streamed manifest re-opens at the
  byte it reached. Against a proxy that cut 5 % of chunk bodies, 6 of 6
  restores completed byte-identical; with the old budget 4 of 6. A failed
  materialisation is now restore.failed, not internal.
• scp:// under the new concurrency stays within sshd's MaxSessions.
• A start checkpoint that takes longer than 90 s no longer trips the
  stream watchdog (own 30-minute window), and verify --full works on
  backups with tablespaces when run on the database host.

A tidy repository, bounded memory

• sftp:// and scp:// prune the empty directories repo gc leaves,
  like file:// already did (#89). scp needs rmdir on the remote host.
• audit export-bundle exports through a disk spool; heap use no longer
  grows with the window (30,000 events: 36 MiB → 2 MiB; 300,000 events:
  1,789 MiB → 405 MiB max RSS). The bundle format is unchanged byte for
  byte (#87).
• A file:// writer no longer races a concurrent prune on a busy disk.
• S3 multipart uploads send up to four parts in parallel.

Security

• Built with Go 1.26.10 and golang.org/x/net v0.62.0; the previous
  toolchain left twelve standard-library vulnerabilities reachable
  (net/http, http2, crypto/tls, crypto/x509, html/template,
  mime/multipart). Every pin — CI, .deb/.rpm builders — is 1.26.10.
• wal commands refused a deployment name that escaped the wal/
  prefix: wal push ../audit <segment> could write into the audit tree.
  Names with a path separator or control character are rejected with
  usage.bad_deployment.
• Redacting a malformed connection string could panic.
• go-jose 4.1.6 (three advisories).

Configuration and error codes

• An explicit keep_daily: 0 (and the other keep_* counts) in a
  deployment's retention block now means what it says; before, it was
  replaced by the default.
• Config-editing commands keep the comments, key order and permission bits
  of pg_hardstorage.yaml.
• A damaged HSREPO is verify.repo_marker_corrupt (exit 9) everywhere;
  read commands with an empty keyring report what is actually wrong
  (notfound.repo, …) instead of notfound.signing_key; wal fetch
  distinguishes a damaged chunk (verify.checksum_mismatch) from an
  unreadable one (wal.fetch.chunk_unreadable) and an absent one.
• repo check and repo audit list deployments that hold only archived
  WAL, with a wal_segments count.
• The Homebrew cask uses postflight_steps instead of the deprecated
  postflight block (#91).

Upgrade notes

• keep_daily: 0 (or keep_weekly, keep_monthly, keep_yearly,
  keep_fulls) now keeps none in that bucket. If a config carries a
  literal 0 only as a placeholder, remove the key before upgrading.
• S3 WORM writers need s3:GetObjectRetention in addition to the
  previous permissions; scp:// repositories need rmdir on the remote
  host.
• The first repo gc --apply on an sftp/scp repository after upgrading
  removes the directories earlier sweeps left.
• Defaults of 16 uploads and 16 reads in flight: on a store with
  per-request pricing or a narrow link, --upload-concurrency /
  --fetch-concurrency tune them down; scp:// is capped at 8 sessions.
• audit export-bundle needs temp space of about twice the uncompressed
  events under $TMPDIR.
• On an S3-compatible endpoint that rejects If-None-Match, a backup
  lease cannot be enforced; the refusal names the knobs
  (conditional_put=native to vouch for a store that does honour the
  header, --allow-concurrent or allow_unenforceable_lease to waive).
• The evaluation stack (docker-compose.yml) runs its S3 endpoint on
  pgsty/silo, a maintained fork of the MinIO server, because the
  minio/minio and minio/mc images are no longer pullable. The bucket,
  credentials and port 9000 are as before; the web console on 9001 is
  gone. Production MinIO deployments are unaffected — the S3 plugin talks
  to any S3 endpoint.

Known limits

• WAL segment manifests are unsigned; signing the chain needs a format
  change (planned for v1.8).
• The header-embedded PG backup_manifest still scales with size (v1.8).
• audit verify-bundle reads a bundle into memory and refuses an
  events.ndjson over 1 GiB; on Windows the export's signing input is
  still held in RAM.
• A PITR-restored cluster keeps archive_mode as configured; decide
  whether it may archive into the production repo.

How this release was tested

Gate	Result
unit tests under -race, all packages, including the Docker-backed storage suites (S3 on Silo/MinIO, SFTP, GCS, Azure)	pass
storage contract through an endpoint that rejects If-None-Match (the #98 path); two concurrent backups plus wal stream through it	pass: objects = references, no orphans, no staging leftovers
chaos: 5 % and 15 % of chunk bodies cut mid-read	6/6 restores byte-identical; a binary without the retry fails at the second drop
make test-stress (-count=10, 20 packages)	20/20 pass
crash suite T1–T5 on the release toolchain; T5 soaks with wal_sender_timeout=5s	21/21 pass; no walsender timeouts
integration suites against real PostgreSQL and storage containers	pass (Docker start timeouts under host IO pressure rerun)
1.6.0 ↔ 1.7.0 repository compatibility (list/show/verify/restore both ways, incremental by 1.7.0 on a 1.6.0 full, audit bundles both ways)	29/29 pass
#97 end to end: 1.3 GB, 18,984 chunk reads through a 25 ms-per-request store	restore 630 s → 41 s
#87: 300,000-event export	1,789 MiB → 405 MiB, bundles cross-verify
Patroni failover lane (make test-patroni: 3-node Spilo/etcd, failover ×9, gameday ×2, lifecycle, PITR across a failover, timeline chain, WAL history capture, wal stream from a demoted node, retention boot) plus the restore boot matrix PG 16–19 on CI	pass (CI, 25 min)
make test-stress -count=40 (17 packages), 38 fuzz targets × 75 s, mutation harness (33)	pass, no findings
govulncheck, cross-builds linux/amd64, linux/arm64, darwin/arm64, windows/amd64	pass
Windows unit subset (14 packages) and macOS unit subset on CI	pass
end to end on Alibaba Cloud OSS by the #98 reporter	pass
full CI workflow on the release commit (28 jobs: PG 15–18 integration, every storage backend, eval stack, packaging, Windows/macOS)	all green

Verifying the download

Every asset is signed with cosign (keyless, Sigstore) and covered by SLSA L3 build provenance (pg_hardstorage.intoto.jsonl); see SLSA L3 provenance for the cosign verify-blob and slsa-verifier invocations. Container images: ghcr.io/cybertec-postgresql/pg_hardstorage:1.7.0 (amd64, arm64).