Skip to content

SSH transfer optimizations: source-side compression and stale staging purge (v3.10.0) - #3

Merged
Asimatasert merged 3 commits into
masterfrom
feat/ssh-transfer-optimizations
Jul 16, 2026
Merged

SSH transfer optimizations: source-side compression and stale staging purge (v3.10.0)#3
Asimatasert merged 3 commits into
masterfrom
feat/ssh-transfer-optimizations

Conversation

@Asimatasert

Copy link
Copy Markdown
Owner

Summary

Two features for SSH dump jobs on slow links, plus a fix they surfaced:

  • --compress-where source|target (default target, old behavior): with zstd/xz/bzip2, compress the dump on the source host before the transfer, so slow uplinks move the compressed artifact instead of the raw dump (20x+ smaller in practice). Falls back to target-side compression with a warning when the tool is missing on the source or compression fails — never blocks the backup. gzip is unaffected (pg_dump -Z already compresses in-format at the source).
  • --from-stale <time> (default 72h, 0 disables): with --from-keep 0, purge this job's leftover dumps older than <time> from the source staging dir before dumping. Failed or interrupted runs never reach the post-transfer cleanup, so their dumps piled up in the source /tmp forever.
  • Fix: the batch runner's YAML→args key list was missing the new options; also widened the --from-keep retention and fetch globs from _*.dump to _*.dump* so compressed staging artifacts are matched.

Both options are validated (enum / strict time format — config and YAML values pass through the same validation), supported in config files and jobs YAML, and documented in --help and all four READMEs.

Testing

Verified end-to-end against a postgres:16 + sshd container:

  • source-side compression: compressed on source, transferred as .dump.zst, pg_restore -l integrity check on the decompressed artifact, source staging cleaned
  • missing-tool fallback: warning + raw transfer + target-side compression
  • stale purge scoping: only this job's old files removed; fresh files and other jobs' files kept
  • --from-keep 1 retention with compressed staging names
  • default path (no --compress-where) byte-for-byte old behavior
  • YAML mapping: invalid compress_where/from_stale values in a jobs YAML are rejected by the child validation (proves the keys flow end-to-end)
  • ./build.sh --check green, bash -n clean

…v3.10.0)

- --compress-where source|target (default: target, old behavior): with
  zstd/xz/bzip2, compress the dump on the SSH source host before the
  transfer, so slow uplinks move the compressed artifact instead of the
  raw dump. Falls back to target-side compression with a warning when
  the tool is missing on the source or compression fails. gzip is
  unaffected (pg_dump -Z already compresses in-format at the source).
- --from-stale <time> (default: 72h, 0 disables): with --from-keep 0,
  purge this job's leftover dumps older than <time> from the source
  staging dir before dumping. Failed or interrupted runs never reach
  the post-transfer cleanup, so their dumps piled up in /tmp forever.
- Widen the --from-keep retention and fetch latest-dump globs from
  _*.dump to _*.dump* so compressed staging artifacts are matched.
- Both options are validated (enum / strict time format), supported in
  config files and jobs.yaml (save_job), and documented in --help and
  all READMEs.

Verified end-to-end against a postgres:16+sshd container: source-side
compression (pg_restore -l integrity check on the transferred artifact),
missing-tool fallback, stale purge scoping (only this job's old files),
from-keep retention with compressed names, and unchanged default-path
behavior.
The batch runner translates YAML defaults/job keys to child CLI args via
an explicit key list in yaml.sh, which the new options were missing from
— a compress_where/from_stale set in a jobs YAML was silently ignored.
@Asimatasert
Asimatasert merged commit 3a82746 into master Jul 16, 2026
2 checks passed
@Asimatasert
Asimatasert deleted the feat/ssh-transfer-optimizations branch July 16, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant