v1.2.0

Backfort v1.2.0 — Docker, Kubernetes & Safer Recovery

Backfort now runs as a containerized backup job or a Kubernetes workload, while keeping the same recovery-first approach: explicit sources, verified bundles, independent copies, and staged restores.

This release adds Docker distribution, Helm deployment for PVC-file backups, expanded deployment documentation, and an important fix for symbolic-link recovery.

🐳 Docker distribution

• Added a Debian-based, multi-stage Docker image with GNU tar and the tools needed for compression, encryption, signing, cloud storage, and Docker Compose integration.
• Added a hardened Compose deployment example with read-only configuration and source mounts, persistent state and working storage, a read-only root filesystem, and privilege-escalation protection.
• The default command is doctor. Backfort remains a one-shot CLI—not a daemon or an in-container scheduler.
• Root-capable execution supports ownership and metadata recovery. A narrower non-root files-only deployment is documented.

The release workflow now builds and smoke-tests linux/amd64 and linux/arm64 images before publishing to GHCR. It includes OCI metadata, build provenance, SBOM generation, immutable exact-version tags, and moving major/minor/latest aliases.

Optional Docker Hub publication is enabled through the DOCKERHUB_USERNAME and DOCKERHUB_TOKEN repository secrets.

☸️ Kubernetes deployment

Added a Helm 3 chart for Kubernetes 1.31+:

• Explicit, read-only source PVC mounts.
• Persistent state and a shared Backfort lock.
• Local backup PVCs or offsite destinations through rclone.
• ConfigMap configuration and references to existing Secrets.
• Backup and prune CronJobs, suspended by default.
• Manual Jobs for diagnostics, full verification, and staged recovery.
• Single-Pod execution, bounded deadlines, termination grace periods, and no automatic retries.
• Guardrails against source, state, backup, and recovery claim overlap.
• Manual restores restricted to an explicit recovery PVC and a safe target beneath /restore.

The default deployment grants no Kubernetes API token, RBAC permissions, Docker socket, hostPath mounts, or privileged-container access. An optional non-root profile is included.

Validate the configuration, create a backup, and complete a recovery drill before enabling schedules.

🔧 Symbolic-link recovery fix

Fixed file-source archive transforms incorrectly adding the internal data/ prefix to symbolic-link targets. This could produce broken links after an otherwise successful restore.

New backups preserve relative, absolute, and dangling symbolic-link targets when follow_symlinks: false. Archive member names and hard-link references retain the required internal prefix, and explicit dereferencing remains supported.

Existing backup bundles are not rewritten automatically. Create a fresh recovery point and inspect symbolic-link targets when restoring older backups.

🧪 Testing and automation

• Added Docker-image smoke tests covering diagnostics, backup, full verification, staged restore, and invalid configuration.
• Added Helm rendering and negative configuration tests.
• Added a dedicated Kubernetes workflow with real-image integration in a disposable kind cluster.
• Kubernetes integration covers metadata recovery—including ownership, ACLs, xattrs, and SGID—read-only sources, cross-Job locking, nonempty-target refusal, invalid configuration, pruning, and non-root operation.
• Added symbolic-link and hard-link regression tests, including explicit dereferencing.
• Extended Dependabot configuration with Docker base-image updates.

📚 Documentation

Expanded README and Wiki guidance for Docker and Kubernetes deployment, persistent storage, Secrets, scheduling, recovery, and troubleshooting.

Added ready-to-adapt Kubernetes values examples for local PVC storage, rclone destinations, non-root execution, and isolated restore Jobs, plus architecture decision records and updated project operating instructions.

Upgrade notes and scope

The native CLI, Backfort YAML schema, and recovery-bundle format remain unchanged.

Kubernetes support covers files on explicitly mounted PVCs, not whole-cluster backup. It does not provide Kubernetes resource discovery, CSI snapshot orchestration, or native database dumps from Pods.

Reading a live database PVC is not an application-consistent backup. Prepare a consistent export or clone through an application-controlled procedure.

Review storage-driver permissions, locking semantics, RWO/RWOP access modes, and working-space capacity before deployment. CronJob concurrency policies do not coordinate separate CronJobs or manual Jobs; the persistent Backfort lock remains essential.

Docker Compose database jobs continue to require a deliberately granted host Docker socket and matching project paths. They are not Kubernetes-native database adapters.

v1.1.0

v1.1.0

Bug-fix release. Four defects were found in review, each capable of producing a backup that looked successful while silently missing data, or a notification that silently failed to retry. All four are fixed with new regression tests, including two run against a real Docker daemon — one of them was verified to actually catch the regression by reverting the fix and confirming the test fails.

Fixed

A backup could silently report success while missing data

• A source file Backfort could not read (permission denied, an I/O error) used to disappear quietly from the archive while the run was still logged as backup-succeeded. tar --create no longer runs with --ignore-failed-read; GNU tar's own exit status now tells a real read failure (fails the job) apart from the tolerated race of a file changing mid-read (still a warning-only success). Applies to file-source jobs and to Compose named-volume/bind-mount snapshots alike.
• The archive sanitizer was silently dropping any file or directory whose name merely started with a space — a false positive, since every archive member is transform-prefixed with the literal string data, which never itself starts with whitespace. That check is removed. A name that genuinely can't be represented (a literal newline, carriage return, tab, or invalid UTF-8) is still excluded, but doing so now fails the job with a clear message=unsupported-entries-removed log line instead of quietly publishing an incomplete backup.

command_timeout_seconds now actually stops the process, not just the client

• Killing the host-side docker/docker compose client — all a timeout wrapper alone can do — does not stop a process it started inside the container's own PID namespace. A hung pg_dump, a restore import, or the volume-snapshot tar kept running, orphaned, long after Backfort had given up and moved on. The actual command is now also wrapped with an in-container timeout, so it's reliably reaped. Requires a timeout binary in the database service and volume-helper images (present in common Debian- and Alpine-based images).

Telegram's plain-text retry could never fire

• curl --fail discarded the response body on Telegram's own HTTP error status for a rejected HTML message (e.g. a malformed tag in a custom template), so the {"ok":false,...} payload the retry logic needed to inspect never reached it — the designed fallback silently never triggered. notification_curl_post now reads the HTTP status explicitly via --write-out instead of leaning on curl --fail.

Testing

• New: tests/unreadable-file.sh (unreadable and unsupported source names) and tests/compose-exec-timeout-real.sh (real Docker; proves the in-container process is actually gone, not just detached from a killed client).
• tests/notify.sh's Telegram-retry scenario now serves a genuine HTTP 400 and exercises the real production code path instead of a canned response.
• tests/archive-resilience.sh and tests/fidelity.sh updated for the new fail-instead-of-silently-warn behavior.
• Full suite: 24/24 passing, including fidelity.sh under root with acl/attr.

v1.0.0

Backfort v1.0.0 — First Stable Release

Backfort 1.0.0 is the first stable release of a recovery-first backup tool for Linux.

It is a one-shot Bash utility: no resident daemon, no proprietary backend, and no hidden state outside the configured locations. Define backup policy in YAML, run it from cron or systemd, and restore only from complete, verified backup versions.

Highlights

• Reliable backups for files, Docker Compose projects, named volumes, bind mounts, and databases.
• Local storage plus any [rclone](https://rclone.org/) destination: S3-compatible storage, AWS S3, Cloudflare R2, DigitalOcean Spaces, Vultr Object Storage, Dropbox, Yandex Disk, pCloud, FTP/SFTP, and more.
• Atomic publication with .complete markers: incomplete copies are never presented as recoverable backups.
• Full verification before recovery, including payload checksums and per-file archive hashes.
• Encryption with age, symmetric GPG, or asymmetric GPG public keys; optional Minisign signatures.
• Safe staged restore, interactive restore selection, Compose recovery assistance, retention, pins, notifications, Prometheus metrics, and a dead man's switch.

Backup sources

Files and directories

Back up one or more explicit paths with configurable exclude patterns and optional symlink following.

source:
  type: files
  paths:
    - /etc
    - /srv/www
  exclude:
    - '*.log'
    - '*/cache/*'
  follow_symlinks: false

GNU tar preserves numeric ownership, POSIX ACLs, extended attributes, sparse files, and timestamps for supported regular files.

Docker Compose projects

Backfort can create an explicit migration-grade backup of a Compose project:

• selected Compose and environment files;
• selected named volumes;
• selected bind-mounted project paths;
• logical PostgreSQL, MySQL, and MariaDB dumps;
• MS SQL Server native backup artifacts;
• Oracle Data Pump export artifacts.

Database volumes are never treated as a replacement for logical database dumps.

Named volumes are read by a short-lived helper container with:

• no network;
• read-only root filesystem;
• read-only source volume;
• no writable host mounts;
• all Linux capabilities dropped except DAC_READ_SEARCH.

The helper streams its archive to Backfort, which writes it into its protected workspace.

Quick backups

For urgent work, no hand-written YAML is required:

backfort.sh quick /etc /srv/www/site \
  --name before-deploy \
  --to /var/backups/backfort \
  --to rclone:cloudflare-r2:backfort/web-01 \
  --min-copies 2 \
  --exclude '*.log'

quick-compose provides the same fast path for an explicit Compose project, selected volumes, bind mounts, and database dumps.

Storage, integrity, and recovery

Every published backup is a complete bundle with payload, metadata, checksum, optional signature, and a final .complete marker.

Backfort provides:

• atomic local and rclone publication;
• configurable success.min_copies across independent destinations;
• SHA-256 payload validation;
• per-file SHA-256 hashes for regular archive entries;
• optional Minisign detached signatures;
• verify --quick and deep verify --full;
• safe staged restore to a new or empty directory;
• restore --pick for terminal-based version selection;
• diff ID1 ID2 with text and JSON output;
• host-scoped automatic discovery in shared destinations.

A backup is never considered valid merely because an archive exists. It must have complete bundle evidence and pass validation.

Encryption and signing

Backfort supports:

• age encryption with one or multiple recipients;
• symmetric GPG encryption with protected passphrase handling;
• asymmetric GPG encryption using exact public-key fingerprints;
• recovery from a separate private-key host;
• optional protected private-key passphrase environment variables;
• Minisign payload signatures.

Configuration stores environment-variable names, never secret values. Backfort does not place credentials, passwords, tokens, or private keys into YAML, logs, manifests, generated recovery configuration, or notifications.

Compose recovery assistant

restore-compose restores and verifies a Compose backup into an explicit staging directory, then prints an actionable recovery plan.

For PostgreSQL, MySQL, and MariaDB, it can optionally import logical dumps into an already running target Compose project:

backfort.sh -c /etc/backfort/config.yaml \
  restore-compose latest \
  --job crm \
  --to /srv/recovery/crm \
  --project-dir /srv/crm-target \
  --apply --confirm

This intentionally does not deploy project files, create or overwrite volumes, start services, restore data in place, apply PostgreSQL global roles, or guess MS SQL/Oracle recovery steps.

Retention and lifecycle controls

Backfort includes conservative cleanup controls:

• GFS-style keep_last, keep_daily, keep_weekly, and keep_monthly;
• retention.max_age_days for automatic expiry;
• retention.min_keep, which protects the newest completed recovery copies;
• pin and unpin markers for migration or incident recovery points;
• date-range delete with dry-run planning and explicit --confirm;
• local and remote deletion that removes .complete first.

Pinned backups remain outside normal retention and age-based expiry.

Operational visibility

Notifications

Unified event notifications support:

• Telegram;
• ntfy;
• generic webhooks;
• local SMTP.

Notifications support failure, success, partial-copy, recovery, prune, watchdog, and digest events. Templates use a fixed safe placeholder set, include restore hints, redact secrets, and keep delivery failures non-fatal.

Prometheus

Optional node_exporter textfile metrics expose per-job:

• latest success state;
• exit code;
• backup duration;
• payload size;
• successful and failed destination copies;
• completion timestamp.

Metrics files are written atomically with stable, low-cardinality labels.

Watchdog

Use watchdog as a dead man's switch to detect missing or stale recoverable backups:

backfort.sh -c /etc/backfort/config.yaml watchdog --job crm --max-age 26

It validates the newest completed version instead of trusting file timestamps or incomplete uploads.

Hooks and automation

Per-job pre and post hooks support safe application quiescing, filesystem snapshots, and cleanup workflows.

Hooks use executable paths with literal argument arrays rather than shell command strings. Backfort validates ownership and permissions, supplies a scrubbed environment, bounds execution time, supports dry-run planning, and invokes post-cleanup after interrupted work where possible.

Schedule Backfort with cron or systemd; it deliberately does not run as a permanent daemon.

Safety model

Backfort 1.0.0 is designed to prefer a visible failure over an unsafe recovery assumption.

• Strict YAML validation rejects unknown or malformed configuration.
• Restore requires an explicit new or empty target directory.
• In-place overwrite is intentionally unsupported.
• Only explicitly configured files, volumes, bind mounts, and databases are included.
• Incomplete or malformed bundles are ignored by list, status, restore, retention, deletion, and watchdog operations.
• Shared destinations are isolated by settings.host_id.
• Preflight failures are observable through logs, metrics, and notifications.
• Remote publishing and deletion preserve commit-marker ordering.

Quality and project automation

The release is backed by automated checks for:

• Bash syntax and ShellCheck;
• Bash 4.3 runtime compatibility;
• configuration parsing and example validation;
• file, archive, retention, encryption, signing, notification, metrics, and watchdog behavior;
• local and rclone recovery paths;
• fake-Docker Compose coverage;
• real PostgreSQL, MySQL, and named-volume Compose recovery integration;
• restore fidelity for owners, ACLs, xattrs, and sparse files;
• GitHub documentation links;
• CodeQL, OpenSSF Scorecard, Dependabot, and release metadata validation.

Known boundaries

Backfort is intentionally not a backup server or a replacement for database-native high-availability tooling.

• It does not perform in-place restores.
• It does not infer what should be backed up from a Compose file or image.
• It does not automatically restore named volumes into a live deployment.
• MS SQL Server and Oracle artifacts require engine-specific operator-led recovery.
• Scheduling is delegated to cron or systemd.
• rclone provider credentials are configured outside Backfort.

Upgrade guidance

Before adopting v1.0.0 in production:

1. Run backfort.sh -c /etc/backfort/config.yaml doctor.
2. Create a backup and run verify --full.
3. Rehearse a staged restore on a disposable host or directory.
4. For Compose workloads, test restore-compose against a separate target project.
5. Configure at least two independent destinations for important workloads.

Thank you to everyone who tested the early releases and helped turn Backfort into a dependable recovery tool.

v0.6.0

Backfort v0.6.0

Backfort v0.6.0 is a major step forward for reliable Linux, Docker Compose, database, and offsite backup recovery.

Highlights

• Docker Compose backups for explicit project files, named volumes, bind mounts, and database dumps.
• Logical dump adapters for PostgreSQL, MySQL, MariaDB, MS SQL, and Oracle.
• Quick backup commands:
  • quick for immediate file and directory backups;
  • quick-compose for portable Compose migration and recovery backups.
• Local and rclone-backed destinations, including S3-compatible providers such as AWS S3, Cloudflare R2, DigitalOcean Spaces, Vultr Object Storage, Yandex Object Storage, and other rclone remotes.
• Multi-destination copy policy with success.min_copies.
• Optional age encryption, multiple recipients, Minisign signatures, and hardened symmetric GPG encryption.
• Retention with GFS-style rules, max_age_days, protected pinned backups, and safe date-range deletion.
• restore --pick for interactive terminal selection of completed backup versions.
• diff for comparing backup manifests in text or JSON form.
• watchdog dead-man’s-switch checks for missing or stale backups.
• Unified notifications for Telegram, ntfy, webhooks, and SMTP, with antiflooding, daily digests, redaction, and actionable restore hints.
• Comprehensive examples, recovery documentation, migration runbooks, and GitHub Wiki pages.

Reliability improvements

• Fixed file-source preflight so valid local backup jobs complete correctly.
• Made executable permissions for the CLI and test adapters a CI-enforced requirement.
• Updated generated Telegram and JSON-list output for Mike Farah yq v4 compatibility.
• Improved encryption test fidelity by modelling streaming age decryption.
• Normalized test backup metadata to valid JSON.
• Expanded CI coverage for Compose, rclone, encryption, watchdog, notifications, pins, deletion, diff, quick backups, and interactive restore.

Upgrade notes

No migration is required for existing YAML configurations.

Before upgrading production systems, run:

backfort.sh -c /etc/backfort/config.yaml doctor
backfort.sh -c /etc/backfort/config.yaml --dry-run run

Then perform a full verification and staged restore drill for at least one important backup:

backfort.sh -c /etc/backfort/config.yaml verify latest --job important-files --full
backfort.sh -c /etc/backfort/con