Version 5.1.0

[5.1.0] — 2026-09-02

Thanks to Alberto Peripolli for putting Cipi through a real multi-tenant SaaS deployment and reporting every one of the problems this release fixes.

Fixed

• The "deploy failed" notification never fired. _deploy_run ran Deployer as a plain command under set -euo pipefail, so a non-zero exit killed the cipi process on the spot: the failure branch, its warnings, the rollback hint and the deploy_fail email were unreachable code. The run is now wrapped so the real status is read from PIPESTATUS[0], and both the CLI and the automatic webhook path report success and failure by email. cipi deploy also exits non-zero when the deploy failed, so CI and webhooks can see it. The same bug and the same fix applied to cipi deploy --rollback.
• Backups silently skipped every database an app created at runtime. cipi backup run dumped exactly one database per app — the one named after the app. A multi-tenant application creating tenant_1, tenant_2, … appears in no registry, so none of them were ever saved. Databases are now discovered from the engine itself (minus system schemas) and selected with glob patterns.
• Backups of --custom apps failed on every run. The archive step ran tar -C /home/<app> shared/, but a --custom app has no shared/ — its files live in htdocs/. Every run errored, marked the backup as failed, and saved nothing for those apps.
• cipi worker horizon enable left Horizon half-enabled. reload_supervisor returns non-zero when supervisorctl reread/update complains, which under set -e aborted the function before apps.json was updated — so enable printed nothing after "Enabling Horizon…" and status still reported disabled. The state is now always written, the command reports what supervisor actually did, and status surfaces drift between the two.
• A server-wide php.ini change could not reach any app. Every FPM pool hardcoded php_admin_value[upload_max_filesize], post_max_size and max_execution_time, and pool values outrank conf.d. Pools now carry only what is genuinely per-app (open_basedir, the error log, explicit overrides) and inherit everything else.
• The CLI SAPI was never configured. Cipi wrote 99-cipi.ini for FPM only, so queue workers, artisan and cron ran on the PHP package defaults. Both SAPIs are now written, and migration 5.1.0 backfills the CLI file on existing servers.
• Deploy logs had no dates and no release numbers. /home/<app>/logs/deploy.log was raw Deployer output appended forever, which made a deploy that failed overnight unreadable afterwards. Every line is now timestamped and each run is bracketed by a banner naming the trigger, the branch, the release and the duration.
• HTTPS requests matching no app landed on an arbitrary app. :80 had a default server but :443 never did, so a request with an unknown Host fell through to whichever vhost nginx loaded first — for a wildcard multi-tenant app, straight into a tenant resolver that cannot parse it.
• Every backup run aborted before archiving anything when an app had no shared/ (or no htdocs/) directory. The helper that decides what to archive ended on a conditional, so "nothing to archive" was returned as a failure and set -e killed the run — before it could even print the "skipped" warning it had for exactly that case.
• cipi worker horizon enable aborted at the first step, not the last. _supervisor_remove_program ended on a conditional too, returning non-zero in the ordinary case where the app's supervisor file still holds other programs — and it is called before Horizon's own program is written. This is the deeper cause of the half-enabled Horizon reported against 5.0.x; the reload_supervisor guard added earlier in this release was necessary but not sufficient.
• local a="$1" b="…${a}…" does not work. Bash expands ${a} before local has assigned it, so under set -u the function dies with a: unbound variable. Eight of these existed; they survived only because a variable of the same name usually happened to exist in the calling scope. All are now split into separate statements, and the test suite fails if one comes back.
• Two more helpers returned non-zero on their success path: the legacy prune reported failure precisely when it had deleted something, and cipi sync export returned a non-zero exit code after a successful export.
• A disabled backup profile kept running. jq's // operator treats false as absent, so reading enabled returned an empty string instead of "false" — cipi backup profile disable had no effect on the schedule or on cipi backup run. Booleans are now read explicitly. The same read made Encrypted display as blank.
• tar warnings were treated as backup failures. tar exits 1 for "file changed as we read it", which happens constantly on a live application writing its logs — so a perfectly good archive was reported as failed and sent a backup_fail email. Only exit 2 and above now counts as a failure.
• A truncated archive was stored as if it were a backup. A dump cut short by a full disk is still a non-empty file, so the existing size check passed it. Every archive is now integrity-checked before it is shipped, and a corrupt one is discarded and reported instead of quietly replacing a good backup.
• cipi backup fetch reported success when it downloaded nothing. aws s3 cp --recursive exits 0 on a prefix that holds no objects, so a mistyped timestamp produced "Backup available at …" pointing at an empty directory.
• cipi backup verify said "passed" after verifying nothing. With no runs yet it now says so plainly rather than offering a false reassurance.
• A newly created backup profile was immediately reported as overdue, including by email, before its first scheduled run could possibly have happened.
• Migration 5.1.0 could abort and pin a server on the old version. A grep that matched nothing, under set -o pipefail, failed the assignment reading the previous retention — and a failed migration makes cipi self-update refuse the release and retry, failing again, every night. Every step of the migration now degrades to a warning and carries on.
• The nginx catch-all reported "already claimed" when it simply could not write its file, sending anyone debugging it after the wrong vhost.
• cipi yml example produced a template that failed validation. Its placeholder names sat outside the target app's namespace, so the documented first step (cipi yml example > cipi.yml, then validate) failed with four errors. It now takes an optional app name — cipi yml example <app> — and writes the template in that app's namespace, valid as-is.
• cipi self-update said only "Download failed". git's stderr was discarded, so the one message that explains what happened — no DNS, no outbound HTTPS, a missing branch, a full disk, an expired CA bundle — never reached anyone. The real error is now printed, the likely cause is named, and the command to reproduce it by hand is shown. The clone also runs under a 180-second timeout instead of being able to hang indefinitely, and a missing git is reported as such.
• set +e blocks re-enabled errexit instead of restoring it. The deploy helpers turned set -e back on unconditionally at the end of a guarded block, which silently changed the shell's contract for whatever called them. Now the previous state is saved and restored, so these functions are safe to call from any context.
• A failed connection was reported as HTTP status 000000. curl -w '%{http_code}' already prints 000 when it cannot connect, so the || echo "000" fallback appended a second one — the wrong value reached healthcheck alerts and the state files under /var/log/cipi/health.
• Wildcard domain aliases were rejected. cipi alias add <app> '*.example.com' failed validation even though nginx matches wildcard server_name natively and multi-tenant apps need it.

Added

• cipi backup profile — backup strategies instead of one hardcoded job. A profile decides scope (all, files, db), which apps and databases it covers, which tables to skip, how often it runs, where it goes (local, s3, or both) and how long it is kept (--keep=N runs, --keep-days, --keep-weeks). Profiles are independent, so a 30-minute database copy kept locally coexists with an encrypted nightly full copy on S3. Retention is mandatory — a profile that would grow without bound is refused.

• The backup schedule is managed for you. cipi backup configure and every profile change rewrite a marked block in root's crontab; the pre-5.1 hand-written lines are absorbed on migration. Nothing outside the block is touched.

• Application files and databases are stored separately — apps/<app>/files.tar.gz next to databases/<engine>/<db>.sql.gz, with a manifest per run — so a database-only profile costs nothing to take and a database can be restored without unpacking an app.

• Client-side backup encryption (--encrypt): AES-256 before anything leaves the server, so the bucket operator never holds readable data. cipi backup key show prints the key to store off-server; cipi backup fetch downloads and decrypts a run for restore.

• cipi backup verify checks the newest run of each profile actually decompresses (--deep downloads from S3 to do it), and cipi backup status shows destinations, profiles and last results. An hourly watchdog raises backup_stale when a profile has not succeeded within twice its own interval.

• cipi ini — guided php.ini editing. cipi ini list shows the effective value of every tunable setting and which layer set it; cipi ini set upload_max_filesize=50M applies it server-wide to both FPM and CLI; --app=<app> scopes it to one app. post_max_size and memory_limit are raised along with an upload limit that would otherwise cap it, and nginx's client_max_body_size is flagged when it would. Settable keys are an explicit whitelist; open_basedir, auto_prepend_file, extension and friends are refused by name with the reason.

• cipi yml — optional per-project cipi.yml. An app can carry a file in its repository declaring domain aliases, PHP version and settings, extra databases, queue workers, its healthcheck (URL, expected status, grace period, whether to check after a deploy and whether to roll an unhealthy release back) and its backup strategy. cipi yml plan shows the exact diff, cipi yml apply applies it, cipi yml auto <app> on reconciles after every successful deploy — from both cipi deploy <app> and the Git webhook, the latter through one narrowly scoped sudoers rule — and cipi yml example prints a commented template. Deploys ignore the file entirely until that opt-in is given; with it on, a release that carries no cipi.yml is a quiet no-op, and a file that fails validation is reported by email and never partially applied. Credentials for databases it creates are written to /home/<app>/shared/cipi-databases.env, never to the repository.

  cipi yml generate <app> prints the app's configuration as it stands on the server — aliases, PHP version and per-app settings, its extra databases, its queue workers (read back out of supervisor), the scheduler and the backup profiles it owns — as a ready-to-commit cipi.yml, so the file never has to be written from scratch. Cron expressions are rendered back to the friendlier every: 30m form where they map cleanly, values are quoted wherever a plain scalar would be misread, and the result is fed back through the validator before printing. Server-wide backup profiles and the app's own database are deliberately left out.

  Because the file arrives over git, it is fail-closed throughout: it can only configure an app that already exists (never create, rename or delete one), its databases must be named <app> or <app>_* and its backup profiles <app> or <app>-*, its healthcheck URL must resolve to one of the app's own domains (otherwise a commit could aim the server's five-minute prober at an internal address and read the answer back out of the alert emails), unknown keys are errors, no field carries a shell command or a path to include, and the parser implements a small YAML subset that refuses anchors, aliases, tags, merge keys, block scalars and flow mappings outright.

• cipi nginx default-server on|off|status — claims whichever of :80/:443 has no default server and closes unmatched requests with an empty reply (444). Enabled on fresh installs and, where nothing else claims it, by migration 5.1.0.

• cipi deploy <app> --log[=N] shows the timestamped deploy log; --releases now lists each release with its date, commit and subject (the directories stay numeric — rollback depends on that ordering).

• Post-deploy healthcheck. Once an app has a healthcheck URL (cipi health set <app>), the release that just went live is verified right after every deploy — from cipi deploy and from the Git webhook alike — and a failure raises deploy_health_fail immediately. The periodic checker deliberately waits for three consecutive failures before alerting, which is right for a running site and useless for the question "did the push I just made break production?". The probe waits a short grace period (8s for Octane apps, 3s otherwise, or --grace=N) and retries five times, so an app that needs a moment to come up is not reported as broken. The verdict never changes the deploy's exit code — the release is live either way — but the alert says so and gives the rollback command. Turn it off per app with cipi health set <app> --no-postdeploy; run it on demand with cipi health postdeploy <app>.

• Auto-rollback of an unhealthy release (opt-in). cipi health set <app> --rollback-on-unhealthy, or cipi deploy <app> --rollback-on-unhealthy for one deploy, undoes a release that fails its post-deploy healthcheck: the symlink moves back to the previous release, the app is probed again, and one email describes the whole sequence — what was published, what it answered, what it was rolled back to and whether that fixed it. The four outcomes are distinguished: recovered, rolled back but still unhealthy (so the cause is probably not the code), the rollback itself failed (the bad release is still live), and there is no earlier release to return to. It is off by default and says so loudly, because database migrations are not undone — a release that migrated the schema and then failed can be worse off after a rollback.

• Deploy notifications now say what was actually deployed. Success and failure carry visibly different subjects (Cipi deploy succeeded: <app> release 55 … / Cipi deploy FAILED: <app> …) and a body naming the branch, the release number, the commit hash and subject, its author and date, the duration, the previous release, the deploy log path and the post-deploy healthcheck verdict — on both the CLI and the webhook path. The success mail is now sent after verification, so it can never announce a successful deploy while the site is returning 500.

• New notification triggers: backup_stale, ini_set, yml_apply, yml_fail, and self_update — the last one so an unattended nightly update no longer lands without a word. All are toggled with cipi notifications enable|disable.

• Table exclusions in dumps (--exclude-tables='*.jobs,*.telescope_*'), expanded against the live table list on MariaDB and passed straight to pg_dump on PostgreSQL.

Changed

• cipi backup configure accepts an empty bucket for local-only backups, and lists Cloudflare R2, Hetzner, DO Spaces, Backblaze B2, Scaleway and MinIO as S3-compatible endpoints.
• cipi backup prune [app] --weeks=N keeps working against the pre-5.1 layout, so existing archives and any hand-written cron line still prune.
• README: the install snippet no longer carries a $ prefix that gets copied with it, and states that it runs on the server as a sudo user, how to produce the SSH public key it asks for (including on Windows), and what the root password it prints is for.

Migration

5.1.0 installs the automatic-deploy wrapper and repoints each app's webhook cron at it, backfills the CLI 99-cipi.ini, rewrites FPM pools so they inherit the server-wide file, converts the hardcoded nightly backup job into a default profile carrying over its existing --weeks retention, takes over the backup schedule with the managed crontab block, and claims the :443 default server when nothing else does.