How Levelrail keeps deploys safe: health gates and instant rollback

[thegdsks]

A deploy platform is judged by its worst day, not its best one. Building, starting and serving is the easy part. The hard part is the new version that never listens on its port, gets OOM-killed ten seconds in, or starts fine and then answers 500 to one request in three.

Here is what Levelrail does between "a deploy was triggered" and "that exact content is serving", and what it does when something goes wrong. It all comes from the deploy safety docs and the code behind them. Every default is named with the setting that changes it.

The short version

Note

The one rule behind all of this: the previous release keeps serving until the new one has proven itself. Every check below is a way of enforcing that.

• A new container takes traffic only after consecutive readiness passes and a minimum stable running time.
• If it fails either, it is stopped and kept for its logs. The old version never stopped serving.
• After cutover the old release drains, then stays held for a few minutes so rollback is instant.
• A five minute watch rolls back automatically on crashloops, OOM kills or a high 5xx share.
• Rollback targets are pinned by digest, so a moved tag or an image cleanup cannot change what "roll back" means.

The cutover, step by step

Blue-green is the default strategy. Here is a healthy deploy:

sequenceDiagram
    participant CP as Control plane
    participant Old as Old container (blue)
    participant New as New container (green)
    participant Ing as Ingress

    Note over Old,Ing: Old release serves all traffic
    CP->>New: create and start
    loop readiness probe
        CP->>New: probe
        New-->>CP: pass or fail
    end
    Note over CP,New: Gate: N passes in a row,<br/>stable window, no exit, restart or OOM
    CP->>CP: digest check (running image vs pinned digest)
    CP->>Ing: cut traffic to the new release
    CP->>CP: pin new image as a rollback target
    Ing->>New: new requests
    CP->>Old: drain (in-flight requests finish)
    Note over Old: held for the previous-release window,<br/>instant rollback target
    loop post-deploy watch (default 5m)
        CP->>New: restarts, OOM, 5xx share
    end
    Note over CP: cleared, or automatic rollback

Loading

With multiple replicas, each is gated on its own. In blue-green, nothing from the old release is retired until every new replica has passed its gate. If replica 0 passes and replica 1 is OOM-killed, the old release stays fully in place.

Readiness gate vs liveness

Readiness answers "should this container receive traffic yet?" and decides whether a deploy proceeds. Liveness answers "is this running container still alive?" and decides whether it gets restarted. The split is the standard pattern for zero-downtime deploys (general overview), but Levelrail asks for more than one 200.

A new container takes traffic only when all of these hold:

Condition	Default	Setting
Readiness probe passed this many times in a row (a failure resets the streak)	2	APP_DEPLOY_SUCCESS_THRESHOLD
It has been running for the stable window	10s	APP_DEPLOY_STABLE_WINDOW
It did not exit, restart or get OOM-killed inside that window	always	n/a

The controller compares Docker's RestartCount, StartedAt, OOMKilled and exit code against what it saw when the window began. Any exit counts, code 0 included, because a service that stops is not serving.

Without a readiness probe, the gate falls back:

• A port but no probe: the gate does a TCP connect to the port, with the same consecutive passes and stable window. A connection that is accepted and then dropped counts as a failure, so an image that never listens fails the deploy.
• Neither a port nor a probe: there is nothing to check, so readiness is only "the container is running". The gate timeline says so, and the deploy suggestions recommend adding a health check.

The readiness budget is 60 seconds by default, and health.readyTimeout in app.yaml overrides it per service.

Drain and the post-cutover watch

After cutover, the previous release keeps running for at least APP_DEPLOY_DRAIN_TIMEOUT (default 15s), measured from the cutover itself, so requests already in flight on it finish before it stops. Ingress moves new requests to the new release in the same reconcile pass.

Then the watch starts. For APP_DEPLOY_WATCH_WINDOW (default 5m) the release is checked every APP_DEPLOY_WATCH_INTERVAL (default 10s). It trips on:

Trigger	Default	Reason
A container was OOM-killed	always	PostDeployOOMKilled
Restarts (or an exit) reach the limit	APP_DEPLOY_WATCH_MAX_RESTARTS=2	PostDeployCrashloop, PostDeployExited
5xx share through the ingress	0.2 after 20 requests	PostDeployErrorRate

The 5xx share comes from existing ingress request metrics. The first sample after cutover is skipped because it can still hold answers from the previous release.

The previous release then stays held for APP_DEPLOY_PREVIOUS_RELEASE_HOLD (default 3m), and for the whole watch window with auto-rollback on. Only one previous release is ever held.

When the new version never becomes ready

This is the case that matters most.

flowchart TD
    A[New container starts] --> B{Readiness gate}
    B -->|never passes, exits or OOM| C[Candidate stopped, logs kept]
    C --> D[Old release keeps serving]
    D --> E[Deploy attempt marked failed with a reason]
    E --> F[Next reconcile retries under restart backoff]
    F -->|keeps exiting| G[Ready False: CrashLoopBackOff, bounded delay]
    B -->|passes| H[Cutover]
    H --> I{Post-deploy watch}
    I -->|quiet for the window| J[Cleared]
    I -->|OOM, crashloop or 5xx share| K[Deploy failed, alert sent]
    K --> L{Auto-rollback on?}
    L -->|yes, default| M[Roll back to newest good pinned release, with its config]
    M --> N[Same readiness gate applies]
    L -->|no| O[Left for an operator]

Loading

Step by step:

1. The old version keeps serving, because the new release never reports ready and ingress never moves.
2. The failed candidate is stopped, and the last 200 lines it printed are captured with the failed deploy, so you can read them on the deploy page, the app overview and the logs page.
3. The attempt is marked failed with a specific reason: ReadinessFailed, OOMKilledDuringReadiness, ExitedDuringReadiness, or the stable-window variants OOMKilledDuringStableWindow, ExitedDuringStableWindow, RestartedDuringStableWindow and StabilityGateTimeout. A container that dies mid-wait is noticed within a second, not at the probe's next interval, with a message such as Container exited with code 0 after 2s, before it became ready.
4. The next reconcile retries under restart backoff. The first restart is immediate. Each further restart waits twice as long as the last, starting at APP_RESTART_BACKOFF_BASE (default 5s) and capped at APP_RESTART_BACKOFF_MAX (default 2m). While it waits, the app's Ready condition is False with reason CrashLoopBackOff and a message like keeps exiting (2 restarts), next restart in 9s. The delay resets once the container stays up for APP_RESTART_BACKOFF_RESET (default 10m).

The 2 minute cap keeps the crashloop alert firing instead of resolving between restarts.

Important

Two limits. Apps with a pinned host port (--host-port) cannot run two releases at once because the port can only be bound once, so for them a deploy is a short outage rather than a zero-downtime cutover. If the new release fails there, it is removed and the previous one restarts on the same port. And for remote nodes, restart counts are not carried over the agent transport yet, so the stable window detects exits and OOM kills but not a restart the container survived.

What you see while it happens

Deploys are recorded as attempts with a status. The deployments list filters on queued, held, awaiting_approval, building, ready, failed, canceled, rolled_back and superseded. A rolled_back row is a succeeded deploy that a later rollback replaced.

Each deploy also records its gate as a timeline (readiness, stable window, cutover, drain, rollback pin, watch, rollback), every step with a status, reason and message. The deploy detail page shows it on the Roll out tab and refreshes until the release clears its watch, fails or is replaced.

From the CLI:

levelrail-cli deploy status web --watch          # newest deploy, follow it
levelrail-cli apps deploys gate web dep_abc123   # one deploy, once
levelrail-cli apps wait web                      # CI gate: exits 5 on failure, 6 on timeout

deploy status exits 7 when the release failed its gate, tripped its watch or was rolled back, so scripts can branch on it.

Strategies and canary

Three strategies, set per app in Deploy settings or app.yaml:

Strategy	Behavior	Use when
blue-green (default)	Start the full new set, gate every replica, cut over, drain the old set	You want the simplest all-or-nothing cutover
rolling	Replace replicas one at a time; the old release's first replica stays until the pass ends so routing never points at a removed container	You have several replicas and want a gradual swap
recreate	Stop the old release, then start the new one	You cannot run two releases at once. Never holds a previous release, and implies a gap

levelrail-cli apps scale web --replicas 3 --strategy rolling

A canary is the more cautious path. It runs a new image next to the current release and sends it a share of requests:

levelrail-cli apps canary start web --image myorg/web:next --weight 10
levelrail-cli apps canary status web
levelrail-cli apps canary weight web --weight 50
levelrail-cli apps canary promote web
levelrail-cli apps canary abort web

The canary docs cover the details:

• The split is per request, not per user, from a repeating cycle of 100, so weight 20 sends exactly 20 of every 100 requests to the canary.
• A canary that is not ready gets no traffic, so a bad image cannot take the app down. Abort sends everything back to the stable release.
• Promote deploys the canary image through the normal path (readiness gate, rollback target, freeze and protection checks), then removes the clone.

Limits: one canary per app, image apps only, single replica, no pinned host port. The canary writes to the same database as the stable release, so keep schema changes compatible with both versions.

Rollbacks: automatic, by digest, instant

There are three rollback paths, and all of them are ordinary deploys that pass the same readiness gate.

Automatic. When the watch trips, the deploy attempt is marked failed with the reason, an alert goes out through the app's deploy notification channels, and with APP_DEPLOY_AUTO_ROLLBACK=true (the default) the app returns to the newest pinned release still marked succeeded, with that release's config. Two guards prevent loops: a release the watch already tripped on is failed, so rollbacks never ping-pong, and a release that is itself an automatic rollback is never rolled back again. That points at something outside the app, so it is left for you.

By digest. Every image deploy resolves its tag to a content digest at deploy time and pins the desired image to it, for example nginx:latest@sha256:.... A floating tag like latest names different content over time, so a platform that compares tags can report success while the old image keeps running. Because the pinned reference is part of the container's identity, new content always gets a new container. The controller also checks that the running image ID matches what the pinned reference resolves to; if not, the app reports Ready: False with ImageDigestMismatch and the previous release keeps serving.

levelrail-cli apps deploys list web
levelrail-cli apps deploys rollback-to web dep_abc123 --watch
levelrail-cli apps rollback web --image myorg/web:v1.4.2

rollback-to redeploys exactly what that past deploy ran, by digest or by its rollback pin, and restores that deploy's literal env values, port, health checks, replicas, strategy, resources and labels. Secret, database and vault references keep their current values, and domains, volumes and host ports stay as they are now.

Instant. For APP_DEPLOY_PREVIOUS_RELEASE_HOLD (default 3m) after a cutover, the previous release's containers are still running, so rolling back to it needs no pull and no cold start. Beyond that, pins take over: every release that took traffic gets an extra local tag, and the newest APP_DEPLOY_ROLLBACK_PINS (default 3) per app are kept. A tagged image is never dangling, so image cleanup cannot remove a rollback target, even after a rebuild moved the original tag.

Warning

"Tag now points at different content." If you roll back to a deploy whose image tag has since been re-pushed with other content, rollback-to refuses with a 409: tag <image> now points at different content than this deploy recorded; redeploy from source instead. The refusal is deliberate: a rollback should restore a known state, and deploying a moved tag would run code you never tested while calling it a rollback. A 410 means the local image was garbage collected (raise APP_DEPLOY_ROLLBACK_PINS to keep more); redeploy from source then too.

Deploy history records how the digest was resolved (Resolved, AlreadyPinned, LocalBuild, PullFailedUsingCached or Unresolved). Pass --pull to refuse the cached fallback.

Hooks, freezes and approvals

Deploy hooks. In app.yaml, a service can declare hooks, run inside its own container through the Docker Engine API exec facility:

hooks:
  preDeploy: rails db:migrate
  postDeploy: curl -f https://hooks.example.com/deployed

• preDeploy runs once per deploy, before the new container's readiness probe and before any old container is removed. A nonzero exit blocks the deploy: the new container is rolled back and whatever was running keeps serving.
• postDeploy runs after the full replica set has cut over. A nonzero exit is reported as a PostDeployHookFailed condition but never rolls back a successful cutover.
• With several replicas, each hook runs once per deploy, on the first replica.
• Not available for static or compose builds. levelrail-cli apps hook-runs web shows the latest outcome.

Freeze windows. A window starts at every match of a 5 field cron expression, lasts a duration, and can be set per app or globally. Pushes, release webhooks and pipeline steps are held and the newest one replays when the window ends. Manual deploys return 423 unless they carry a recorded override reason.

levelrail-cli apps freeze set web --cron "0 17 * * 5" --duration 64h \
  --timezone Europe/Berlin --reason "weekend freeze"
levelrail-cli apps deploy web --image app:hotfix --override-freeze --override-reason "CVE fix"

Approvals. Deploying into a protected environment needs a second person: the request becomes a pending approval (202), someone else with the deploy ability approves it, and unanswered requests expire after 24 hours. Approval runs the normal path, so the readiness gate still applies.

levelrail-cli deploy-approvals list
levelrail-cli deploy-approvals approve ID

What we test

Safety claims are cheap, so here is the evidence and where it stops.

• Deploy safety, live. test/e2e/reconcile/deploy_safety_live_test.go runs the real application and ingress controllers against Docker while four clients hammer the ingress. It deploys a good release, one that exits inside its stable window, and one that answers 5xx until the watch rolls it back. The fixture drops in-flight requests on SIGTERM, so only a real drain keeps them alive. One run on Docker Desktop: 2617 requests, 0 failed (the bad release's own 500s excluded). With the drain turned off, the same test loses requests at cutover.
• Wrong health path. In one of our conformance runs on a scratch control plane, a redeploy whose new container failed readiness on a wrong health path kept the old version serving, with 0 of 243 requests failing.
• Burst of apps. In another run, a burst of 12 apps came up on one host with no failures after the network-pool fix.
• Pruning. internal/docker/prune_images_live_test.go checks against a real daemon that a pinned tag and an untagged image backing a running container survive cleanup, and only a truly orphaned image is removed.

Note

These are results from specific runs on particular hosts with particular fixtures, not a guarantee. Your probe quality, memory limits and database migrations still shape your outcome.

Troubleshooting

Symptom	Likely cause	What to do
Deploy failed, old version still serving	Readiness gate failed (ReadinessFailed, exit or OOM during the window)	Read the captured last 200 lines on the deploy page, then fix the health path or port and redeploy
Ready: False, CrashLoopBackOff	Container keeps exiting, backoff is waiting	levelrail-cli apps status web, then check logs; the delay is capped at 2m
Rolled back on its own	Post-deploy watch tripped (PostDeployOOMKilled, PostDeployCrashloop, PostDeployErrorRate)	Open the Roll out tab for the reason, fix, redeploy
rollback-to returns 409	The tag now points at different content	Redeploy from source, or roll back by a digest pinned deploy

Failure codes are listed in Deploy failures, and levelrail-cli apps diagnose web explains a failed deploy or crashloop.

Try it

Break a rollout on purpose in a throwaway app before you trust it:

• Deploy an app with a real readiness path
• Redeploy with a wrong health path and confirm the old version keeps serving
• Open the Roll out tab and read the gate timeline
• Run levelrail-cli apps deploys rollback-to against an earlier deploy
• Try a canary at weight 10, then abort it

Docs: Deploy safety, Canary deploys. Code: github.com/glincker/levelrail.

If a deploy misbehaves in a way this post does not explain, open an issue with the gate timeline attached. We want to find that bug before you do.