Faster rolling deploys without losing the gates: boot.canary, per-phase timings, unbuffered CI output, proxy v1.1.0.1

[mhenrixon]

Faster rolling deploys without losing the gates: canary boot groups, per-phase timings, unbuffered CI output, proxy v1.1.0.1

Problem / Goal

A cosmos deploy (3 web hosts behind the dash loadbalancer + 1 job host) takes ~7.5 minutes end to end; dash deploy itself reports Finished all in 195.9 seconds (run 33214441782) and 205.2 seconds (run 33215194597). Measured breakdown of the 196s:

Phase	Time	Why
Validate + pull image	~28s	4 hosts in parallel; docker pull is 12s of it
Ensure dash-proxy + loadbalancer	~13s	5 hosts in parallel
App boot	~150s	4 hosts one at a time (boot: limit: 1, wait: 5), ~38s each
Loadbalancer deploy + prune	~7s

Each web host costs 26s of dash-proxy deploy (see below) + 3s stopping the old container + 5s boot.wait + ~4s of small commands. None of it is Ruby CPU or GitHub Actions overhead — it is configured waiting, serialised.

The 26s is the proxy's pre-healthy probe schedule (dash-proxy v1.1.0.0 doubles 50ms → 20s interval: probes land at 12.75s and 25.55s, the app is ready in between, six out of six deploys finished at 26.2–26.7s). That half is zoolutions/dash-proxy#124 and ships first.

Diagnosing this was harder than it should have been: dash writes to a block-buffered stdout under GitHub Actions, so the runner's timestamps show dozens of lines at +0.0s followed by a +38s line — the flush boundary, not the slow step. There is also no per-phase timing at the end of a deploy, only the total.

Done looks like:

1. boot: canary: N boots the first N primary-role hosts one at a time, then the rest of the hosts in parallel (or paced by limit/wait if set). Cosmos goes from 4 serial groups to 2 (canary web-1, then web-2 + web-3 + job together): ~150s → ~80s, and with Stage 3c: container identity — proxy container, docker network, volumes, image label, in-image paths #124 in the proxy ~55s. Capacity never dips because each host's dash-proxy only swaps to the new container once it is healthy.
2. dash deploy prints a per-phase and per-host timing table after Finished all in.
3. Output is unbuffered so CI timestamps are attributable.
4. MINIMUM_VERSION moves to v1.1.0.1 once the proxy image is published.

Context (read these first)

• lib/dash/cli/app.rb — boot (groups loop, run_hook "pre-app-boot" per group, sleep DASH.config.boot.wait between groups) and host_boot_groups (the only place boot.limit slices hosts). Canary changes only host_boot_groups.
• lib/dash/cli/app/boot.rb — Dash::Cli::App::Boot#run: per host/role boot, the health barrier (gatekeeper? = primary role opens it, queuer? = other roles wait). The first primary host in the canary group opens the barrier, so a non-primary host in a later group boots immediately. Record per-host timings here.
• lib/dash/cli/healthcheck/barrier.rb — Concurrent::IVar barrier; unchanged, but the canary design relies on its semantics.
• lib/dash/configuration/boot.rb — Dash::Configuration::Boot (limit_for, limit?, wait, parallel_roles, runner_options_for). Add canary.
• lib/dash/configuration/docs/boot.yml — the example YAML that BOTH validates boot: (via Dash::Configuration::Validation#validate!) and generates the docs page (docs/app/models/config_doc.rb, registered in docs/app/models/doc.rb as Config::Boot). A new key must be added here or validation rejects it.
• lib/dash/configuration.rb — ensure_boot_wait_paces_something (warns when wait is set without limit); canary also creates groups, so wait with canary must not warn.
• lib/dash/configuration/role.rb — role-scoped boot (Role#boot, boot_runner_options). Canary is whole-deploy only; reject it in a role-scoped boot.
• lib/dash/sshkit_with_ext.rb — SSHKitDslRoles#on_roles (role-first threads when parallel:), SSHKit::Runner::Group::NoTrailingWait. No change; read to understand how a group boots.
• lib/dash/cli/base.rb — print_runtime (prints Finished all in), modify. Timings are printed from here.
• lib/dash/cli/main.rb — deploy, redeploy, setup, rollback: the say "...", :magenta phase headers to wrap with timings; print_config_banner.
• lib/dash/commander.rb — DASH singleton; reset clears per-process state. Add timings here.
• bin/dash — the executable (gemspec executables = %w[ dash ]); set $stdout.sync.
• lib/dash/configuration/proxy/run.rb — MINIMUM_VERSION = "v1.1.0.0"; bump to v1.1.0.1 and run bin/sync-proxy-flags.
• test/cli/app_test.rb — boot group tests ("boot does not wait after the final host group", "boot with web barrier opened", "boot paces only the role that declares its own boot limit") show the Dash::Cli::App.any_instance.stubs(:on) / Object.any_instance.expects(:sleep) patterns to copy.
• test/configuration/boot_test.rb — accessor and validation tests for boot.
• test/fixtures/deploy_with_boot_limit_one.yml, deploy_with_role_boot.yml — fixture shapes; multi-host web roles need proxy: loadbalancer: false.
• test/cli/main_test.rb — "deploy", "deploy with skip_push" output assertions.
• Cap the pre-healthy health-check backoff at 2s so boot detection is decoupled from --health-check-interval dash-proxy#124 — the proxy half; must be released as v1.1.0.1 before step 5 below.

Decision

Canary as a first-class boot group. boot: canary: N (integer ≥ 1) makes host_boot_groups return [[c1], [c2], …, [cN], rest…] where c1..cN are the first N hosts of the primary role (within the hosts this run boots), each a group of one, and rest is every remaining app host sliced by limit if set, otherwise one group. wait keeps its meaning (sleep between consecutive groups, never after the last). A boot failure in a canary group raises before any later group starts — the existing loop already does this.

Why: the gate people want from limit: 1 is "prove the image on one host before touching the fleet". Serialising every host after that buys nothing behind per-host zero-downtime proxies (dash-proxy keeps routing to the old container until the new one passes its health check), it only multiplies the wait. The health barrier already encodes "first primary host healthy" — canary reuses it rather than adding a second gate.

Alternatives considered:

• boot: limit: 1, then: parallel — rejected: leaks strategy vocabulary into limit, and "then" reads as ordering not sizing.
• Make canary implicit whenever the primary role has >1 host — rejected: changes the meaning of every existing limit: 1 config silently; operators who want strictly serial keep it.
• Per-role canary (servers.web.boot.canary) — rejected for now: role-scoped boot is paced inside on_roles by the SSHKit runner, which has no notion of a first group with different sizing; whole-deploy only, validated as such. Can be added later without changing the key.

Timings as a collector on the commander. Dash::Timings (new, lib/dash/timings.rb, ~60 lines): phase(name) { } records wall time for a named phase, record(name, seconds, detail: nil) for point measurements, Mutex-guarded because Boot#run runs in SSHKit threads, lines renders an indented table. DASH.timings is created in Commander#reset. Cli::Base#print_runtime prints DASH.timings.lines after Finished all in when any were recorded — deploy, redeploy, setup, rollback all pick it up. Not a metrics system: no export, no thresholds.

Unbuffered output. $stdout.sync = $stderr.sync = true at the top of bin/dash. Ruby only buffers when stdout is not a TTY, which is exactly CI; the cost is one write(2) per line, negligible against SSH round-trips.

Proxy version bump. MINIMUM_VERSION = "v1.1.0.1" after zoolutions/dash-proxy#124 ships. Existing tests interpolate the constant, so no assertion changes.

Settled in interview:

• Add boot: canary: N as a new deploy.yml key (gem-only change to the boot strategy).
• Include the proxy backoff cap: dash-proxy#124 first, then bump MINIMUM_VERSION here. Cosmos may lower proxy.healthcheck.interval in the meantime; that is a workaround, not the fix.
• Ship both observability pieces: unbuffered stdout and the timing table.

Implementation steps

Order matters: 1–4 are independent of the proxy release; 5 waits for ghcr.io/zoolutions/dash-proxy:v1.1.0.1 to be pullable.

1. Configuration layer — boot.canary

1. lib/dash/configuration/docs/boot.yml: add a canary: 1 example under boot: with a doc comment: boots the first N primary-role hosts one at a time before the rest boot together (or paced by limit/wait); whole-deploy only; a failed canary stops the deploy before any other host is touched.
2. lib/dash/configuration/boot.rb: add def canary; boot_config["canary"]; end and def canary?; canary.present?; end. In initialize, when context is role-scoped (i.e. a context was passed) and canary?, raise Dash::ConfigurationError, "#{context}/canary is only supported at the top-level boot: it slices the primary role's hosts ahead of every other role" (match the message style used by ensure_boot_wait_paces_something). Validate canary is an Integer ≥ 1 (the example-driven validator checks Integer type; add the ≥ 1 check explicitly).
3. lib/dash/configuration.rb#ensure_boot_wait_paces_something: treat boot_config.canary? like limit? — a canary creates groups, so wait is no longer idle.
4. Tests, test/configuration/boot_test.rb: "canary is nil by default", "canary reads an integer", "canary below one is rejected", "a role-scoped boot rejects canary", and in test/configuration_test.rb (or wherever ensure_boot_wait_paces_something is tested) "wait with canary does not warn".

2. CLI layer — canary groups

1. New fixture test/fixtures/deploy_with_boot_canary.yml: web: 1.1.1.1, 1.1.1.2, 1.1.1.3; workers: hosts: [1.1.1.4], cmd: bin/jobs, healthcheck: false; proxy: loadbalancer: false; boot: canary: 1, wait: 2. A second fixture deploy_with_boot_canary_and_limit.yml identical but boot: canary: 1, limit: 2 (no wait) to cover slicing the remainder.
2. lib/dash/cli/app.rb#host_boot_groups:

   def host_boot_groups
     hosts = DASH.app_hosts
     canary_hosts = canary_boot_hosts(hosts)
     rest = hosts - canary_hosts
     limit = DASH.config.boot.limit_for(rest)
   
     canary_hosts.map { |host| [ host ] } + (limit && rest.any? ? rest.each_slice(limit).to_a : [ rest ].reject(&:empty?))
   end
   
   def canary_boot_hosts(hosts)
     count = DASH.config.boot.canary
     count ? (DASH.primary_role.hosts & hosts).first(count) : []
   end

   Note limit_for(rest) — a percentage limit counts the non-canary hosts, which is the set it slices; document that in the boot.yml comment. With canary and no primary hosts in the run (--roles workers), canary_hosts is empty and behaviour is unchanged.
3. Tests in test/cli/app_test.rb following the existing stubs(:on) + expects(:sleep) pattern:
   • "boot with a canary boots the first primary host alone, then the rest together" — stub on, expect sleep(2).once (one gap between two groups), and assert on_roles is invoked with hosts: ["1.1.1.1"] then hosts: ["1.1.1.2", "1.1.1.3", "1.1.1.4"] (use Dash::Cli::App.any_instance.expects(:on_roles).with(anything, has_entry(hosts: [...]), ...) in sequence, or capture the hosts: kwarg into an array and assert on it).
   • "a canary with a limit slices the remaining hosts" — canary_and_limit fixture: groups ["1.1.1.1"], ["1.1.1.2","1.1.1.3"], ["1.1.1.4"].
   • "a canary narrows with --roles" — --roles workers yields a single group ["1.1.1.4"], sleep never.
   • "a failing canary stops the deploy before the next group" — make the first group raise (on_roles raises SSHKit::Runner::ExecuteError on first call) and assert on_roles is called exactly once and pre-app-boot is not run for the second group.
   • Keep "boot with web barrier opened" green: the canary fixture with the barrier assertion ("First web container is healthy, booting workers on 1.1.1.4") proves the job host in group 2 does not block.

3. Timings

1. New lib/dash/timings.rb:

   class Dash::Timings
     Entry = Struct.new(:name, :seconds, :detail, :depth)
   
     def initialize; @entries = []; @mutex = Mutex.new; @depth = 0; end
   
     def phase(name)
       started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       depth = @mutex.synchronize { @depth += 1 } - 1
       yield
     ensure
       @mutex.synchronize { @depth -= 1 }
       record name, Process.clock_gettime(Process::CLOCK_MONOTONIC) - started, depth: depth
     end
   
     def record(name, seconds, detail: nil, depth: 0)
       @mutex.synchronize { @entries << Entry.new(name, seconds, detail, depth) }
     end
   
     def any? = @mutex.synchronize { @entries.any? }
   
     def lines
       @mutex.synchronize do
         @entries.map { |e| format("  %s%-34s %7.1fs%s", "  " * e.depth, e.name, e.seconds, e.detail ? " (#{e.detail})" : "") }
       end
     end
   end

   Insert order is completion order; nested phases record after their children, so sort children before parents is not needed — print in insertion order but give each entry its depth so the table reads as a tree. Adjust the exact format to taste; keep it one line per entry.
2. lib/dash/commander.rb: attr_reader :timings; in reset, @timings = Dash::Timings.new.
3. lib/dash/cli/main.rb: wrap each phase in deploy (and mirror in redeploy, setup, rollback): DASH.timings.phase("Pull app image") { invoke "dash:cli:build:pull", [], invoke_options } etc. Phase names: Validate, Pull app image / Build and push app image, Ensure dash-proxy, Detect stale containers, Boot, Loadbalancer, Prune.
4. lib/dash/cli/app/boot.rb#run: wrap the body in a timing: DASH.timings.phase("#{role} #{host}") { ... } at depth 1 under Boot, and inside start_new_version record the proxy wait: time the execute *app.deploy(target: endpoint) call and record("#{role} #{host} healthy", seconds, depth: 2) — or pass detail: "healthy after #{seconds}s" on the host entry (simpler: capture the healthy time in an ivar and use it as detail when the host phase closes). Non-proxy roles record the Poller.wait_for_healthy duration the same way.
5. lib/dash/cli/base.rb#print_runtime: after the Finished all in line, puts DASH.timings.lines if DASH.timings.any?.
6. Tests: test/timings_test.rb (phase nesting, record, thread-safety smoke with two threads, lines format); test/cli/main_test.rb "deploy" and "deploy with skip_push": assert_match(/Boot\s+\d+\.\ds/, output) and assert_match(/web 1\.1\.1\.1\s+\d+\.\ds/, output); test/cli/app_test.rb one assertion that boot output contains healthy after.

4. Unbuffered output

1. bin/dash: add $stdout.sync = true and $stderr.sync = true before require "dash".
2. No test (the executable is integration-covered); mention in the PR body that CI timestamps become per-line.

5. Proxy version bump — only after ghcr.io/zoolutions/dash-proxy:v1.1.0.1 is public

1. lib/dash/configuration/proxy/run.rb: MINIMUM_VERSION = "v1.1.0.1".
2. bin/sync-proxy-flags — commit whatever it regenerates.
3. docker pull ghcr.io/zoolutions/dash-proxy:v1.1.0.1 locally to prove the gate before pushing.

6. Docs

• The Config::Boot page regenerates from boot.yml; run cd docs && bundle exec rspec spec/config_docs_spec.rb.
• docs/app/views/docs/pages/load_balancing.rb (or overview.rb, whichever describes rolling deploys): one paragraph recommending boot: canary: 1 over limit: 1 for multi-host primary roles behind the loadbalancer, and noting the timing table.

Operator follow-through in zoolutions/cosmos (not part of this PR; record in the PR body)

• config/deploy.yml: replace boot: {limit: 1, wait: 5} with boot: {canary: 1} once this gem version is released; drop wait regardless (the proxy health check already gates each host — the 5s is dead time, 15s per deploy). Until Stage 3c: container identity — proxy container, docker network, volumes, image label, in-image paths #124 ships, proxy.healthcheck.interval: 3 cuts ~10s per web host.
• .github/workflows/deploy.yml, deploy job: Install packages (libjemalloc2/libvips, ~14s) is not needed to run dash deploy, and fetch-depth: 0 (~5s) is not needed either. Folding build and deploy into one job removes the second job's ~35s of setup entirely; the trade is packages: write on the whole job.

Verification gates

• bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].grep_v(/integration/).each { |f| require File.expand_path(f) }' — all green (the suite is host-independent; any failure is real).
• bundle exec rubocop --parallel — no offenses.
• cd docs && bundle exec rspec — green (boot.yml drives the generated page).
• bin/test — full suite incl. integration; required because step 5 changes the proxy image every integration deploy pulls.
• Manual: run dash deploy against the integration fixtures with boot: canary: 1 and confirm the output shows one canary group, then one parallel group, and the timing table after Finished all in.

Out of scope

• No direct pushes to main; no manual lib/dash/version.rb bump — releases go through rake release[X.Y.Z].
• No renames of frozen server artifacts (kamal-proxy labels, KAMAL_* env vars, legacy volumes).
• No per-role canary, no new proxy flags, no change to drain/deploy timeouts.
• No changes to the pull or proxy-boot phases (parallel already; the remaining cost is network and docker).
• No metrics export or thresholds on timings — a printed table only.
• Do not edit zoolutions/cosmos from this PR; the operator section above is guidance.

Execution

Hand this issue to a fresh implementation session on the sonnet tier. Steps 1–4 can start immediately on a branch off main; step 5 waits for zoolutions/dash-proxy#124 to be released and the image to be pullable.