feat(core): start a service from the connection that needed it (T70) - #54
Merged
Conversation
… it needs (T70) T69 shipped idle detection switched off because nothing started a service back up. This is the design for the thing that does: one activator that accepts a connection, starts the service, and copies bytes — never parsing FastCGI, MySQL or RESP, which is what makes it one mechanism rather than four. The roadmap's "hold the socket" survives for databases and not for the web path. Handing php-fpm its own socket back means closing the daemon's listener first, and every request in the several hundred milliseconds before the pool binds it is refused. So a site names a second, permanently bound activator address after the pool's own; a database, having no front end to say that in, does hold its address, and the window is written down rather than hidden. The rendering was measured against a real Caddy 2.10.0 and nginx 1.24.0 before the rest was written, and it changed the design twice over. The bare two-address form answers 8 of 20 — Caddy load-balances between the pair, so half of a healthy site's traffic would cross the activator. Adding lb_policy first and a retry budget makes it 0 of 20, each burning the full 5 s, because nothing marks the refusing pool unavailable. All three of lb_policy first, lb_try_duration and fail_duration answer 20 of 20. fail_duration was then measured to be exactly how long a recovered pool is still reached the slow way. Marked [~]: the design and its gating measurement are in, the implementation is not.
T70 is the web path and T70a is the database path. The split is by where the activator binds, which is the one decision the two halves do not share: a pool keeps its own address and the front end names a second one after it, while a database has no front end to say that in and so the daemon holds its address while it is stopped. T70a is not optional. M7 asks for a machine where only the daemon and the web server survive 30 idle minutes, so a database that idle-stops and never comes back moves the broken case rather than fixing it. The design stays one document: everything but D2 and D4 is common to both, and splitting the paper would split the argument that makes them one mechanism. D9 now divides the recipe defaults by which task can start each service again — T70 turns on php-fpm and nothing else, which is the only state in which half a mechanism is safe to ship.
The first half of on-demand activation, and the half that decides where the daemon will listen: every activatable service gets a second address, and a site file will name it after the pool's own so a request arriving while the pool is idle-stopped is retried rather than answered with a 502. The two shapes are not symmetrical, and starting the implementation is what showed it. A Unix socket's activator is derived — run/php-fpm-8.3.sock becomes run/php-fpm-8.3.activate.sock — and the derivation is fallible, because `.activate` is nine characters and sockaddr_un's limit is 103: a home that fits the pool's socket may not fit the activator's, and the failure that would otherwise follow is a bind refused at run time and a site that 502s for no visible reason. A TCP pool's cannot be derived at all. `port + 1` hands the first pool's activator the second pool's own port, so the port is allocated onto the row in a new nullable column and never computed again — the rule services::ports already applies to the port a service listens on. A row written before the column carries none, and that renders a site with no fallback rather than one pointing at an address nothing binds: a home without on-demand activation is the home it was yesterday, and a broken site is not. The design's D3 said "derived by a fixed suffix" for both shapes. It is updated here rather than worked around.
A site file now names the pool first and the activator second, so a request arriving while the pool is idle-stopped is retried instead of answered with a 502. Both front ends, because either one of them may be the one serving. Caddy needs three directives and any two of them are worse than none — measured before this was written and again after. `lb_policy first` alone keeps choosing the address that is already refusing until the retry budget runs out; `fail_duration` is what lets it move on; without either, Caddy treats the pair as peers and load-balances, sending half of a healthy site's traffic through the activator. nginx says the same thing in one word: `backup` in an upstream group already means "only once the others have refused". The group nginx needs is named after the site and never after the pool. Two sites sharing one pool are ordinary, and nginx refuses a configuration that declares one upstream name twice — which takes the whole front end down rather than the one site. There is a test for the name, and it fails on a name derived from the pool. Both renderings were shown to the real servers rather than asserted against by substring alone: `nginx -t` on nginx 1.24.0 and `caddy validate` on Caddy 2.10.0 both accept them. A pool with no activator renders exactly what it rendered before this commit — no group, no policy, `fastcgi_pass` straight at the pool. A home whose row predates the column is the home it was yesterday.
The column added with the address needed filling. `ports::allocate_activation` finds the lowest free port above the service's own, and counts a port taken when *either* column holds it — `allocate` reads `port` alone, which is right for a service's own address and would hand two activators one number. A repair at boot rather than a step inside `service.create`, on the same reasoning `pools::ensure` is written down with: it reaches a pool created a moment ago and a pool written by a build that predates the column, with no data migration and no port chosen against a listening table months before anything binds it. Which services are owed one is the catalogue's answer, through a new `Recipe::activation_port_needed`. It takes no Context because the port has to exist before there is a context to render with, and it answers false for a recipe whose activator derives its address from a socket path — that service takes no number out of circulation at all. php-fpm now states the socket-versus-TCP rule once, in `listens_on_tcp`, read by all three of the places that branch on it. Three `cfg!(windows)`s would be three chances for two of them to disagree, and a disagreement here is a site pointing at an address nothing binds.
…shape (T70) Where a stopped service is woken from differs by system — a Unix socket on Linux and macOS, a TCP port on Windows — and `mixengine-daemon` may hold no `#[cfg(unix)]`, so the binding lives here behind one `Listen` value. Beside `ipc` and emphatically not the same thing. That endpoint is the daemon's own, singular, and carries a peer check answering "who is this". An activator holds one address per service, and the programs that dial it are the user's own — a browser, a database client, a FastCGI client — so asking any of them who they are would be a question with no possible use. Nothing here parses anything, which is what lets one activator serve FastCGI, the MySQL protocol and RESP alike. There is a test for each direction: a client that speaks first and one that waits to be greeted go through the same code. A stale socket file is cleared before binding, because `bind` on a path that still has one fails with AddrInUse even when nothing is listening — a daemon that was killed would otherwise leave every activator address in that home unusable, and what a user would see is sites that 502 until somebody works out that a file in `run/` is why. Only when nothing answers there: unlinking a path a live process is serving would take that service off the air. Windows has no socket arm, and that is upstream's shape. No service MixEngine runs there listens on one, so a socket path arriving is a home built on another system and is refused by name rather than through an obscure bind failure. Proved on both: 3 of 3 on Windows and 3 of 3 in WSL, which is the run that compiles `unix/` at all.
The activator. A connection arrives on the address a site file names after the pool, the service is made to run, and bytes are copied to it — nothing in that sentence mentions a protocol, which is what lets one activator serve FastCGI now and the MySQL protocol under T70a. There is a test for each conversational order: a client that speaks first and one that waits to be greeted go through the same code, and the waiting one is what a web-only design would never meet. Three answers, each bounded. A service the daemon idled is started, waited for and proxied. A service a person stopped closes the connection: `mix service stop` followed by the next request undoing it is the tool overruling its user. A service already running is proxied straight through, because a fault in a service that is up is not this code's to diagnose. The plan said the last of those could be read from `StateReason::Idle`. It cannot — a transition is not stored anywhere, so a daemon that restarted would forget which of its stopped services it had stopped itself, and every pool idled before the restart would stay stopped with its site answering 502 for ever. Migration 0010 adds `services.idle_stopped`, written on every arrival at `stopped` so it can never be left over from an older stop. Nothing here starts a process. It calls the same `Registry::start` a person's `mix service start` calls, with the plan from the graph, so a dependency starts in order and a service already mid-start is joined rather than started twice. Activation is a second caller, never a second way. `Generator::activators` computes both addresses through `declarations`, the same path a render takes — the address the daemon binds and the address a site file names cannot disagree, and what a disagreement would look like is a site that 502s beside an activator holding an address nothing dials. The registry test fixture moved to `fixture.rs`, which is the module that exists for scaffolding more than one test module needs; the activator is the third.
The first recipe in this build to answer `idle_default` with anything but None, and the last commit of the task on purpose: a default that idles a pool is safe only once something starts it again, which is what the four commits before this one built. A pool and nothing else. The databases and the caches keep answering None until T70a can start them again — turning them on here would idle-stop a database that nothing could bring back, which is a connection refused with no way out. The two front ends answer None for ever: the thing that starts everything else back up cannot be the thing that gets stopped. Three comments said the opposite and are corrected rather than left. "No recipe ships an idle default" in the daemon's sweeper block, "which is nothing, in every recipe this build ships" where a row's idle policy is assembled, and the trait's own "Every shipped recipe answers None, on purpose" were each true this morning and would each have sent the next reader looking for a feature that is now here.
The web path is done: a pool idled after thirty minutes is started by the request that finds it down, and the front end is what notices. Two findings written down rather than left for the next reader. A TCP activator's port cannot be derived and the "taken" rule has to read both columns, or two services are handed one address. And D8 was unanswerable as designed — it asks whether the daemon idled the service, the reason lives on a transition, and a transition is not stored, so a daemon restart would forget and every pool idled before it would stay stopped for ever. The gap is stated rather than implied: no test drives a real front end through a real stopped pool. Both renderings were accepted by a real Caddy and a real nginx and the retry was measured against both, but the two ends meeting is left to the suites that run a real pool. Phase 7 is 3 of 8.
Both of the previous run's failures were one mistake made twice: a new `mod` line was inserted between a `#[cfg]` attribute and the item that attribute belonged to, so the new module took the gate and the old one lost it. In the daemon, `pub(crate) mod activate;` took `#[cfg(debug_assertions)]` from `mod fakeservice;` — so the activator was configured out of every release build and `mixengined` did not compile. That is why all three bench legs failed and all three test legs passed: the test legs are debug. In the platform crate, `pub(crate) mod activation;` took `#[cfg(feature = "ipc")]` from `pub(crate) mod ipc;`, and rustfmt then sorted the modules so the two ended up far apart. `mixengine-elevate` depends on `mixengine-platform` without the `ipc` feature, so the system suite compiled `windows/ipc.rs` against a `crate::ipc` that was not there. Neither was reachable from the gates run before pushing: those build one OS, in debug, with default features. `cargo check --workspace --release --all-targets` and `cargo test -p mixengine-elevate --test system --no-run` reproduce them exactly, and both are clean now.
…t (T70) I said the three `test (windows-latest)` failures were not a regression. That was wrong, and the code had already written down why. `main.rs` binds the endpoint and does not reach `accept` until much later, and the block that fills `bin/` explains the cost in as many words: a bound listener that is not yet accepting has exactly one pending connection on Windows, so every moment spent in between is a moment a second client meets ERROR_PIPE_BUSY. It even records that putting nineteen file copies there once made an ordinary parallel test run fail. T70 put two awaited blocks squarely inside that window: a port allocation that probes by binding, and a full render pass followed by one bind per service. Three legs in a row failed there, in three different suites — sites.rs, api.rs, packages.rs — every one of them on "All pipe instances are busy". That master had hit the same error once before proved the window exists. It did not prove T70 had not widened it, and I treated it as if it had. Both now run in one spawned task, so `accept` is reached immediately. Being a moment late costs nothing it could cost: nothing dials an activator until a site is being served, and a site is served by a front end this daemon has not started yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A service that nobody is using should not be holding a port. T70 gives the web
path the other half of that: after the daemon stops an idle php-fpm pool, the
next request still has to work. An activator holds the pool's fallback
address, and the first connection that reaches it starts the service and splices
itself through.
The activator is protocol-blind. It never parses FastCGI — it binds, accepts,
starts the service if that is what the connection means, dials the real address,
and hands the two halves to
copy_bidirectional. The same shape will carry thedatabase path in T70a; the split between the two tasks is where the activator
binds, nothing else.
Design:
docs/superpowers/specs/2026-08-29-t70-on-demand-activation-design.mdWhat the front end had to be told
The rendering was measured against real servers before any of it was written,
and the measurement refuted two plausible renderings in opposite directions:
upstreamgroup,server …:9002 backup,next_upstream error timeoutreverse_proxy …:9001 …:9002+ lb_policy first,lb_try_duration 5s+ fail_duration 10sCaddy needs all three directives; any two of them fail, and they fail in
opposite directions. Both generated files were then checked with
nginx -tandcaddy validate.The commits
c974eb7a service carries an activator address beside its own83bae98the activator renders as a site's fallback upstream94ebb8athe activator's port is allocated onto the row, not derived from it6a629d8a listener the activator can hold, on a socket or on a port551e348the connection that needed a service starts it6393d25a php-fpm pool idles out after thirty minutesaf66063roadmapTwo follow-up commits:
7160e0arestores two#[cfg]gates that ananchor-based insert had taken from their items — both compiled cleanly here and
only went red in CI.
3d3453bmoves the boot work out of the window betweenbinding the IPC endpoint and reaching
accept, which on Windows admits exactlyone pending connection; three unrelated suites were failing on it.
Verification
CI 33257632063:
9 of 10 green,
bench (ubuntu-latest)over budget. That one was settled bymeasurement rather than by argument — a rerun of the branch and a control run of
master, side by side:
The distributions overlap and master's worst round is the worse of the two, so
M3's spread is the known bimodality, not a T70 regression. Both reruns passed.