fix(store,supervisor): complete the instance handover on entry update so hot installs do not strand consumers

[wolfy-j]

Root cause

Updating a registry entry for a store must hand the running instance over to a replacement. Two independent halves of that handover were broken, so a single hot module install (which bumps entry meta and fires Update on every affected entry) left the replacement orphaned.

Resource half. The managers recreated the store but never sent resource.Update, so system/resource/registry.go kept its active generation pointing at the superseded instance. For service/store/memory the superseded instance was stopped first and Store.Acquire returns resource.ErrReleased once closed is set, so the store was unusable for the rest of the process lifetime.

Supervisor half. supervisor.ServiceUpdate has no inbound handler. The supervisor consumes ServiceRegister, ServiceRemove, ServiceStart and ServiceStop (system/supervisor/supervisor.go handleEvent), and uses ServiceUpdate for its own outbound state notification carrying a State payload. All four store managers sent ServiceUpdate with a *supervisor.Entry, so the replacement never reached the supervisor: its Start was never called and the controller kept supervising the stopped original. Cleanup loops stayed dead and shutdown targeted the stale instance.

Switching to ServiceRegister alone does not fix it either. execute creates a controller only if _, exists := s.controllers[id]; !exists, so a registration naming a live ID is silently dropped. And remove-plus-register cannot express the handover, because entry changes are applied inside a registry transaction (system/registry/runner/bus_runner.go wraps the whole ChangeSet in TxBegin/TxCommit) and regTx.registerService clears a pending remove for the same ID, collapsing the pair back to a bare register. That is why service/pg's Delete+Add model, and the ServiceRemove-then-ServiceRegister pattern in service/queue/amqp and service/aws/sqs, could not hand over either.

Field incident

After a hub hot install on prod C, the settings cache (a store.memory entry) went dead: every read through the resource registry returned resource has been released, and stayed dead until the instance was restarted. The entry itself was healthy; the registry was pointing at the stopped *Store, and the supervisor was still holding it too.

Lifecycle design

The supervisor owns service lifecycle, so it owns retirement. An entry update expresses exactly one thing — "this ID is now served by this instance" — and the supervisor performs the handover.

Manager side. The manager builds the replacement, installs it in its own map, then:

1. Repoints the resource registry at the replacement and waits for the registry to confirm that specific operation applied (service/store/await.go).
2. Sends supervisor.ServiceRegister with the replacement.

Ordering falls out of step 1. The registry and the supervisor are independent bus subscribers with their own goroutines, so publishing gives no ordering against the supervisor retiring the old instance at commit. Returning only once the registry serves the replacement puts the repoint strictly before the TxCommit that stops the old one. The confirmation is an event, not polling: the registry publishes resource.accept / resource.reject after its state is visible to Acquire.

The wait is correlated by a per-operation Entry.OpID, which the registry echoes as the outcome event's path. Registrations and updates for one resource share a resource path, and the await dispatcher keys pending waits by path — so a path-keyed wait could be satisfied by an earlier operation's outcome, and two waits on one resource would overwrite each other. Callers that do not wait leave OpID empty and keep publishing under the resource path, so the eight other resource managers are unaffected.

A dropped or unapplied repoint therefore fails the entry operation and rolls the manager's map back, instead of half-applying. This replaces an earlier ctx.Err() guard that was dead code: the listener context is the router's long-lived context, not the caller's, so it can never be cancelled at that point.

Supervisor side. On commit, planReplacements computes the retirement without side effects, and applyReplacements performs it:

1. A registration whose service instance differs from the one its controller supervises is a replacement (sameServiceInstance; comparable dynamic types answer identity directly, maps by pointer, anything unprovable counts as a replacement).
2. The stop batch covers the dependent closure, not just replaced IDs — a running service that depends on a replaced one captured the superseded instance and must come down before it and come back up after (buildStopOperations walks dependents first, which is also the order the sequencer needs).
3. A stop failure rejects the whole retirement: the services the batch already stopped are restored, no controller is dropped, and nothing is adopted. If a restore itself fails, that travels with the rejection and names the services left down — a service running before the commit and not running after it is not reduced to a log line.
4. Retired controllers are cancel()ed, ending their supervise goroutine and releasing the superseded service; dropping the map entry alone is not the lifetime.
5. Dependents the retirement took down are added to the start roots so they come back against the adopted instance.
6. Re-registering the instance already supervised is a no-op, so nothing churns.

Exactly-once retirement. Controller status is bookkeeping, not proof about the instance, so exactly-once cannot rest on it alone. pg and terminal stopped their own instance inside Delete while the supervisor still owned it, which left the controller believing the service was running; once retirement started working, their delete-then-register update handed that instance a second Stop. Stopping belongs to the supervisor — ServiceRemove stops the service on commit, and an update that re-registers a replacement in the same transaction retires it there — so both managers now leave it alone. serviceNeedsStop additionally skips a controller already known to be down.

Sites fixed

Engine:

• system/supervisor/supervisor.go — planReplacements, applyReplacements, restoreStopped, serviceNeedsStop, sameServiceInstance, called from execute
• system/supervisor/controller.go — Controller.Service() accessor
• system/supervisor/errors.go — NewRetirementRestoreError
• system/resource/registry.go — publishes resource.accept / resource.reject, under the operation id when one is carried
• api/resource/resource.go — Entry.OpID, Accept / Reject kinds

Store managers (ServiceUpdate to ServiceRegister, confirmed resource.Update, no manager-side stop):

• service/store/memory/manager.go
• service/store/sql/manager.go
• service/store/kv/manager.go (RaftManager)
• service/store/kv/crdt_manager.go (CRDTManager)
• service/store/await.go — AwaitResourceUpdate

Managers that stopped instances the supervisor owns:

• service/pg/manager.go
• service/terminal/manager.go

Test coverage

service/store/await_test.go:

• IgnoresOutcomesOfOtherOperations — a stream of accepts on the resource path must not confirm this operation. Red with path-only correlation: "An error is expected but got nil".
• ConfirmsAppliedUpdate / ReportsRejection — the confirming and declining paths against the real registry.

system/supervisor/handover_test.go:

• RegisterAdoptsReplacementInstance — red: "timeout waiting for service to start".
• RegisterKeepsIdenticalInstanceRunning, TestSameServiceInstance — no churn; value and pointer identity both directions.
• ReplacementRetiresRunningDependents — A→B, replace only B; A is stopped and restarted. Red: "timeout waiting for dependent to be restarted".
• RejectedRetirementRestoresRunningSet — a stop fails; the supervisor keeps its controllers and the service it stopped comes back. Retries disabled so the controller cannot mask a lost service.
• RejectedRetirementReportsUnrestoredServices — the restore also fails; the failure is visible in supervisor state and in the commit outcome. Red with a log-only restore: "timeout waiting for the rejected commit to report the unrestored service".
• RetirementCancelsController — 10 hot updates leave one controller and no leaked goroutines. Red: "timeout waiting for retired controllers to release their goroutines".
• DeleteThenRegisterStopsInstanceOnce — the pg/terminal update shape with a non-idempotent Stop that errors on a second call; asserts exactly one stop of the superseded instance and none of the adopted one. Red with retirement disabled: "timeout waiting for replacement to start".

service/store/{memory,sql,kv}/manager_resource_test.go — each drives the real supervisor, real resource registry and real await service over a shared bus, inside real TxBegin/TxCommit transactions: the registry serves the replacement, the superseded store is stopped, the replacement runs, and a later ServiceStop reaches the replacement. Plus the emitted resource.Entry shape (path, id, meta) and the two failure modes (unconfirmed handover, no coordination available) with rollback. kv is table-driven over raft and crdt.

Validation

go test --tags "fts5 sqlite_vec treesitter" ./system/... ./service/... ./boot/... ./api/...   # 208 ok, 0 failures
go test ... -race -count=1 ./service/store/... ./system/resource/... ./system/supervisor/... \
                          ./service/pg/... ./service/terminal/...                             # ok
go vet ./service/... ./system/... ./api/...                                                   # clean
gofmt -l service system api boot                                                              # clean
make build-wippy-local                                                                        # ok

For reviewer attention

ServiceUpdate remains an outbound-only state notification with no inbound handler. Managers outside this change still publish *supervisor.Entry under that kind, where it is silently ignored — pre-existing, and the same class of bug this PR fixes for the store managers. Worth a follow-up sweep.

https://claude.ai/code/session_01P1t1gnD4Qwd4afzCwwmtQj