Skip to content

Close the drain and runner re-registration race #152

Description

@jmcte

Finding

drainRunnerPool remembers a cordoned runner by name. After deleting an idle registration, later polls skip any registration with that name, even if a restarting service has registered a new runner ID. The function then reports drained as soon as no currently visible target is busy.

All Docker plane services use restart: unless-stopped, so same-name/new-ID re-registration is an expected lifecycle behavior, not a theoretical edge case.

Impact

During teardown, scale-in, or token rotation, a replacement registration can become dispatchable, accept a job, and then be destroyed after the controller has already reported a successful drain.

Recommended work

  • Track registration identity, not only runner name.
  • Require target names to remain absent (or their services to be stopped) before returning drained.
  • Coordinate host/container pause or shutdown with GitHub deregistration.
  • Re-check for replacement registrations after deletion and before destructive follow-up.

Acceptance criteria

  • A same-name runner with a new ID is deleted or causes the drain to continue.
  • The drain deadline covers the full cordon/absence verification sequence.
  • A regression test models delete, same-name re-registration, and eventual stable absence.
  • Teardown/scale/rotation do not report success while a target is dispatchable.

Evidence

  • src/lib/drain.ts:46-49,84-116
  • src/lib/compose.ts
  • src/lib/linux-docker-compose.ts:92-102
  • src/lib/windows-compose.ts:93-106
  • test/drain.test.ts:4-125

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:infraInfrastructure, CI, release, governance, scripts, or repo setup.bugSomething isn't workingpriority:P1Codex Connector P1; blocks execution until Athena and Ares validate.reliabilityrisk:highHigh-risk change; validation required.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions