GitHub-hosted runner VM cleanup guarantees and Docker CREATE timeout behavior

[artoooo-max]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

ARC (Actions Runner Controller)

Discussion Details

Hello GitHub Community,

We are evaluating a controlled test workload using GitHub-hosted Ubuntu 24.04 runners.

Before executing any tests, we would appreciate clarification on two technical questions.

1. VM cleanup and recovery guarantees

If the workload host, independent watchdog, and Docker daemon fail, GitHub documentation states that the hosted runner VM is eventually decommissioned.

• Is there a guaranteed maximum time for this cleanup?
• Is it possible to obtain independent confirmation that the VM and its runtime resources have been fully removed?
• Can this be guaranteed within 1,100 seconds from the beginning of the workload?

2. Docker CREATE request with a lost response

Suppose a Docker CREATE request reaches the daemon, but the client times out or loses the response.

• Is there a supported way to verify that the CREATE operation has fully settled?
• How can we ensure that no container will appear later from the original request?
• Does GitHub provide any platform-level recovery mechanism for this situation?

We understand that an empty container list or a terminated Docker CLI process does not necessarily prove that an outstanding CREATE operation has completed.

We are looking for documented guarantees, supported interfaces, or explicit limitations.

No workload has been executed yet.

Thank you for your guidance.

[sohom-since-006]

1. VM Cleanup and Recovery Guarantees

• Guaranteed Maximum Time: GitHub does not publish a strict, hard- SLA or guaranteed maximum duration (such as 1,100 seconds) for complete infrastructure-level VM decommissioning if a total host/watchdog failure occurs, though standard ephemeral cleanups typically happen rapidly upon job termination or timeout.
• Independent Confirmation: There is no user-accessible API interface or out-of-band telemetry hook to obtain independent cryptographic or infrastructure-level proof that a specific ephemeral runner VM and its underlying block storage have been fully purged.
• 1,100-Second Guarantee: Because teardown timing depends on underlying cloud provider orchestration queues, a strict, universal guarantee within 1,100 seconds cannot be formally certified through public documentation.

2. Docker CREATE Request with a Lost Response

• Verifying Settled State: If a client connection drops or times out during a POST /containers/create call, there is no automatic synchronous acknowledgement. You must query the Docker daemon state directly via docker ps -a using filters (such as matching container names, labels, or creation timestamps) to see if the container object was successfully registered in the daemon's metadata store.
• Preventing Ghost Containers: The Docker daemon processes container creation sequentially in its state machine. If the API client lost the response, the container may either exist in a Created state or failed initialization. To ensure no orphaned container runs or lingers, an explicit list/inspect query or a prune operation (docker container prune) is required, as the Docker daemon does not automatically garbage-collect unattached created containers unless configured via specific lifecycle flags.
• Platform-Level Recovery: GitHub does not provide custom platform-level reconciliation hooks for dropped local Docker daemon TCP/socket streams; container state management remains bound to standard Docker engine behaviors on the runner instance.

[christoferpedrosanchezzuzunaga-eng]

Good questions @artoooo-max,

Here is how GitHub-hosted runner lifecycles behave under the hood:

1. VM cleanup guarantees: GitHub-hosted runners are strictly single-tenant ephemeral Azure virtual machines. Once your job completes, times out (maximum 6 hours or 360 minutes), or errors out, the entire VM is destroyed and reimaged at the hypervisor level. No processes, Docker containers, or disk state survive past the VM deprovisioning.
2. Docker CREATE hanging: If a client loses connection to the Docker daemon inside a hosted runner, that container only lives for the remaining duration of that specific runner job. It is physically impossible for an orphaned container to persist into another workflow or another user's job, because the next job gets an entirely clean, freshly booted VM image.

[Sagargupta16]

GitHub publishes no cleanup SLA and no way to confirm from outside that a VM is gone, so none of the three guarantees in your first question exist in writing. What is documented: each job runs in a fresh instance of the runner image (https://docs.github.com/en/actions/how-tos/write-workflows/choose-where-workflows-run/choose-the-runner-for-a-job), a hosted job is terminated at 6 hours of execution time (https://docs.github.com/en/actions/reference/limits), and a queued run not picked up within 45 minutes is discarded (https://docs.github.com/en/actions/concepts/runners/github-hosted-runners). The bound you control is timeout-minutes on the job: set it to 18 and the runner service kills the job at 1,080 s and the VM goes with it, but "at or after" is all you get, not "within".

For the second question, make the create idempotent instead of trying to prove a negative. Pass --name and a label such as --label run=$GITHUB_RUN_ID. The Engine API returns 409 when a container with that name already exists, so a retry after a lost response either succeeds or tells you the first request landed (https://docs.docker.com/reference/api/engine/latest/, ContainerCreate). docker ps -a --filter label=run=$GITHUB_RUN_ID then lists every container the daemon created for this job, including ones whose create response you never received. There is no platform-level recovery beyond the VM being discarded with the job.