Skip to content

cleanup_routine force-deletes ALL exited containers on the host (cross-tenant destruction) #176

Description

@AUTHENSOR

Summary

cleanup_routine() in airtbench/kernel.py lists all containers on the Docker daemon and force-deletes any in exited state, with no filter on image, label, name, or ownership.

# kernel.py:690-707
containers = await client.containers.list(all=True)
for container in containers:
    container_info = await container.show()
    if container_info.get("State", {}).get("Status") == "exited":
        await container.delete(force=True)

This routine is called every 5 steps inside each concurrent challenge's loop (main.py:660-665, default concurrency=3). On a shared Docker host, any other workload's stopped container — another benchmark run, a developer's containers, CI sidecars — gets docker rm -f'd.

Reproduction

# Terminal 1: start an unrelated container
docker run -d alpine sleep 1

# Terminal 2: run AIRTBench
python -m airtbench --concurrency 3 ...

# Within 5 steps, the alpine container is force-deleted by cleanup_routine.

Impact

Cross-tenant container destruction on shared Docker hosts. AIRTBench is designed to run with concurrency=3 on a shared host; the cleanup routine destroys any unrelated exited container it finds, with no label/image/name filter to scope it to AIRTBench-owned containers.

Proposed fix

I have a fix on a branch at AUTHENSOR/AIRTBench-Code:fix/cleanup-routine-scope-to-airtbench-containers. (Opening as an issue rather than a PR because fork-based PRs appear to be restricted at the org level.)

The fix:

  1. Adds a "Labels": {"airtbench": "true"} field to the container config in PythonKernel.__aenter__ (kernel.py container creation).
  2. Filters cleanup_routine to only delete containers with that label (client-side label check on the already-fetched container_info, avoiding Docker API filter-param encoding subtleties).

The label check is done client-side rather than via the Docker API filters query param, to avoid query-param encoding subtleties in aiodocker's containers.list(**kwargs) passthrough.

# Fixed cleanup_routine
containers = await client.containers.list(all=True)
for container in containers:
    container_info = await container.show()
    labels = container_info.get("Config", {}).get("Labels") or {}
    if labels.get("airtbench") != "true":
        continue
    if container_info.get("State", {}).get("Status") == "exited":
        await container.delete(force=True)

ruff check introduces zero new errors (verified: 11 errors before and after, all pre-existing BLE001 in unchanged code).

Happy to open a PR directly if the org allows it, or feel free to cherry-pick from the branch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions