What happened?
The gauge ate.workerpool.workers emits a series with ate_sandbox_class="".
The empty value comes from the worker record, not from the WorkerPool CR. Control.CreateWorker is a public RPC and it does not validate sandbox_class, so a client can register a worker with an empty one. Worker records written before this keep the empty value.
The pool is seeded at 0 under its own class, so the ghost series sits next to a correct series for the same pool. This makes the pool look like it has two populations of workers.
Expected Behavior
The gauge must never emit ate_sandbox_class="".
A worker with no sandbox class cannot run an actor. The scheduler matches the class exactly, so it places nothing on that worker. Therefore the worker must not be counted as free capacity of its pool. It must report as unknown.
A pool of 3 workers with no sandbox class must emit:
ate_workerpool_workers{ate_sandbox_class="gvisor",ate_worker_state="idle",...} 0
ate_workerpool_workers{ate_sandbox_class="gvisor",ate_worker_state="assigned",...} 0
ate_workerpool_workers{ate_sandbox_class="unknown",ate_worker_state="idle",...} 3
The pool keeps its own series at 0, so an idle == 0 alert still fires. The broken workers stay visible under unknown.
Steps to Reproduce
- Register a worker with an empty sandbox class. Call
Control.CreateWorker
with sandbox_class: "". The RPC accepts it.
- Scrape the Prometheus endpoint on ate-api-server:
curl -s http://localhost:9090/metrics | grep "ate_workerpool_workers"
- The pool reports 0 under its own class, and a second series with
ate_sandbox_class="" reports the worker.
Sandbox Runtime
Both / Runtime Agnostic
Agent Substrate Version / Commit SHA
main
Kubernetes Version & Environment
No response
Host OS & Architecture
No response
Relevant Logs and Diagnostic Output
Additional Context
No response
Confirmation
What happened?
The gauge
ate.workerpool.workersemits a series withate_sandbox_class="".The empty value comes from the worker record, not from the WorkerPool CR.
Control.CreateWorkeris a public RPC and it does not validatesandbox_class, so a client can register a worker with an empty one. Worker records written before this keep the empty value.The pool is seeded at 0 under its own class, so the ghost series sits next to a correct series for the same pool. This makes the pool look like it has two populations of workers.
Expected Behavior
The gauge must never emit
ate_sandbox_class="".A worker with no sandbox class cannot run an actor. The scheduler matches the class exactly, so it places nothing on that worker. Therefore the worker must not be counted as free capacity of its pool. It must report as
unknown.A pool of 3 workers with no sandbox class must emit:
The pool keeps its own series at 0, so an
idle == 0alert still fires. The broken workers stay visible underunknown.Steps to Reproduce
Control.CreateWorkerwith
sandbox_class: "". The RPC accepts it.ate_sandbox_class=""reports the worker.Sandbox Runtime
Both / Runtime Agnostic
Agent Substrate Version / Commit SHA
main
Kubernetes Version & Environment
No response
Host OS & Architecture
No response
Relevant Logs and Diagnostic Output
Additional Context
No response
Confirmation
main.