test(integration): de-flake otel pre-job config test#803
Conversation
The test dispatched a quick workflow, waited for it to complete, then SSHed into the runner to read the otel config the pre-job script wrote. Ephemeral runner VMs are torn down on job completion, so the inspection raced the runner-manager cleanup: when cleanup won, get_single_runner found zero VMs and the test failed with an empty runner list. Dispatch the long-running wait workflow and inspect the runner while its job is in progress, so the VM is guaranteed alive. Poll on the config file to absorb the pre-job hook write timing.
| dispatch_input={"runner": app, "minutes": "5"}, | ||
| wait=False, | ||
| ) | ||
| wait_for(lambda: workflow.update() or workflow.status == "in_progress") |
There was a problem hiding this comment.
An AI has flagged the follwoing, which seems reasonable:
The lambda: workflow.update() or workflow.status == "in_progress" expression does not reliably wait for the run to reach in_progress. PyGithub's CompletableGithubObject.update() returns True whenever it performs a successful HTTP GET (200 OK) and False only on a 304 Not Modified. Because update() is the left operand of or, any 200 response short-circuits the expression before status is ever consulted. On the first loop iteration the run object was returned from a list call and has no stored etag, so the first GET is unconditional — it returns True immediately, regardless of what status actually is.
The guard therefore degrades to "refresh once and continue" rather than "wait until the job is running." This undercuts the invariant that lines 219–221 rely on — that the VM is guaranteed to be alive when the inspection begins. The downstream _read_otel_config poll (120 s) masks this in typical conditions, but the structural guarantee the PR intends is lost and the effective startup budget is reduced to a single wait.
The fix is to call update() for its side effect and evaluate status in a separate step:
def _in_progress() -> bool:
workflow.update()
return workflow.status == "in_progress"
wait_for(_in_progress, timeout=300, check_interval=10)
What this PR does
De-flakes
test_otel_collector_endpoint_pre_job_installs_config. Instead of dispatching a quick workflow, waiting for it to complete, then SSHing into the runner, it now dispatches the long-running wait workflow and inspects the runner while its job is in progress. The config read is polled to absorb the timing of the pre-job hook write.Why we need it
The runner is ephemeral, so its OpenStack VM is torn down on job completion. Reading the otel config after completion races the runner-manager cleanup loop — when cleanup wins,
get_single_runnerfinds zero VMs and the test fails withfound more than one runners or no runners: []. This was observed failing on the 22.04 base while passing on 24.04 in the same run (e.g. PR #802 CI). Keeping the job in progress guarantees the VM is alive during inspection.Checklist
docs/changelog.mdwith user-relevant changes — N/A, no user-facing changeterraform fmtpasses andtflintreports no errors — N/Agithub-runner-manager/pyproject.toml— N/A, not changed