Skip to content

zuul: capture glance-api log in post-run#2915

Merged
berendt merged 1 commit into
mainfrom
collect-glance-api-logs
Jun 28, 2026
Merged

zuul: capture glance-api log in post-run#2915
berendt merged 1 commit into
mainfrom
collect-glance-api-logs

Conversation

@ideaship

@ideaship ideaship commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Problem

The nightly testbed lanes intermittently fail when the Octavia Amphora image import gets stuck in Glance's queued state (tracked as octavia-stuck-queued). The image-import step (osism manage image octavia) only reports that the image is stuck; the actual reason — web-download, qcow2→raw conversion, or Ceph write — is logged by glance-api itself.

kolla-ansible configures glance-api with log_file = /var/log/kolla/glance/glance-api.log, so the import error goes to that on-host file, not the container's stdout. As a result docker logs glance_api does not contain it, and nothing in the published job output reveals the failing stage — so the failure can't be diagnosed from CI logs today.

Latest occurrence (for reference)

testbed-upgrade-stable-ubuntu-24.04, periodic-midnight 2026-06-27 — the import stalled at 05:00 UTC and failed the Bootstrap services task (the .CHECKSUM fetch returned 200, so this is the Glance import-state face, not the checksum face):

In job-output.txt you can see ... seems stuck in queued state from the import step — but there is no glance-api log anywhere in that build's artifacts (only job-output.* and the orchestrator debug files). That missing log is exactly what this PR adds. The prior night (2026-06-26, build 4e26fc42, testbed-deploy-stable-in-a-nutshell-with-tempest) shows the same failure on a different lane.

Change

Adds a best-effort post-run play (playbooks/collect-glance-logs.yml) that captures the tail of glance-api.log from every testbed host into the job output, so the failing import stage is visible in the published build logs.

  • The Zuul orchestrator can only reach the manager directly (the nodes are on a private network), so the play hops through the manager and fans out to every host returned by osism get hosts — the same shape as the existing tboperator container-log collector.
  • Runs before the cleanup play tears the environment down.
  • Never fails the job: a missing manager address, an unreachable host, or an unreadable log are all tolerated (sudo -n so it can't hang). The final debug surfaces both stdout and stderr (with an empty-result fallback) so a failed SSH still shows why.
  • Wired into abstract-testbed-deploy's post-run, so every deploy / update / upgrade lane inherits it.

Validation

  • yamllint clean; ansible-lint passes at the production profile.
  • bash -n on both the local wrapper and the remote (manager-side) heredoc body; host-extraction awk checked against a sample osism get hosts table.
  • Not yet exercised against a live failing build — once it captures a real log, the failing import stage (download / qcow2→raw convert / Ceph write) can be read straight from the build's job output.

🤖 Generated with Claude Code

The nightly testbed lanes intermittently fail when the Octavia Amphora
image import gets stuck in Glance's "queued" state. The image-import
step ("osism manage image octavia") only reports that the image is
stuck; the reason (web-download, qcow2->raw conversion, or Ceph write)
is logged by glance-api itself.

kolla-ansible configures glance-api with
"log_file = /var/log/kolla/glance/glance-api.log", so the import error
goes to that on-host file and not to the container's stdout. As a
result "docker logs glance_api" does not contain it, and nothing in the
published job output reveals the failing stage.

Add a best-effort post-run play that captures the tail of
glance-api.log from every testbed host into the job output before the
cleanup play destroys the environment. The Zuul orchestrator can only
reach the manager directly (the nodes are on a private network), so the
play hops through the manager and fans out to every host returned by
"osism get hosts". It never fails the job: a missing manager address,
an unreachable host, or an unreadable log are all tolerated. The final
debug surfaces both stdout and stderr (with an empty-result fallback)
so a failed SSH to the manager still shows why.

Wire the play into abstract-testbed-deploy's post-run so every deploy,
update, and upgrade lane inherits it.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship ideaship marked this pull request as ready for review June 28, 2026 09:31
@berendt berendt merged commit 052a01b into main Jun 28, 2026
3 checks passed
@berendt berendt deleted the collect-glance-api-logs branch June 28, 2026 20:32
@github-project-automation github-project-automation Bot moved this from Ready to Done in Human Board Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants