[aw-failures] Fix the Warm up model retry loop in daily-byok-ollama-test.md — 100% failure rate

[github-actions]

Fix the Warm up model retry loop in daily-byok-ollama-test.md — it has failed every single scheduled run for the past 8 days

Problem

Daily BYOK Ollama Test has a 100% failure rate — 10/10 scheduled runs since 2026-07-27 have failed at the same step, Warm up model, with:

::error::Model 'qwen2.5:0.5b' failed to warm up after 10 attempts.
##[error]Process completed with exit code 1.

No open issue currently tracks this — it wasn't in the correlated-issue set for this or the prior report window.

Affected workflow / runs

• .github/workflows/daily-byok-ollama-test.md → .github/workflows/daily-byok-ollama-test.lock.yml
• All failing at the agent job, Warm up model step:
  • §31134143397 — 2026-08-07
  • §31053504900 — 2026-08-05
  • §30957197799 — 2026-08-04
  • §30859350200 — 2026-08-03
  • §30670580818 — 2026-07-31
  • §30606886128 — 2026-07-31
  • §30587927074 — 2026-07-30
  • §30496744608 — 2026-07-29
  • §30405288231 — 2026-07-28
  • §30311449929 — 2026-07-27

Probable root cause

From §31134143397 job log:

• Start Ollama service succeeds ("Ollama is ready").
• Pull small model succeeds — qwen2.5:0.5b (397 MB) downloads and verifies cleanly.
• Warm up model then calls curl -sf (localhost/redacted) -d '{"model":"qwen2.5:0.5b","prompt":"hi","stream":false}' in a loop of 10 attempts, 3s apart. Every single attempt fails (not "eventually succeeds after a few retries" — zero successes across ~95 seconds), and because -sf is used, the actual curl exit code / HTTP status / response body is never logged, so the real failure reason (timeout, 4xx/5xx, OOM-killed ollama serve, or an /api/generate request-shape change in the pinned Ollama v0.31.1) is invisible.
• Given the model just finished pulling (not yet resident in memory) and 0 of 10 attempts ever succeed, this looks less like "model needs one warm-up call" and more like a hard incompatibility or crash — worth checking whether ollama serve is even still alive by the last attempt, and whether v0.31.1's /api/generate schema matches this payload.

Proposed remediation

1. Add diagnostics to the warm-up loop: drop -sf, capture and print curl's HTTP status code and body on each failed attempt, and run ollama ps / check the ollama serve process is still alive before giving up.
2. Verify Ollama v0.31.1's /api/generate request/response contract still matches what's being sent (compare against ollama --version changelog since this pin was set).
3. Confirm the GitHub-hosted runner has enough free memory/CPU to load qwen2.5:0.5b — check for an OOM kill of ollama serve in dmesg/journalctl during a failing run.
4. Once root cause is identified, either fix the incompatibility or replace the fixed 10-attempt/3s-sleep loop with a longer exponential backoff if the real issue is just slow first-load latency.

Success criteria / verification

• Daily BYOK Ollama Test completes the Warm up model step successfully on the next scheduled run.
• Re-run the workflow manually (workflow_dispatch if available, or wait for next schedule) and confirm agent job conclusion is success.
• If the fix is diagnostics-only, the next failure must produce an actionable HTTP status/body instead of a bare "failed to warm up" message.
  Related to [aw-failures] [aw] Failure Investigator Report — 2026-08-07 (6h) #50968

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 214.9 AIC · ⊞ 5.5K · ◷

• expires on Aug 13, 2026, 5:54 PM UTC-08:00

[github-actions]

🍪 Issue Monster selected this for Copilot

I have identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · gpt54 · 1.28 AIC · ⊞ 8.2K · ◷