[Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs

[weiqingy]

Search before asking

• I searched in the issues and found nothing similar.

Description

The it-java and it-python CI jobs fail intermittently for reasons unrelated to the code under test. A recent run shows both failure modes at once, with every unit-test job green.

1. The Ollama install step fails on a transient network blip. tools/start_ollama_server.sh pipes https://ollama.com/install.sh into sh. That upstream script probes for ollama-linux-${ARCH}.tar.zst with a curl --fail --silent --head, and on any failure of that single probe it falls through to a .tgz asset. Current releases publish only .tar.zst — .tgz returns 404. So one flaky HEAD request turns into a hard failure:

>>> Downloading ollama-linux-amd64.tgz
curl: (22) The requested URL returned error: 404

gzip: stdin: unexpected end of file
tar: Child returned status 1
##[error]Process completed with exit code 2.

Two it-java jobs failed this way, before compiling or running a single test. Retrying the install would recover, since the underlying asset does exist.

2. The wrapper masks a failed download. In tools/start_ollama_server.sh, ret=$? is read after curl -fsSL ... | sh, so it captures the exit status of sh, not curl. Without pipefail, a failed curl leaves sh reading empty stdin and exiting 0 — CI then proceeds with no Ollama installed and fails later somewhere less obvious.

3. Integration jobs have no runtime bound. No job in ci.yml sets timeout-minutes. In the run above, it-python [python-3.12] [flink-2.2] hung and was killed at GitHub's 6-hour ceiling (6h0m16s), consuming shared runner capacity for a result that was knowable in minutes. A timeout-minutes on the integration jobs would surface the same failure far sooner.

Open question — the hang itself. In that job Ollama installed correctly, then llama-server crashed while serving qwen3:1.7b:

ollama._types.ResponseError: llama-server process has terminated: signal: segmentation fault (core dumped)

Every failing test there is Ollama-backed; the py4j errors are downstream of the dead model server. I don't yet have evidence for the root cause (upstream regression, runner memory, or model artifact). One option is pinning OLLAMA_VERSION — install.sh supports it, and the workflow already pins uv to 0.11.0 in the adjacent step — which would at least make the toolchain deterministic. I'd welcome opinions on whether that's worth doing, and on how the project wants to handle bumping such a pin.

If this direction sounds right, I'd propose splitting the work into two independent PRs: the retry plus pipefail fix in tools/start_ollama_server.sh, and the timeout-minutes addition in ci.yml.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[weiqingy]

Will work on this.

[weiqingy]

Status map for anyone picking this up, since the work has spread across a few PRs and a sibling issue.

This issue produced two independent PRs:

• [ci] Retry Ollama install and propagate download failures #891 — retry the Ollama install and propagate download failures (fixes the transient .tgz 404).
• [ci] Bound the runtime of the integration-test jobs #892 — bound the integration-test jobs with timeout-minutes so a hung job fails fast instead of running to the 6-hour ceiling.

Both currently show red CI, but the red is not these changes. The Ollama install succeeds in every failing run; only the Ollama-backed integration tests crash, and the failing jobs roam from run to run while the changes stay the same. That intermittent flake is tracked separately in #893.

Root cause (detail in #893): a subset of GitHub ubuntu-latest runner VMs trip an upstream llama.cpp segmentation fault while serving qwen3:1.7b on CPU. A given VM is effectively deterministic, so re-running the whole matrix keeps hitting some cursed VM. The dated onset lines up with a GitHub runner-image rollout — and notably the installed Ollama version has been constant across the green→red boundary, so pinning the Ollama version does not apply here.

The fix is #895: a workflow_run companion that re-runs only the failed integration-test legs on a fresh runner (which re-rolls the VM), capped at two attempts, with unit/lint/build failures never re-run. It was validated end-to-end on a fork. One caveat noted on that PR: workflow_run companions only run from the default branch, so #895 cannot heal its own CI and takes effect only once merged.

Suggested order, entirely at the maintainers' discretion: #895 first — once merged it heals the integration-test flake for every subsequent run, including #891 and #892 — then #891/#892, which are otherwise green apart from the flake. They can also be reviewed independently. I'll leave the sequencing and the decisions to the maintainers.

[weiqingy]

@wenjin272 When you get a chance, could you take a look at this?

CI has been failing more often recently, so it would be good to mitigate this to make PR creation, validation, and review smoother. My suggestion is to prioritize #895 first, since the fresh-runner auto-rerun would unblock the flaky integration-test jobs and give us the most immediate impact.

The map above has the full context.