Skip to content

[Klaud Cold] Update gptoss-fp4-mi325x-vllm vLLM ROCm image to v0.21.0#1467

Merged
functionstackx merged 4 commits into
mainfrom
update-gptoss-fp4-mi325x-vllm-v0.21.0
May 18, 2026
Merged

[Klaud Cold] Update gptoss-fp4-mi325x-vllm vLLM ROCm image to v0.21.0#1467
functionstackx merged 4 commits into
mainfrom
update-gptoss-fp4-mi325x-vllm-v0.21.0

Conversation

@functionstackx
Copy link
Copy Markdown
Collaborator

Summary

Update vLLM ROCm image from v0.17.0 (70d old) to v0.21.0

Recipes touched: `gptoss-fp4-mi325x-vllm`

Test plan

  • full-sweep-enabled sweep passes.

🤖 Generated with Claude Code

Update vLLM ROCm image from v0.17.0 (70d old) to v0.21.0

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

2 similar comments
@github-actions
Copy link
Copy Markdown
Contributor

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

@github-actions
Copy link
Copy Markdown
Contributor

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

@github-actions
Copy link
Copy Markdown
Contributor

Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — straightforward vLLM ROCm image version bump for a single recipe, with corresponding changelog entry.

Extended reasoning...

Overview

This PR bumps the gptoss-fp4-mi325x-vllm recipe's image from vllm/vllm-openai-rocm:v0.17.0 to v0.21.0 (a single-line change in .github/configs/amd-master.yaml) and adds a matching entry to perf-changelog.yaml. Only one recipe is touched.

Security risks

None. These are benchmark config files (image tag + changelog text), not production code or auth/permissions changes.

Level of scrutiny

Low. This is a mechanical image version bump on a perf benchmark recipe, with no logic changes. The target version v0.21.0 is already in use elsewhere in the same file (kimik2.5-*-vllm, minimaxm2.5-*-vllm, gptoss-fp4-mi355x-vllm), so it's a well-validated image tag in this repo.

Other factors

The bug hunting system found no issues. The full-sweep-enabled label means the sweep itself will validate the new image works for this recipe before merge — if the image is broken, CI will catch it. Changelog format matches the existing pattern in the file.

@functionstackx
Copy link
Copy Markdown
Collaborator Author

Diagnosis

All 11 failed jobs landed on the same broken node chi-mi325x-pod1-121.ord.vultr.cpe.ice.amd.com. The enroot squashfs import fails with enroot-aufs2ovlfs: failed to set capabilities: Operation not permitted (repeated 30+ times per job), leaving the container image missing ([ERROR] No such file or directory: /nfsdata/sa/gharunner/gharunners/squash/vllm_vllm-openai-rocm_v0.21.0.sqsh). The benchmark script never starts — this is a node-level infrastructure issue, not an OOM or code bug.

The 8 successful jobs ran on healthy nodes (018, 019, 020, 027) and completed normally. The successful tp=8 benchmark shows 223.71 GiB available KV cache memory with --gpu-memory-utilization 0.95, confirming no memory pressure.

Failed run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26009955168

No code fix needed

This failure is caused by node chi-mi325x-pod1-121 being unable to run enroot container imports (capability-setting permission error). The fix is infrastructure-level: either repair or exclude that node from the Slurm partition. Re-running the sweep should schedule failed jobs on healthy nodes and pass.

Note: the --gpu-memory-utilization 0.90 fix from PR #1403 (mi300x sibling) is not needed here — mi325x has sufficient headroom at 0.95 based on successful runs.

functionstackx added a commit that referenced this pull request May 18, 2026
Root-caused via the failed sweeps on #1467, #1468, #1469 (all three
[Klaud Cold] vLLM v0.21 bumps on different mi325x recipes): every
failure landed on chi-mi325x-pod1-121 with

  enroot-aufs2ovlfs: failed to set capabilities: Operation not permitted

before the .sqsh import even completes; subsequent pyxis mount then
fails with "No such file or directory". The same image works cleanly
on every other up node (017/018/019/020/027) — confirmed not OOM and
not a recipe issue.

This matches the existing pattern for mi300x in #1462 (pin salloc away
from chronically-bad nodes); for mi325x there's currently only the one
node to exclude, so use --exclude rather than --nodelist so we don't
have to maintain the allow-list as nodes come and go.

pod1-121 has separately been drained on the controller with a watchdog
(per KLAUD_DEBUG.md §5.6) so it stays out of the pool until ops fix
the underlying setcap regression.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

@github-actions
Copy link
Copy Markdown
Contributor

@functionstackx
Copy link
Copy Markdown
Collaborator Author

/reuse-sweep-run

@functionstackx functionstackx merged commit bb7e848 into main May 18, 2026
3 of 5 checks passed
@functionstackx functionstackx deleted the update-gptoss-fp4-mi325x-vllm-v0.21.0 branch May 18, 2026 04:39
@github-actions
Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

1 participant