Reliability Fix: Raise container nofile ulimit to prevent FD exhaustion in AMD multi-node jobs

[hongxiayang]

while doing DI optimization, found that some eval jobs failed.
The failed eval all have max concurrency 1024.

https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28959246164/job/85926289851 , https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28959246164/job/85926289806 , and https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28959246164/job/85926289736 ,

• 85926289851 (1P TP8 x 1D TP8): floods of Too many open files connecting to 10.24.112.105:2584 starting ~line 1341.
• 85926289806 (1P TP4 x 1D TP4): same, connecting to 10.24.112.101:2584 starting ~line 1322.
• 85926289736 (2P TP4 x 1D TP8): same, connecting to both prefill nodes 10.24.112.103/104:2584 starting ~line 1600.

tcp open error caused by: Too many open files (os error 24)

this is because the default ulimit nofile 1024.
We can raise the ulimit nofile to make the high-concurrency eval jobs less error-prone.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Claude finished @hongxiayang's task in 57s —— View job

---

PR Review

• Read the PR diff and surrounding context in benchmarks/multi_node/amd_utils/job.slurm
• Verify the ulimit flag is applied to the correct container and is syntactically valid
• Check for other container launches that might need the same fix

LGTM - no blocking issues found

The 1-line change adds --ulimit nofile=1048576:1048576 to the main engine container launch (job.slurm:576), which is where the vLLM proxy/benchmark client runs and where the Too many open files (os error 24) errors originated. The value and placement are consistent with the existing --ulimit flags, and the router container launch at line 535 already carries the same nofile limit, so both container launches in this script are now covered.

[claude]

LGTM — one-line ulimit bump matching the existing vllm-router container in the same file, with clear FD-exhaustion repro from linked CI runs.

Extended reasoning...

Overview

Single-line change to benchmarks/multi_node/amd_utils/job.slurm adding --ulimit nofile=1048576:1048576 to the main engine docker run invocation. This raises the per-container open-file limit from the default 1024, which the PR description ties to "Too many open files" TCP connect failures during high-concurrency (1024) eval jobs across three linked CI runs.

Security risks

None. Raising an open-file soft/hard limit inside a benchmarking container has no auth, crypto, or data-exposure surface — it's a resource ceiling on an isolated ephemeral container that already runs --privileged with host networking.

Level of scrutiny

Low. The change is benchmark infrastructure only (Slurm launcher for AMD multi-node disagg jobs), not production serving code. It also mirrors an existing pattern: the vllm-router container started ~40 lines above already uses the exact same --ulimit nofile=1048576:1048576, so this just brings the engine container into line with a value the codebase has already vetted.

Other factors

The bug hunter found no issues. The PR description provides concrete failure evidence (three specific job IDs with line-number pointers into the logs) and a plausible root cause (default 1024 nofile vs. 1024 max concurrency). No prior reviewer comments to address.

[Oseltamivir]

lgtm