Skip to content

[Klaud Cold] Revert #1512 "Fix glm5-fp8-b300 DeepGemm regression" (experimental-only PR)#1536

Merged
functionstackx merged 1 commit into
mainfrom
klaud-cold/revert-1512-experimental-merge
May 20, 2026
Merged

[Klaud Cold] Revert #1512 "Fix glm5-fp8-b300 DeepGemm regression" (experimental-only PR)#1536
functionstackx merged 1 commit into
mainfrom
klaud-cold/revert-1512-experimental-merge

Conversation

@functionstackx
Copy link
Copy Markdown
Collaborator

Summary

Reverts commit 12fb33ea (#1512) which was explicitly marked "Not for merge" — it was an upstream-debugging experiment for sgl-project/sglang#25551, pinning sgl-deep-gemm==0.0.1 and re-enabling JIT DeepGemm in the glm5-fp8-b300 launch scripts.

Result reported upstream (sglang#25551 comment): the pin fixes conc 4..64 but still crashes at conc=128 with the same runtime_utils.hpp:143 signature. Since the sweep matrix runs conc=128 (and #1512 is now in main), production glm5-fp8-b300{,-mtp} sweeps would regress.

This restores the prior SGL_ENABLE_JIT_DEEPGEMM=0 workaround so production sweeps stay green at all conc levels while the upstream investigation continues with Trevor's experiment #2.

Test plan

  • glm5-fp8-b300-sglang + glm5-fp8-b300-sglang-mtp sweeps complete green at all conc tiers (including 128).

🤖 Generated with Claude Code

@functionstackx functionstackx requested a review from a team May 20, 2026 21:15
@functionstackx functionstackx merged commit f4b05bc into main May 20, 2026
6 checks passed
@functionstackx functionstackx deleted the klaud-cold/revert-1512-experimental-merge branch May 20, 2026 21:16
@github-actions
Copy link
Copy Markdown
Contributor

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, clean revert of the experimental #1512 PR to restore the working SGL_ENABLE_JIT_DEEPGEMM=0 workaround.

Extended reasoning...

Overview

This PR reverts commit 12fb33e (#1512), which was an explicitly-experimental "Not for merge" PR that tried pinning sgl-deep-gemm==0.0.1 and re-enabling JIT DeepGemm to debug an upstream B300 regression (sgl-project/sglang#25551). The experiment confirmed upstream that the pin fixes conc 4..64 but still crashes at conc=128. Since the production sweep matrix includes conc=128, the previous SGL_ENABLE_JIT_DEEPGEMM=0 workaround needs to be restored. The diff touches only two benchmark launch scripts and removes the corresponding perf-changelog entry.

Security risks

None. The changes are confined to benchmark launch scripts that run in CI/test environments. No auth, crypto, permissions, or input handling code is touched. The revert actually removes a --break-system-packages pip install flag, slightly reducing risk by no longer overriding PEP-668 protection.

Level of scrutiny

Low. This is a mechanical revert of a known-experimental PR back to a known-working configuration. The diff is small (~30 lines across 3 files) and the rationale is clearly documented with upstream evidence. Benchmark scripts only.

Other factors

The PR description provides clear evidence (upstream comment link) that the reverted experiment did not fully resolve the regression at higher concurrency tiers. The restored SGL_ENABLE_JIT_DEEPGEMM=0 value matches what was in place before #1512 landed. No outstanding reviewer comments to address — only the auto-generated recipe-reminder bot comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant