Skip to content

v0.29.0.dev20260911011154

v0.29.0.dev20260911011154: [CI] Move model parity to nightly and /ci parity runs (#706)
@WindChimeRan WindChimeRan tagged this 11 Sep 01:09 
## Motivation

Our current CI is running slow e2e smoke test on every commit / PR. 

This PR aims to 
* make per commit / PR CI **faster**, by removing the slow e2e test.
* move the slow e2e tests to nightly CI, and it can be manually
triggered by `/ci parity` with write access. In the future we can
improve the test coverage, it's ok to be **slower** (e.g., add
speculative decoding. It should have been catch a recent correctness
regression).


## Summary

Adds a daily run at 07:17 UTC on `main` and `/ci parity` for PRs,
restricted to users with repository write access.

Each run tests the captured commit against native `mlx-lm`: Qwen3-0.6B
and Qwen3.5-0.8B, macOS 15/26, 40 shared prompts, 20 output tokens,
top-K 5, and HTTP request batch sizes 1/2.

CI and local use invoke the same `tools/check_parity.py` command. For
each model/OS, it generates one native MLX reference, exits that
process, starts one `vllm serve` instance, checks `/health`, compares
prompts through `/v1/completions` first individually, then in pairs
within one request, and shuts the server down. The offline Metal
execution path is removed.

The `Parity` check reports the tested SHA and whether it matches the
current PR head at report time. It links to per-model summaries and
artifacts containing the native reference, server log, per-batch
results, and environment versions, retained for 14 days. Regular PR CI
retains wheel validation, Metal platform checks, and non-slow tests.

Depends on #705. The comment and scheduled triggers activate once the
workflow is on the default branch.

## Validation

- The canonical CLI passed all 160 local comparisons across both models,
40 prompts, 20 output tokens, and HTTP request batch sizes 1/2: 124
EXACT, 36 TOP_K_MATCH, 0 FAIL. Each model used one server for 40
single-prompt and 20 paired requests, followed by verified process
cleanup. Local validation used `VLLM_METAL_MEMORY_FRACTION=0.10`; CI
retains `0.65`.
- All 18 focused comparison, request-batching, orchestration, and
reporting tests passed, including response ordering, partial final
batches, and missing/duplicate choices. Ruff and workflow validation
passed.
- The full hosted matrix has not been completed for this version.

## Read the result

click `Actions` tab in github, scroll down and find the run. 

<img width="1900" height="812" alt="image"
src="https://github.com/user-attachments/assets/c540546f-08d6-495e-a089-d7a22eb1ad4a"
/>


## Lesson learnt

1. VLLM_METAL_MEMORY_FRACTION
- **No**: `VLLM_METAL_MEMORY_FRACTION=0.80`: hit the 45-minute timeout
- **Yes**: `0.65`: the same job passed in 9m33s Keep this setting for
the small GHA runners.

2. Different Metal compilation behavior
- Metal Shading Language 3.2 on macOS 15 
- Metal Shading Language 4.0 on macOS 26
- So this CI keep both

## Acknowledgement

Learnt a lot from this post:
https://vllm.ai/blog/2026-07-16-keeping-vllm-production-quality

---------

Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
Signed-off-by: Ranran <ranranhaoranzhang@gmail.com>
Assets 2
Loading