[RFC]vime NPU CI Integration

[Meihan-chen]

1. Motivation & Goals

vime has been extended to Ascend NPU. As NPU features evolve, CI must guard their quality automatically and prevent regressions — making NPU end-to-end validation a standard part of vime CI, alongside the existing x86 (CPU/GPU) CI.

• Goal: Bring up an NPU CI that automatically runs end-to-end validation on real Ascend hardware for commits carrying NPU changes, and reports pass/fail.
• Success criteria:
  • Every push / PR carrying NPU changes automatically runs NPU end-to-end tests;
  • New tests can be brought under coverage without editing the pipeline body;
  • Changes touching the image / dependencies / patches get their full build chain validated.

---

2. Overall Architecture

NPU CI is a standalone Buildkite pipeline, vime-npu-ci, isolated from the x86 CI (own entry YAML, triggers, queue, image), in three layers — image / gating / suite:

vime-npu-ci (standalone pipeline; trigger: push on the branch carrying NPU changes + its PRs)
 └─ .buildkite/pipeline-npu.yaml (entry)
      ├─[image layer]  runnable NPU env for the commit under test  → §3.1
      ├─[gating layer] which builds run, and which tier of tests   → §3.2
      └─[suite layer]  NPU e2e suite (generated by npu_suites.py)  → §3.3

Each layer has a single responsibility and evolves independently: changing the image approach doesn't touch suite definitions; adding a suite doesn't touch gating logic.

A standalone pipeline is used (rather than folding into the x86 pipeline): NPU uses a different hardware queue, image, and credentials; merging them into one pipeline would subject x86 PRs to unrelated queueing and failures. Standalone, its triggers, coverage scope, and notifications are configured independently. Which branch is covered is set via the branch filter (see §4); the pipeline itself is not tied to any branch name.

---

3. Component Design

3.1 Image layer: reuse by default, rebuild on demand

By default, reuse a prebuilt NPU base image (tag vime-latest, with Ascend dependencies already installed); rebuild per commit only when a change may affect the image itself. Most PRs only change Python code, where reusing the image is fastest and accurate; only changes touching the image build chain pay the full build cost (vllm-ascend/megatron/mindspeed builds take a long time).

• Reuse image (default): test steps pull the prebuilt base image, then git checkout ${BUILDKITE_COMMIT} + pip install -e . in-container to install the code under test.
• Rebuild (on demand): enable image-build-npu — build with Dockerfile.npu per commit and push to a private registry; test steps pull that image. Triggered when a PR changes docker/** or docker/npu_patch/**, or on nightly.
• Equivalent to the gating/suite layers: both only expose "a runnable image + code under test."

3.2 Gating layer: core automatic, nightly tiered

The core tests must run automatically to provide real coverage without gaps; large-model / long-running tests go in nightly to control cost.

• pipeline-npu.yaml uploads by tier: core unconditionally (push/PR on the covered branch); nightly triggered by the run-ci-npu-nightly label or a scheduled variable.
• Gating only "selects the tier and uploads the steps"; it does not inline concrete tests.

3.3 Suite layer: npu_suites.py

Dynamically generated, same data-driven style as vime's GPU gpu_suites.py (eases tiering and scaling).

• Each SUITES entry (model/script, card count / resource_class, tier, flags), generated then | buildkite-agent pipeline upload.
• Each step: Ascend queue + resource_class, imagePullSecrets, NPU env vars, model download (modelscope/hf-mirror), run the script and verify the result.
• Features not yet supported on Ascend are added to SOFT_FAIL_ON_NPU: marked soft_fail, does not fail the build.
• Single source of truth: adding a test only edits SUITES.

---

4. Prerequisites

1. Ascend queue: queue name and resource_class (card counts) available to vime-npu-ci; whether core/nightly card needs are met.
2. Private registry + credentials (needed only when rebuilding the image): image namespace, build credentials (Vault), pull secret; reachable remote buildkitd.
3. Network & model sources: stable modelscope / hf-mirror / HF_TOKEN; a clear owner and refresh cadence for the prebuilt base image.
4. Buildkite config: create vime-npu-ci with its entry at .buildkite/pipeline-npu.yaml; set the branch filter to the development branch currently carrying NPU support.

---

5. Roadmap

• Step 0 — Prototype validation: validate one NPU end-to-end test on Ascend hardware (Qwen3-4B training smoke test, modelscope + hf-mirror + training-log check), confirming the queue / image / credentials / network path work. (merged, PR [NPU]: add npu pipeline for CI #282)
• Step 1 — Standalone tiered pipeline: create vime-npu-ci; add npu_suites.py generating core/nightly tiers with existing tests moved into core; remove the always-run NPU step from the shared pipeline.yml to avoid duplicate triggering. 【NPU】vime NPU CI Integration #316
• Step 2 — Rebuild image on demand: enable image-build-npu with path conditions (docker/**, npu_patch/**) and nightly.
• Step 3 — Grow & harden: add model / card-count / feature combinations to SUITES and settle the core/nightly split; update CI docs and establish that pipeline-npu.yaml and npu_suites.py are maintained in sync. [NPU] refactor(ci): device-aware test harness for GPU/NPU + Qwen3 NPU smoke tests #349

Enhancements (later, as needed): diff-aware test pruning by changed path / label.

---

Review and discussion of this RFC are welcome in this issue.