Skip to content

test(e2e): gate heavy GPU serves to the merge queue #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fredespi wants to merge 1 commit into
base: main
Choose a base branch
from
+112 −33
Open

test(e2e): gate heavy GPU serves to the merge queue #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -756,6 +756,12 @@ jobs:
# workflow sets this unconditionally; here it lets a scoped dispatch confirm
# a single nightly scenario (e.g. the 27B serve) without the full nightly run.
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Opt-in @merge-queue serves: the heavy real serves (default-engine +
# readiness) run only in the merge queue, where a cheaper per-engine canary
# (scenarios 5 vLLM / 7 lemonade) has already guarded the PR. This keeps the
# per-PR GPU run short while still exercising the full serve matrix before a
# change lands. Only set on the `merge_group` event.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

Expand Down Expand Up @@ -954,6 +960,8 @@ jobs:
# Match the MI300X dispatch path: opt into the platform-adaptive large-model
# scenario only when the manual include_nightly input is enabled.
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Heavy @merge-queue serves run only in the merge queue; see e2e-gpu.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

Expand Down Expand Up @@ -1097,6 +1105,8 @@ jobs:
# Match the Linux Strix dispatch path: opt into the platform-adaptive
# large-model scenario only when the manual input is enabled.
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Heavy @merge-queue serves run only in the merge queue; see e2e-gpu.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

Expand Down
15 changes: 10 additions & 5 deletions tests/e2e-cucumber/features/model_serving.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ Feature: Model serving

# vLLM serve + inference (safetensors model). Engine coverage: vLLM. This is the
# deliberate vLLM half of a per-engine pair with `serve-lemonade-inference`
# below, so it stays pinned to vLLM (the slug names the engine).
# below, so it stays pinned to vLLM (the slug names the engine). It is also the
# vLLM per-PR canary: one real vLLM serve runs on every PR so a broken serve is
# caught before merge, while the heavier `@merge-queue` serves (6, 6b, 8) run
# only in the merge queue.
@id:serve-vllm-inference @requires-gpu @requires-engine:vllm
Scenario: 5 - A served model responds to inference requests on vLLM
Given a managed runtime is active
Expand All @@ -54,7 +57,9 @@ Feature: Model serving
Then the response contains a model reply
And the response identifies the correct model

# Lemonade serve + inference (GGUF model). Engine coverage: Lemonade.
# Lemonade serve + inference (GGUF model). Engine coverage: Lemonade. The
# lemonade per-PR canary: one real lemonade serve runs on every PR (the
# counterpart to the vLLM canary above).
@id:serve-lemonade-inference @requires-gpu @requires-engine:lemonade
Scenario: 7 - A model served on lemonade responds to inference requests
Given a managed runtime is active
Expand All @@ -66,15 +71,15 @@ Feature: Model serving
# Default-engine serve (no --engine): the effective engine is the platform
# default from the capability probe. xfail only where that resolves to vLLM
# (EAI-7333) — see expectations.toml.
@id:serve-default-engine-working-endpoint @requires-gpu
@id:serve-default-engine-working-endpoint @requires-gpu @merge-queue
Scenario: 6 - Serving a model without specifying an engine produces a working endpoint
Given a managed runtime is active
When the user serves a model without specifying an engine
Then an engine is selected automatically
And the model is reachable

# The inference half of scenario 6.
@id:serve-default-engine-inference @requires-gpu
@id:serve-default-engine-inference @requires-gpu @merge-queue
Scenario: 6b - A default-engine served model responds to inference requests
Given a managed runtime is active
When the user serves a model without specifying an engine
Expand All @@ -96,7 +101,7 @@ Feature: Model serving
# Engine-agnostic — the served model+engine follow the host (see
# `a model is being served on GPU`), so this holds the contract on every GPU
# platform. Where it resolves to vLLM, EAI-7333 makes it xfail (expectations.toml).
@id:serve-readiness-contract @requires-gpu
@id:serve-readiness-contract @requires-gpu @merge-queue
Scenario: 8 - A service reported ready can immediately serve inference
Given a managed runtime is active
And a model is being served on GPU
Expand Down
107 changes: 80 additions & 27 deletions tests/e2e-cucumber/src/expectation.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ const REQUIRES_NO_GPU_TAG: &str = "requires-no-gpu";
const SERVE_TIMEOUT_PREFIX: &str = "serve-timeout:";
const NIGHTLY_TAG: &str = "nightly";
const LIFECYCLE_TAG: &str = "lifecycle";
const MERGE_QUEUE_TAG: &str = "merge-queue";

/// The resolved expectation for one scenario on one host.
#[derive(Debug, Clone, PartialEq, Eq)]
Expand Down Expand Up @@ -78,6 +79,12 @@ pub struct ScenarioDecl {
/// so the fast E2E suite stays fast, and only runs when the caller opts in via
/// `E2E_INCLUDE_LIFECYCLE`. Runs explicitly in heavy CI and on demand.
pub lifecycle: bool,
/// `@merge-queue`: a heavy real-GPU serve that is redundant with a cheaper
/// per-engine canary on the fast path, so it is skipped on ordinary per-PR /
/// on-demand runs and only runs in the merge queue, which opts in via
/// `E2E_MERGE_QUEUE`. Keeps the PR feedback loop short while still exercising
/// the full serve matrix before a change lands.
pub merge_queue: bool,
}

impl ScenarioDecl {
Expand All @@ -92,6 +99,7 @@ impl ScenarioDecl {
let mut serve_timeout_secs = None;
let mut nightly = false;
let mut lifecycle = false;
let mut merge_queue = false;
for tag in tags {
let tag = tag
.as_ref()
Expand All @@ -113,6 +121,8 @@ impl ScenarioDecl {
nightly = true;
} else if tag == LIFECYCLE_TAG {
lifecycle = true;
} else if tag == MERGE_QUEUE_TAG {
merge_queue = true;
}
}
Self {
Expand All @@ -124,6 +134,7 @@ impl ScenarioDecl {
serve_timeout_secs,
nightly,
lifecycle,
merge_queue,
}
}

Expand Down Expand Up @@ -311,8 +322,9 @@ pub struct PlatformManifest<'a> {
/// Resolve a scenario's expectation on this host.
///
/// 1. Not-applicable → `Skip`: a `@nightly` scenario when nightly isn't included,
/// a `@requires-gpu` scenario on a host with no AMD GPU, a `@requires-os:<os>`
/// scenario on a different OS, or a scenario whose effective engine can't start.
/// a `@merge-queue` scenario outside the merge queue, a `@requires-gpu`
/// scenario on a host with no AMD GPU, a `@requires-os:<os>` scenario on a
/// different OS, or a scenario whose effective engine can't start.
/// 2. First matching `expectations.toml` condition → `ExpectXfail`.
/// 3. Otherwise → `ExpectPass`.
///
Expand All @@ -321,12 +333,16 @@ pub struct PlatformManifest<'a> {
/// scenarios stay out of the fast path. `include_lifecycle` is set (via
/// `E2E_INCLUDE_LIFECYCLE`) only when the caller opts into the expensive,
/// OS-mutating release-lifecycle scenarios; the default fast suite keeps them out.
/// `include_merge_queue` is set only in the merge queue (via `E2E_MERGE_QUEUE`);
/// per-PR runs pass `false` so heavy `@merge-queue` serves stay off the PR path (a
/// cheaper per-engine canary covers them) and run once before the change lands.
pub fn resolve(
decl: &ScenarioDecl,
cap: &HostCapability,
matrix: &Expectations,
include_nightly: bool,
include_lifecycle: bool,
include_merge_queue: bool,
) -> Expectation {
// (1) Applicability / skip.
if decl.nightly && !include_nightly {
Expand All @@ -339,6 +355,11 @@ pub fn resolve(
reason: "lifecycle-only scenario; set E2E_INCLUDE_LIFECYCLE=1 to run".to_owned(),
};
}
if decl.merge_queue && !include_merge_queue {
return Expectation::Skip {
reason: "merge-queue-only scenario; set E2E_MERGE_QUEUE to run".to_owned(),
};
}
if decl.requires_gpu && !cap.has_amd_gpu {
return Expectation::Skip {
reason: "requires an AMD GPU; none detected on this host".to_owned(),
Expand Down Expand Up @@ -524,17 +545,17 @@ serve_timeout_secs = 90
let d = decl(&["id:big", "requires-gpu", "nightly"]);
assert!(d.nightly);
assert!(matches!(
resolve(&d, &cap("mi300x"), &m, false, false),
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::Skip { .. }
));
assert_eq!(
resolve(&d, &cap("mi300x"), &m, true, false),
resolve(&d, &cap("mi300x"), &m, true, false, false),
Expectation::ExpectPass
);
// The nightly gate is cheapest-first: a @nightly scenario that ALSO can't
// run here (no GPU) still skips regardless of the include flag.
assert!(matches!(
resolve(&d, &cap("mock"), &m, true, false),
resolve(&d, &cap("mock"), &m, true, false, false),
Expectation::Skip { .. }
));
}
Expand All @@ -543,24 +564,56 @@ serve_timeout_secs = 90
fn lifecycle_scenario_skips_unless_included() {
let m = Expectations::default();
// A @lifecycle scenario is skipped on the default fast path and runs only
// when the caller opts in via E2E_INCLUDE_LIFECYCLE (the last arg).
// when the caller opts in via E2E_INCLUDE_LIFECYCLE.
let d = decl(&[
"id:lifecycle-linux-install",
"requires-os:linux",
"lifecycle",
]);
assert!(d.lifecycle);
assert!(matches!(
resolve(&d, &cap("strix-ubuntu"), &m, false, false),
resolve(&d, &cap("strix-ubuntu"), &m, false, false, false),
Expectation::Skip { .. }
));
assert_eq!(
resolve(&d, &cap("strix-ubuntu"), &m, false, true),
resolve(&d, &cap("strix-ubuntu"), &m, false, true, false),
Expectation::ExpectPass
);
// Even when included, an inapplicable OS still skips (os gate is checked).
assert!(matches!(
resolve(&d, &cap("strix-windows"), &m, false, true),
resolve(&d, &cap("strix-windows"), &m, false, true, false),
Expectation::Skip { .. }
));
}

#[test]
fn merge_queue_scenario_skips_unless_included() {
let m = Expectations::default();
// A @merge-queue GPU serve on an applicable host: skipped on the per-PR
// fast path (a cheaper canary covers it), runs in the merge queue.
let d = decl(&[
"id:serve-default-engine-inference",
"requires-gpu",
"merge-queue",
]);
assert!(d.merge_queue);
assert!(matches!(
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::Skip { .. }
));
assert_eq!(
resolve(&d, &cap("mi300x"), &m, false, false, true),
Expectation::ExpectPass
);
// Independent of the nightly axis: a merge-queue scenario is not opted in
// by E2E_INCLUDE_NIGHTLY.
assert!(matches!(
resolve(&d, &cap("mi300x"), &m, true, false, false),
Expectation::Skip { .. }
));
// Cheapest-first: still skips where it can't run at all (no GPU).
assert!(matches!(
resolve(&d, &cap("mock"), &m, false, false, true),
Expectation::Skip { .. }
));
}
Expand All @@ -585,17 +638,17 @@ serve_timeout_secs = 90

// MI300X: default engine vLLM → xfail.
assert!(matches!(
resolve(&d, &cap("mi300x"), &m, false, false),
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::ExpectXfail { .. }
));
// Strix Ubuntu: gfx1151 → lemonade default → NOT vLLM → expect-pass.
assert_eq!(
resolve(&d, &cap("strix-ubuntu"), &m, false, false),
resolve(&d, &cap("strix-ubuntu"), &m, false, false, false),
Expectation::ExpectPass
);
// Strix Windows: lemonade default → expect-pass (this is the XPASS fix).
assert_eq!(
resolve(&d, &cap("strix-windows"), &m, false, false),
resolve(&d, &cap("strix-windows"), &m, false, false, false),
Expectation::ExpectPass
);
}
Expand All @@ -605,7 +658,7 @@ serve_timeout_secs = 90
let m = eai7333_matrix();
let d = decl(&["id:serve-default-engine-inference", "requires-gpu"]);
assert!(matches!(
resolve(&d, &cap("mock"), &m, false, false),
resolve(&d, &cap("mock"), &m, false, false, false),
Expectation::Skip { .. }
));
}
Expand All @@ -616,16 +669,16 @@ serve_timeout_secs = 90
let d = decl(&["id:serve-no-gpu-fails-fast", "requires-no-gpu"]);
// The mock host has no AMD GPU → the no-GPU premise applies → runs.
assert_eq!(
resolve(&d, &cap("mock"), &m, false, false),
resolve(&d, &cap("mock"), &m, false, false, false),
Expectation::ExpectPass
);
// Every GPU host skips it — the premise can't hold there.
assert!(matches!(
resolve(&d, &cap("mi300x"), &m, false, false),
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::Skip { .. }
));
assert!(matches!(
resolve(&d, &cap("strix-ubuntu"), &m, false, false),
resolve(&d, &cap("strix-ubuntu"), &m, false, false, false),
Expectation::Skip { .. }
));
}
Expand All @@ -641,12 +694,12 @@ serve_timeout_secs = 90
]);
// MI300X: vLLM available → not skipped (expect-pass here, no matrix entry).
assert_eq!(
resolve(&d, &cap("mi300x"), &m, false, false),
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::ExpectPass
);
// Strix Windows: vLLM can't start → skip (N/A).
assert!(matches!(
resolve(&d, &cap("strix-windows"), &m, false, false),
resolve(&d, &cap("strix-windows"), &m, false, false, false),
Expectation::Skip { .. }
));
}
Expand All @@ -660,15 +713,15 @@ serve_timeout_secs = 90
// Runs on a Linux GPU host; skips where os_family != linux (windows, and
// the "other" fixture host).
assert_eq!(
resolve(&d, &cap("strix-ubuntu"), &m, false, false),
resolve(&d, &cap("strix-ubuntu"), &m, false, false, false),
Expectation::ExpectPass
);
assert!(matches!(
resolve(&d, &cap("strix-windows"), &m, false, false),
resolve(&d, &cap("strix-windows"), &m, false, false, false),
Expectation::Skip { .. }
));
assert!(matches!(
resolve(&d, &cap("mock"), &m, false, false),
resolve(&d, &cap("mock"), &m, false, false, false),
Expectation::Skip { .. }
));
}
Expand All @@ -678,11 +731,11 @@ serve_timeout_secs = 90
let m = Expectations::default();
let d = decl(&["id:examine-version"]);
assert_eq!(
resolve(&d, &cap("mock"), &m, false, false),
resolve(&d, &cap("mock"), &m, false, false, false),
Expectation::ExpectPass
);
assert_eq!(
resolve(&d, &cap("mi300x"), &m, false, false),
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::ExpectPass
);
}
Expand All @@ -701,11 +754,11 @@ reason = "short-name not surfaced"
let d = decl(&["id:serve-short-name-expansion"]);
// No requires-gpu → runs everywhere, always xfail.
assert!(matches!(
resolve(&d, &cap("mock"), &m, false, false),
resolve(&d, &cap("mock"), &m, false, false, false),
Expectation::ExpectXfail { .. }
));
assert!(matches!(
resolve(&d, &cap("mi300x"), &m, false, false),
resolve(&d, &cap("mi300x"), &m, false, false, false),
Expectation::ExpectXfail { .. }
));
}
Expand All @@ -728,12 +781,12 @@ reason = "lemonade vulkan fallback"
]);
// Strix Ubuntu (linux, lemonade) → xfail.
assert!(matches!(
resolve(&d, &cap("strix-ubuntu"), &m, false, false),
resolve(&d, &cap("strix-ubuntu"), &m, false, false, false),
Expectation::ExpectXfail { .. }
));
// Strix Windows (windows, lemonade) → os mismatch → expect-pass.
assert_eq!(
resolve(&d, &cap("strix-windows"), &m, false, false),
resolve(&d, &cap("strix-windows"), &m, false, false, false),
Expectation::ExpectPass
);
}
Expand Down
Loading