Proposal: Feature-Driven Conformance for AI Workloads

[hh]

Proposal: Feature-Driven Conformance for AI Workloads

Motivation: Why .feature files?

The AI Conformance project faces a unique challenge: The people defining the requirements are rarely the people writing the code.

Our stakeholders are GPU hardware vendors, Large Model builders, and AI platform engineers. They think in terms of "Capabilities" and "Contracts" (e.g., “A node with this taint must accept this toleration”), not in terms of Go structs or ClientSets.

We propose using Gherkin (.feature) files not as a test framework, but as the Domain-Specific Language (DSL) for our Conformance Contract.

1. Removing the "Language Barrier"

• Current State: To understand what is required for "AI Conformance," a stakeholder must read test/e2e Go code. This effectively excludes non-Go developers (PMs, Data Scientists, Hardware Vendors) from the review process.
• Proposed State: Feature files provide a plain-English, vendor-neutral format. An NVIDIA or AMD representative can read a .feature file and approve the logic without needing to understand the implementation.

2. Separating "Policy" from "Verification"

• Current State: The "Rule" (the Spec) and the "Check" (the Test) are tightly coupled in the same file. Changing a requirement looks like a code refactor, making it hard to track policy changes over time.
• Proposed State:
  • The .feature file is the Policy. PRs here represent governance decisions ("Should we require this behavior?").
  • The .go file is the Verification. PRs here represent engineering decisions ("Does this code correctly test the policy?").
    This separation allows the AI WG to iterate on policy rapidly without getting bogged down in code review.

3. The "Lost in Translation" Effect

• Current State: A domain expert writes a requirement in a Google Doc. A developer interprets it and writes a Go test. Often, nuances are lost (e.g., specific timing, error states, or resource quantities), and the Go test becomes the de-facto spec, even if it's wrong.
• Proposed State: The .feature file is the spec. The developer's job is simply to map steps to code. If the code logic drifts from the English description, our proposed tooling flags it immediately.