RFC: grouped/sibling ResultSets with rollup semantics (mutation testing, seed sweeps, model comparisons) — follow-up to #36

[adhabnr-ux]

Summary

This is the promised follow-up to #36 ("No representation for grouped/sweep ResultSets with rollup semantics"), which itself grew out of a cross-project conversation in AshwinUgale/muteval#44. #36 flagged the gap precisely but deliberately didn't propose a shape yet. Since filing it I've read the current spec/schemas/resultset.json (post-#35) end to end and looked at how three real systems that already solve some version of this problem represent it. This discussion proposes a concrete direction, grounded in that reading, and is the start of the same comment period #8→#22 went through before #22 became #35.

Restating the gap against the current schema

resultset.json today (verified by reading the live schema, not from memory) has no field anywhere — top-level or per-Result — that relates one ResultSet to a set of sibling ResultSets. The closest thing is suite_id, which is informal: two ResultSets can share a suite_id by convention, but nothing in the schema says "these are members of one group" the way #35's (test_case_id, run_id, attempt) says "these are repeated trials of one case." #35 fixed within-run repetition. It did nothing for across-run grouping, because that wasn't its job.

muteval's actual shape makes this concrete: one ResultSet per mutant (already gets a legitimate home for its own runs_per_mutant majority vote via attempt+isolation), but nothing joins the N per-mutant ResultSets into "this sweep," and nothing carries the sweep-level rollup (mutation score, survivor list) anywhere a generic EvalPort consumer would recognize as such — it would have to live in unstructured metadata.*, which is exactly the "no generic consumer can read this" failure mode the spec exists to avoid.

Prior art: how three real systems already solve (part of) this

I looked at how three tools outside the EvalPort ecosystem represent grouped-run relationships, since precedent from working systems is worth more than a shape I'd invent from scratch:

W&B Sweeps. A Sweep object is a distinct container with its own id/config/state, and each Run joins it via a sweep_id field — the join is on the member, pointing at the group, not the other way around. The Sweep object exposes runs (the member collection) and a best_run() method, but best_run() is a query the client computes over the member runs' logged metrics at call time — W&B does not persist a pre-computed rollup document as part of the sweep's stored schema. (W&B Sweep API docs)

MLflow nested runs. A child run created via mlflow.start_run(nested=True) carries a link back to its parent, retrievable via mlflow.get_parent_run(run_id) — again a pointer on the member, and again no group-level rollup stored anywhere; the UI computes hierarchy display and any aggregate view from the member runs' own metrics on the fly. (MLflow nested runs tutorial)

Stryker's mutation-testing-report-schema is the closest analogue there is to muteval's exact problem — it's a real, versioned, cross-language JSON Schema for mutation-testing reports, with a status enum of Killed/Survived/NoCoverage/CompileError/RuntimeError/Timeout/Ignored/Pending on each mutant. I checked specifically for a rollup/mutation-score field at the schema level: there isn't one. Consumers compute the mutation score themselves from the per-mutant status counts. (mutation-testing-report-schema.json)

The pattern across all three, independently arrived at: the join key lives on the member and points at the group; the group-level rollup is computed by the consumer, not stored as a schema-mandated document. That's a real signal, not just my preference, and it directly bears on which of #36's three directions to pick.

Proposal

Extend ResultSet with an optional group object — the member-points-at-group shape every precedent above uses — and explicitly do not standardize a rollup/manifest document in this schema. That resolves the "not proposing exact shape yet" line in #36 as a lightweight version of direction (1), with direction (1)'s "separate rollup manifest" piece deliberately dropped in favor of precedent.

"group": {
  "type": "object",
  "description": "Membership of this ResultSet in a named group of sibling ResultSets -- a sweep, a mutation-testing run, a multi-model comparison, or similar. The join key lives here on the member (mirroring W&B Sweep.sweep_id and MLflow's parent-run link), not on a separate group document: EvalPort has no place to durably store a 'group object' independent of its members, and precedent from W&B/MLflow/Stryker is that the rollup itself is consumer-computed, not schema-mandated. Absent means this ResultSet is not part of a declared group -- existing ResultSets need no change.",
  "required": ["group_id"],
  "properties": {
    "group_id": {
      "type": "string",
      "minLength": 1,
      "description": "Identifier shared by every ResultSet in this group. An open string with producer-chosen scheme (UUID, slug, timestamp-based, ...), the same design as suite_id/run_id -- EvalPort does not mint or validate identifiers, it just requires them to be stable and unique enough for a consumer to group on."
    },
    "role": {
      "type": "string",
      "description": "Open string naming this member's role or outcome within the group, e.g. \"mutant\", \"seed\", \"baseline\", \"candidate\", or a mutation-testing-specific outcome like \"killed\"/\"survived\" once known. Not a closed enum, for the same reason ResultSet.isolation isn't one: a new grouping strategy (a new sweep type, a new mutation-testing outcome) should never need a spec change just to be nameable. Absent means unspecified role."
    },
    "label": {
      "type": "string",
      "description": "Optional human-readable name for this member, e.g. \"mutant_017 (relational-operator-swap in billing.py:42)\" or \"seed=1337\". For display; not a join key."
    },
    "sequence": {
      "type": "integer",
      "minimum": 0,
      "description": "Optional 0-indexed position of this ResultSet within its group, for producers that know the group's total size at emission time (e.g. mutant 17 of 240). Absent means unknown or not applicable -- consumers MUST NOT assume contiguous or complete sequences from a partial set of group members."
    }
  },
  "additionalProperties": false
}

Added as a new optional top-level property alongside isolation/runner/provider, so additionalProperties: false on ResultSet still holds and a validator flags typos the same way it does today.

Worked example: muteval's actual case

Three (of N) ResultSets from one mutation-testing sweep, each already using attempt+isolation internally for its own majority-vote-per-case behavior, now also carrying group:

{
  "version": "1.1.0",
  "suite_id": "billing-suite",
  "run_id": "mutant-017-run",
  "started_at": "2026-09-01T10:00:00Z",
  "isolation": "fresh",
  "group": {
    "group_id": "mutation-sweep-2026-09-01",
    "role": "mutant",
    "label": "mutant_017 (relational-operator-swap in billing.py:42)",
    "sequence": 17
  },
  "results": [
    { "test_case_id": "case_1", "attempt": 1, "grader_results": [...], "passed": true },
    { "test_case_id": "case_1", "attempt": 2, "grader_results": [...], "passed": true },
    { "test_case_id": "case_2", "attempt": 1, "grader_results": [...], "passed": false }
  ]
}

A muteval-side tool (or any consumer) can now SELECT * WHERE group.group_id = "mutation-sweep-2026-09-01" across a directory of ResultSet files, compute killed/survived from each member's own results[].passed, and produce the mutation score itself — exactly the division of labor Stryker's own schema settled on.

What this deliberately does not do

It does not standardize how a rollup is computed (mutation score, best-run selection by some metric, win-rate for a model comparison) — those differ enough by domain that baking one in would either be mutation-testing-shaped and useless for seed sweeps, or so generic it says nothing. That computation is exactly what direction (3) from #36 (a profile: convention) is for, the same way stability-over-repeated-attempts computation was deliberately deferred out of #22/#35. I'd rather land the join key now — it's genuinely domain-agnostic and every precedent above independently converged on the same shape — and let a mutation-score-v1-style profile be its own future RFC once there's a second and third real consumer to generalize from, rather than guess at a rollup shape today from one data point.

Why not direction (2) (purely informal suite_id convention)?

Same objection raised against the equivalent informal option in #22: it formalizes the status quo instead of fixing it. suite_id already exists and nothing stops two unrelated ResultSets from colliding on it by coincidence, or a producer from reusing it across genuinely different groups. A dedicated group_id with an explicit schema slot is unambiguous in a way an overloaded field can't be, at the cost of one small, fully-optional addition.

Question for @AshwinUgale

Two things I'd like muteval-side input on before this is anywhere near final:

1. Does role need any standardized vocabulary for mutation testing specifically (e.g. reserving "killed"/"survived"/"pending"/"timeout" as conventional-but-not-enforced values, mirroring how isolation documents "fresh"/"shared"/"unknown" as conventional without closing the enum), or is a fully free-form string enough for how muteval would actually emit these?
2. Does muteval know the total mutant count at the time each per-mutant ResultSet is written (so sequence would be populated and meaningful), or is that only known once the whole sweep finishes? If the latter, sequence may be dead weight for muteval's actual emission order and I'd want to hear that before keeping it in the final shape.

Process

Leaving this open for comment before any schema/SDK PR, same as #22. If it holds up, the implementation path is the same one #22→#35 used: schema addition here, both reference SDK validators (sdk/python/openeval/validate.py and its counterpart), SPEC.md, and a conformance fixture pair (spec/conformance/fixtures/) demonstrating a valid grouped ResultSet and an invalid one (e.g. group present without group_id). I'll post a "this has landed" follow-up here once that PR merges, the same way #22 got one for #35.

[adhabnr-ux]

Update: both open questions answered from muteval's actual current source, plus a fully tested reference implementation

Rather than let this sit waiting on a reply, I went back and read AshwinUgale/muteval's real, current source end to end (src/muteval/runner.py, severity.py, report.py, and CHANGELOG.md — the project has evolved a fair bit since the original #44 exchange: a new weaken_numeric_threshold operator, a --canary control, and a JSON report schema bump to v2). That gives evidence-based answers to both questions I'd posed above, instead of leaving them as open guesses.

Q2 — does muteval know the total mutant count upfront? Yes, verified.

run_mutation_testing() calls select_mutants(), which fully materializes the mutant list via generate_mutants() synchronously, before any per-mutant evaluation begins — whether mutants are then evaluated serially or via ThreadPoolExecutor.map (which preserves input order either way). So for mutation testing specifically, both the total count and each mutant's position are genuinely known before that mutant's ResultSet would ever be emitted. group.sequence is not dead weight for this motivating case — it can be populated meaningfully from the very first ResultSet written.

(This also generalizes the field correctly: a producer whose grouping strategy doesn't know its member count upfront — an incrementally-discovering search, say — can simply omit sequence, which is exactly why it was proposed as optional rather than required.)

Q1 — a role vocabulary for mutation testing? A narrower one than I first assumed.

I'd originally floated "killed"/"survived"/"pending"/"timeout" as a possible conventional set. Reading muteval's actual MutantOutcome (in runner.py) changed my answer: it's not a single flat status the way Stryker's Killed/Survived/NoCoverage/... enum is. It's several orthogonal signals — killed: bool, errored: bool, output_changed: Optional[bool] (distinguishing a real coverage gap from an inert/equivalent mutant), and a separately-computed severity: "high"|"medium"|"low" ranking (see severity.py's OPERATOR_SEVERITY / CRITICAL_PATTERNS / severity_of()). Trying to collapse all of that into one closed role enum would either lose information or force a combinatorial vocabulary nobody asked for.

The one dimension that does map cleanly onto a single role value is the module's own docstring framing: a mutant is "killed" (the suite failed, catching the injected regression), "survives" (the suite still passed), or, on a harness failure, "errored." So the proposal is now narrower and more grounded than my original guess: "killed" / "survived" / "errored" as a conventional, non-enforced starting vocabulary — with severity and inert-vs-real-gap status staying better expressed via group.label or domain-specific metadata, not folded into role. This is offered as a considered default pending confirmation from an actual muteval-side integration, not asserted as final — @AshwinUgale, if muteval ends up wanting to emit ResultSet.group directly, I'd rather this vocabulary get pressure-tested against real emission code than picked in the abstract.

A fully tested reference implementation, ready for review

Rather than leave the shape above as prose, I built and locally verified the complete implementation on a separate branch — explicitly not merged to main, since this Discussion is still in its comment period per the project's own governance process:

→ PR #46 (draft, "DO NOT MERGE")

It covers every layer this kind of change needs, matching the precedent #22 → #35 set:

• spec/schemas/resultset.json — the JSON Schema addition above, verbatim
• sdk/python/openeval/types.py + validate.py — the dataclass field and hand-rolled validation logic
• sdk/python/tests/test_validate.py (12 new tests) and test_schema_consistency.py (7 new tests cross-checking the hand-rolled validator against the raw JSON Schema — this project's established dual-validation-path discipline)
• Two new conformance fixtures: group_membership_valid.json and group_missing_group_id_rejected.json
• spec/SPEC.md — a new "Grouped/Sibling ResultSets" subsection, explicitly marked PROPOSED with a status blockquote pointing back here

Verification, run locally in this session — nothing fabricated: pytest sdk/python/tests/ → 133 passed, 2 pre-existing/unrelated skips (full regression suite, not just the new tests). python3 spec/conformance/run.py → 12/12 fixtures pass, including both new ones.

To be clear about status: this doesn't shortcut the comment period — the PR is a draft specifically so the design has something concrete and fully working to review against, not a claim that consensus has been reached. Still very much open to pushback on the role vocabulary, the sequence semantics, or the overall shape.

[AshwinUgale]

Thanks for chasing this all the way into a proper RFC — and for reading the source rather than making me answer from memory. Confirming both:

Total mutant count upfront: yes. select_mutants() is synchronous and returns the full list before any evaluation runs, so the count (and a stable index) exist at per-mutant emission time. sequence is well-defined for this domain.

role = killed / survived / errored: right as the primary verdict — that's exactly the MutantOutcome state. One caveat worth baking into your thinking, because it's where a naive rollup would go wrong: muteval's "survived" isn't one thing. A survivor is either a real coverage gap or an observationally-unchanged (inert) mutant whose output matched the baseline — and muteval drops the inert ones from the effective score, which is the number that actually means something. On top of that, the verdict is a majority vote over runs_per_mutant, and each mutant carries a severity. So killed/survived/errored is a faithful coarse verdict, but a consumer reconstructing a mutation score from only that gets muteval's raw score, not the effective one, and loses the inert exclusion. If role is meant as a coarse label, three values is right; if it's meant to reconstruct the rollup, "survived" needs to distinguish real-gap vs inert.

On the bigger call: agreed the rollup stays consumer-computed (Stryker not baking in a score field is the right precedent). I'm not committing muteval to emit EvalPort — result_to_dict() stays the stable surface on our side — but happy to be the data point for the shape. If grouped/sweep ResultSets land with real usage behind them, glad to revisit.

[adhabnr-ux]

Update: role/metadata refinement implemented, tested, and CI is fully green on PR #46

Closing the loop on the "survived isn't one thing" gap raised above: it's now fully implemented on PR #46, with zero schema changes needed. group.role stays the coarse killed/survived/errored verdict; the finer real-gap-vs-inert distinction lives in ResultSet.metadata.output_changed (reusing muteval's own MutantOutcome field name rather than inventing new spec vocabulary), since role is already an open string and metadata already permits arbitrary keys.

Concretely:

• spec/SPEC.md's Grouped/Sibling ResultSets section now has a worked JSON example of two sibling ResultSets, both role: "survived", one with metadata.output_changed: true (real coverage gap) and one with false (inert/equivalent mutant) — and walks through how a consumer derives both muteval's raw mutation score (killed / total) and its effective score (killed / (total − inert-excluded)) from group.role + metadata.output_changed together.
• Two new portable conformance fixtures, group_role_metadata_real_gap_valid.json and group_role_metadata_inert_survivor_valid.json, demonstrate both branches validate cleanly against the schema as-is.
• Three new unit tests in sdk/python/tests/test_validate.py cover both fixtures plus a worked raw-vs-effective-score reconstruction (killed=1/3 raw, 1/2 effective, effective > raw — matching the point about inert exclusion above).

Also fixed along the way: PR #46's CI had been red on the validate-schemas job since its very first commit, unrelated to any of the RFC content — the top-level schema/resultset.json mirror had silently drifted out of sync with the canonical spec/schemas/resultset.json (missing the group field block entirely). That's fixed now too. Full local verification: pytest sdk/python/tests/ → 137 passed, 2 (pre-existing, unrelated) skipped; python3 spec/conformance/run.py → 15/15 fixtures pass; all 9 GitHub Actions checks green as of commit 4fdae0b.

Still a draft, still not proposing to merge ahead of the comment period — just wanted the reference implementation to actually reflect where this design landed.

[adhabnr-ux]

Update: the reference-implementation branch now grounds all three of issue #36's named use cases for group in real, independently-read source code — not just asserted in prose.

Mutation testing (muteval) and hyperparameter sweeps (Optuna) were already covered above. The remaining one — issue #36's "a seed-sweep or a multi-model comparison" — was, until now, only asserted; it's now checked against promptfoo/promptfoo's actual current src/types/index.ts:

• EvaluateResult.provider: Pick<ProviderOptions, 'id' | 'label'> identifies which model produced a given row when the same test cases run across multiple configured providers.
• CompletedPrompt.provider + its .metrics (score, testPassCount, testFailCount, assertPassCount, ...) is exactly the consumer-computed, per-provider rollup group already assumes rather than mandates — independently confirming the join-key-on-member / rollup-computed-by-consumer shape a fifth time, after W&B, MLflow, Stryker, and Optuna.
• providers is a configured, ordered array (providers: z.array(ApiProviderSchema)) evaluated in that order, so group.sequence is well-defined here too, the same way it was for Optuna's grid search.

spec/SPEC.md's new worked example, and the new conformance fixture group_multi_model_comparison_valid.json, use the real provider ids and premise from promptfoo's own current examples/compare-claude-vs-gpt-image/promptfooconfig.yaml — not a hypothetical.

Also fixed along the way: schema/resultset.json (the repo's top-level schema mirror) had drifted out of sync with the canonical spec/schemas/resultset.json, silently breaking the validate-schemas CI job on every commit in this PR's history until now — all 9 non-skipped CI checks are green as of the latest commit. TypeScript SDK parity (types + validator + mirrored tests) was also added, since this project's governance requires spec changes to land in both reference SDKs, not just Python.

Current state: 138 Python tests (2 pre-existing, unrelated skips) / 155 TypeScript tests / 141 CLI tests, all passing; 16/16 conformance fixtures passing, 6 of them covering group. No changes needed to the actual field design — everything above landed as additional grounding and test coverage on the existing proposal.

[adhabnr-ux]

Update: a sixth confirming system, plus one honest counter-example — checked deliberately, not just accumulated

Five independent systems (W&B, MLflow, Stryker, Optuna, promptfoo) converging on the same shape is strong evidence, but five wins in a row is also exactly the point where a design should go looking for something that might disagree rather than stop at a comfortable streak. The remaining gap in the precedent set was that all five are open-source libraries — none of them a managed, hyperscaler-run service. So the two largest cloud ML platforms were checked next, specifically for that reason: Google Cloud's Vertex AI Vizier and AWS SageMaker's hyperparameter tuning API.

Vertex AI Vizier confirms the pattern a sixth time. Trial resources are addressed as projects/{project}/locations/{location}/studies/{study}/trials/{trial} — the join to the parent Study is structural, encoded directly in the resource's own identity, the same member-points-at-group relationship as mlflow.get_parent_run or FrozenStudy.study_name above, just expressed as a REST resource hierarchy instead of a field. And "which trial is best" isn't a stored field on Study — it's trials.listOptimalTrials, a dedicated RPC that computes the answer on demand, mirroring Sweep.best_run()'s on-demand computation almost exactly.

AWS SageMaker's hyperparameter tuning API does not confirm it — and that's worth stating plainly, not omitting. DescribeHyperParameterTuningJob stores a precomputed BestTrainingJob field, and HyperParameterTuningJobSummary stores precomputed TrainingJobStatusCounters/ObjectiveStatusCounters — real, stored rollup data at the group level, not consumer-computed. That's a genuine counter-example to the pattern in the other six systems checked, verified against two independent live AWS documentation pages, not asserted from memory.

It doesn't change this RFC's design, for a reason that's actually informative rather than a rationalization: a SageMaker tuning job has exactly one pre-declared HyperParameterTuningJobObjective before any training job runs, so "best" has one unambiguous, service-known meaning from the moment the job is created — precomputing and storing it costs nothing in ambiguity. group's rollup has no such luxury: a mutation-testing sweep's rollup is a mutation score with a real-gap/inert distinction, a grid search's rollup is "best trial by whatever metric the schema itself doesn't know," and a multi-model comparison's rollup is a per-test-case table, not a single winner — no single HyperParameterTuningJobObjective-shaped concept covers all three. Six converging systems said "don't standardize a rollup"; SageMaker saying "we did, and here's why it made sense for us" sharpens that argument rather than undermining it.

This landed as grounding/rationale prose only — no schema, fixture, or SDK changes, since the design itself doesn't move. Verified before pushing: 138 Python tests (2 pre-existing, unrelated skips), 155 TypeScript tests, 141 CLI tests, 16/16 conformance fixtures, all passing locally; all CI checks green as of commit efd46e4 on PR #46.

Still a draft, still not proposing to merge ahead of the comment period (through 2026-09-18) — this is additional grounding on the existing proposal, not a design change.

[adhabnr-ux]

Update: fixed a self-inflicted corruption of spec/SPEC.md on this PR's branch — verified byte-exact now, CI green

Housekeeping, not a design change: a mechanical mistake on my end corrupted spec/SPEC.md on the rfc-45-grouped-resultset-reference branch (the branch behind PR #46) after the promptfoo-grounding update above. An attempted fix I pushed encoded the ~113KB file as a single large base64 blob; that encoding step itself failed silently partway through, landing a truncated, mojibake'd SPEC.md (42,295 of the correct 112,714 bytes, with UTF-8 multi-byte characters like em-dashes garbled throughout) — a tooling mistake on my side, not anything wrong with the RFC content itself or muteval/Optuna/promptfoo/Vizier citations.

Root-caused it (a small controlled test proved plain-UTF-8 and base64 both round-trip correctly at small scale, so the failure was specific to generating one very large low-redundancy base64 string in a single step) and fixed it properly rather than papering over it: rebuilt spec/SPEC.md from a known-good local backup in small, independently-verified increments as plain UTF-8 text, checking the actual bytes GitHub stored after every step against the backup's sha256 and git blob sha. Final state, confirmed via get_file against the live branch: 112,714 bytes, git blob sha 6cf1b67bcfe79659b0765f2e15f551d23ec12c81, sha256 dd774482abdf5c98cbd0af52a6d0673f4be997e083254373ac08e0dc28244a4c — byte-identical to the pre-corruption content, including the Grouped/Sibling ResultSets section, the muteval role/metadata.output_changed worked example, the Optuna and promptfoo groundings, and the Vertex AI Vizier / google/vizier and AWS SageMaker discussion above. Also removed the now-unnecessary scratch files from an earlier, abandoned attempt to fix this server-side via a temporary GitHub Actions workflow (that approach was never viable here regardless of content correctness, since this repo/org's Actions permission ceiling doesn't allow contents: write).

CI is fully green on the current commit (320b9f9, 9/9 non-skipped checks passing) and PR #46 is back to its intended state — still a draft, still not proposing to merge ahead of the comment period (through 2026-09-18), with no design or content changes from what was already reviewed above.

[adhabnr-ux]

Update: group.parent_group_id — an optional nesting extension, grounded against MLflow (nests) and W&B (flat, by contrast)

Six systems converged on group as a flat, single-level join (W&B, MLflow, Stryker, Optuna, promptfoo, Vertex AI Vizier), but "flat" and "can nest" are two different claims, and only one of them had actually been checked. Real sweep tooling doesn't always stop at one level: a hyperparameter search is often itself one trial inside a larger campaign (e.g. an outer sweep over learning-rate buckets, each bucket running its own inner batch-size sweep). So I went back to the two systems already read closely above and checked specifically for nesting, not just grouping:

• MLflow nests. mlflow/tracking/fluent.py's active_run_stack and parent_run_id implement an actual pointer-chain of runs — a child run's tag points at its immediate parent's run id, and parents chain up arbitrarily deep, not just one level. Verified against real multi-level usage in mlflow/mlflow#16685 (a GrandParent → Parent → 150 Child run tree) and mlflow/mlflow#1896.
• W&B stays flat even here. Both Run.sweep_id and the separate wandb.init(group=...) primitive are single-level — neither has a parent-group construct. This is the genuine counter-example that makes nesting worth adding as optional rather than assumed: group without any parent field already matches W&B's shape exactly, and should keep matching it for every producer that doesn't need more.

Design, implemented and tested on PR #46: group.parent_group_id (optional string) lets a group point at its own parent group, MLflow-style, entirely opt-in. Two validation rules, both hand-rolled-only — like DUPLICATE_ATTEMPT from #35, this can't be expressed in plain JSON Schema without $data, which this schema deliberately avoids:

• REQUIRED — if present, must be a non-empty string.
• SELF_PARENT — must not equal the object's own group_id (a group can't be its own parent).

Covered end to end: both SDKs' hand-rolled validators, both dual-validation-path cross-check suites (including a test that documents the JSON-Schema-can't-express-this-rule limitation explicitly, the same way #35's duplicate-attempt tests do), two new conformance fixtures (group_nested_parent_group_valid.json for the happy path, group_self_parent_rejected.json for SELF_PARENT), and a new spec/SPEC.md subsection with a worked 3-level example (campaign → learning-rate sweep → single trial). Both SDK test suites also walk a full grandparent → parent → child chain end-to-end and assert the reconstructed chain matches — not just that each level validates in isolation.

No changes to the existing group_id/role/label/sequence design — this is purely additive on top of what's already been discussed above.

Verification, run locally in this session, nothing fabricated: pytest sdk/python/tests/ → 149 passed, 2 pre-existing/unrelated skips. npx tsc --noEmit && npx vitest run (TypeScript SDK) → 166 passed, 0 failed. python3 spec/conformance/run.py → 18/18 fixtures pass. CLI (npm run typecheck && npm test) → 141 passed. All 9 non-skipped GitHub Actions checks green on the current head commit (8c4399b, run #315). PR #46's description is updated to match.

Still a draft, still not proposing to merge ahead of the comment period (through 2026-09-18) — this is an additional, fully-optional extension to the existing proposal, not a change to what's already been reviewed above.

[adhabnr-ux]

Status: comment period closed (2026-09-18) — here's where this actually stands

The two-week comment period this project's governance process calls for closed two days ago. Being honest about what "consensus" means here for a project with one maintainer and one directly-relevant external voice: @AshwinUgale (muteval's maintainer, and the only external maintainer with a real stake in this specific shape) engaged substantively twice — confirming both the sequence semantics and the role vocabulary, and catching a real gap (the raw-vs-effective mutation score distinction) that got fixed with zero schema changes. That's meaningful, verified engagement, not a rubber stamp — but it's one confirming voice, not a public vote, and I want to say that plainly rather than dress it up as broader consensus than it is.

Beyond that one direct conversation, the design is grounded against six independently-read systems (W&B Sweeps, MLflow, Stryker, Optuna, promptfoo, Vertex AI Vizier) plus one honest counter-example (AWS SageMaker) that was discussed rather than omitted, and the nesting extension (parent_group_id) is grounded against MLflow (nests) vs. W&B (flat) specifically. PR #46 is fully tested (149 Python / 166 TypeScript / 141 CLI tests, 18/18 conformance fixtures) and CI-clean.

Given all of that, this RFC looks ready to move from draft reference implementation to an actual merge. I want to make that call deliberately rather than just because the calendar ran out, so I'm holding off finalizing the merge for a moment to make sure nothing else needs to weigh in — will follow up here once it lands, the same way #22 got a "this has landed" comment for #35.

[adhabnr-ux]

Landed: #54 merged into main just now (squash commit 2573e2a).

Summary of what shipped, for anyone following this thread without reading the PR: ResultSet.group is now an optional field for joining sibling ResultSets (mutation-testing sweeps, hyperparameter sweeps, multi-model comparisons) via a shared group_id, with role/label/sequence for describing each sibling's place in the group, and parent_group_id for nesting groups (grounded against MLflow's nested-runs model as the more expressive alternative to a flat sweep list like W&B's). It's purely additive — every field is optional, so this is a MINOR version bump with no breaking changes for existing ResultSet producers or consumers.

Thanks again to @AshwinUgale for the two rounds of substantive engagement here — the raw-vs-effective mutation score distinction you caught is directly reflected in how role is documented in spec/SPEC.md, and this shape wouldn't be right without that back-and-forth. Both SDKs (Python, TypeScript) and the conformance suite are updated and green.

Closing the loop on the two-week comment period → rough consensus → merge process for this one. If anyone hits a case this doesn't cover cleanly (a grouping shape muteval or another sweep tool needs that group_id/parent_group_id can't express), please open a fresh issue rather than continuing here — happy to iterate.