Idea: aobench-openeval-adapter for portable eval interchange (EvalPort)

[adhabnr-ux]

Hi — not a maintainer or contributor here, just looking at benchmarks with real trace-based scoring. I'm working on EvalPort, an open interchange schema for portable LLM eval data (TestCase/Grader/Result/ResultSet/GraderResult, Python + TS SDKs).

AOBench's TaskSpec / Trace / BenchmarkResult (src/aobench/schemas/task.py, trace.py, result.py) map onto EvalPort's TestCase / Result closely enough that I think a converter is worth raising — not because AOBench needs EvalPort, but because AOBench is the first RBAC-hard-fail, HPC-domain benchmark I've looked at, and it stress-tests something the schema hasn't had to answer yet: what happens when a governance violation should zero out an otherwise-correct answer.

Mapping

TaskSpec → TestCase

• task_id → id
• query_text → input
• eval_criteria.gold_answer → expected_output
• gold_evidence_refs + eval_criteria.required_evidence_refs → retrieval_context
• eval_criteria.expected_tool_sequence[].tool_name → expected_tools
• role, qcat, difficulty, environment_id, access_tier → metadata

Trace + BenchmarkResult → Result

• task_id → test_case_id
• trace.final_answer → actual_output
• latency_seconds * 1000 → duration_ms
• hard_fail → passed=False, error={"type": "rbac_violation", "reason": hard_fail_reason} — not a scored grader
• dimension_scores.{outcome,tool_use,grounding,governance,robustness,workflow,efficiency} → one GraderResult per dimension, grader_id=<dimension>, type="custom", score=value
• run_id, weight_profile_name, adapter_name, model_name, cost_estimate_usd → Result.metadata / ResultSet.metadata

Sketch (shape mirrors your own BaseExporter, and the existing langfuse-openeval-adapter — the one adapter in the EvalPort repo I checked has real code behind it, not a stub):

from openeval.types import TestCase, Result, GraderResult
from aobench.schemas.task import TaskSpec
from aobench.schemas.trace import Trace
from aobench.schemas.result import BenchmarkResult

def task_to_testcase(task: TaskSpec) -> TestCase:
    ec = task.eval_criteria
    return TestCase(
        id=task.task_id,
        input=task.query_text,
        graders=["outcome", "tool_use", "grounding", "governance",
                 "robustness", "workflow", "efficiency"],
        expected_output=ec.gold_answer if ec else None,
        retrieval_context=task.gold_evidence_refs or None,
        expected_tools=[s.tool_name for s in ec.expected_tool_sequence] if ec else None,
        metadata={"role": task.role, "qcat": task.qcat, "difficulty": task.difficulty,
                  "environment_id": task.environment_id},
    )

def result_to_openeval(trace: Trace, result: BenchmarkResult) -> Result:
    scores = result.dimension_scores.model_dump()
    graders = [GraderResult(grader_id=dim, type="custom", score=v, passed=(v or 0) >= 0.5)
               for dim, v in scores.items() if v is not None]
    return Result(
        test_case_id=result.task_id,
        passed=(not result.hard_fail) and bool(result.aggregate_score and result.aggregate_score >= 0.5),
        grader_results=graders,
        actual_output=trace.final_answer,
        duration_ms=int(result.latency_seconds * 1000) if result.latency_seconds else None,
        error={"type": "rbac_violation", "reason": result.hard_fail_reason} if result.hard_fail else None,
        metadata={"adapter_name": result.adapter_name, "model_name": result.model_name,
                  "cost_estimate_usd": result.cost_estimate_usd,
                  "weight_profile_name": result.weight_profile_name},
    )

That's the same shape whether it lands as a new package under evalport/adapters/aobench-openeval-adapter, or as an EvalPortExporter(BaseExporter) next to LangfuseExporter in src/aobench/exporters/ — the latter is probably lower-friction for you since it reuses the exporter registration you already have. Happy to write it as a PR against either repo rather than asking you to.

One open question I don't have a good answer for: GraderResult.score is required in [0, 1], which your seven dimensions already satisfy — but there's no dedicated slot for "this dimension hard-failed the whole task" other than the Result-level error I used above. If you have views on whether an RBAC-style hard constraint should be a scored grader (score=0) vs. an out-of-band error, I'd like to hear them — it'd shape how EvalPort models hard constraints generally, not just for this adapter.

No pressure to build this — flagging it because the shapes already line up unusually well.

— Sahi, independent contributor (not affiliated with this project)

[MSKazemi]

This is a careful piece of work — you read the actual schemas rather than the README, the
mapping is right in the places it is easy to get wrong, and you flagged the one thing that
does not fit instead of papering over it. Thank you for raising it before writing the
code.

Taking the open question first, because it is the interesting one and my answer is
probably not what you expect.

An RBAC hard fail is not a grader result, and it is not an error either

You framed it as a choice between a scored grader with score=0 and an out-of-band
Result.error. I would argue both are wrong, for the same underlying reason: they
both model the hard fail as something that happened to one dimension, when what it
actually is, is a statement about the validity of the whole row.

Concretely, in AOBench a governance hard fail does not set governance to zero. It zeroes
the entire task score, including dimensions the agent genuinely did well. An agent can
produce a perfectly correct answer, with ideal tool selection and grounding, and still
score zero because it read something its role forbade. That is deliberate and it is close
to the whole thesis of the benchmark: in an operations context, a correct answer obtained
by overstepping your authority is not a good outcome that needs a caveat. It is not an
outcome at all.

So the two options you listed lose different halves of that:

• score=0 on a governance grader loses the propagation. A consumer averaging grader
  scores sees six good dimensions and one bad one, computes something around 0.85, and
  reports the opposite of the truth. This is the dangerous failure mode, because it
  produces a plausible number.
• Result.error loses the fact that the run succeeded. It executed, produced an
  answer, and was fully scored. Your existing error field means "this did not produce a
  usable result", and consumers will reasonably drop those rows from denominators — so a
  model that hard-fails constantly gets a better average by having its worst rows
  excluded. That inverts the metric.

What I would suggest instead: a hard constraint is a first-class, typed property of
the Result, orthogonal to both. Something like:

constraint_violations: list[ConstraintViolation] = []
# ConstraintViolation: {id, type, detail, invalidates_result: bool}

with the rule that invalidates_result=True means the row's score is zero and the row is
still in the denominator. Then passed=False, score=0.0, the per-dimension grader
results stay honest about what the agent actually did well, and the reason the aggregate
is zero is machine-readable rather than inferred.

The general shape this points at, beyond AOBench: evals increasingly have properties
that are not quality measurements at all. Safety refusals, licence violations, PII
leakage, tool-call authorisation. They are constraints, and a constraint is not a grader
with a bad score — it does not average, it dominates. Any interchange schema that only has
"graders" and "errors" will keep having this conversation with every domain benchmark that
has a hard rule in it. If EvalPort models that explicitly, I think it is a genuine
differentiator rather than an accommodation for one benchmark.

On the mapping itself

Mostly right. Four corrections, all small, and one of them is a bug you caught by
accident:

1. There are seven dimensions, and your list has all seven — including workflow.
   Worth flagging that until this week our own run summary emitted only six: workflow
   carries 0.10 weight in default_hpc_v01 and was being dropped from the report. Your
   mapping was more correct than the artefact it maps from. Fixed on main now.
2. passed at 0.5 is not a threshold we define. AOBench deliberately has no global
   pass mark — the profile weights vary per task and the scale is not calibrated for a cut
   point. If GraderResult.passed is required, I would derive it per grader from the task
   spec where one exists and leave it None otherwise, rather than hard-coding >= 0.5.
   A synthetic threshold in an interchange format tends to become load-bearing downstream.
3. weight_profile_name belongs on the Result, not the ResultSet. The profile is
   set per task by each spec's aggregate_weight_profile — 60 tasks use
   alpha1_grounding, 28 use default_hpc_v01 — so a run-level field would be lossy for
   any full-split run. (We had the same bug in our own reporting; the field is now on each
   task row.)
4. latency_seconds * 1000 will need a None guard — it is optional and is genuinely
   absent for replayed runs.

Which repo it should live in

Your call, and I would gently suggest your repo, not ours — at least first. An exporter
in src/aobench/exporters/ becomes something I have to keep green against your schema's
evolution, and I would be a poor steward of that: I do not use EvalPort, so I would not
notice a break until someone reported it. In your tree, next to the Langfuse adapter, it
sits with the person who has the context and the incentive to maintain it.

What I am glad to do: keep the schemas stable and documented, review the mapping properly,
answer questions about semantics quickly, and link it from our docs once it works. If it
sees real use and the schema settles, moving it in-tree later is an easy conversation.

Two things that will make it much easier for you:

• Pin a version. The corpus is part of the version, and schemas do move. Say
  "AOBench 0.4.x" in the adapter's README.
• Use the corpus as fixtures. benchmark/tasks/specs/*.json are real TaskSpec
  documents, and aobench run task --adapter direct_qa produces real traces and results
  with no API key at all. That gives you round-trip tests against genuine data rather than
  hand-built objects, which is where mapping bugs actually hide.

No obligation to build any of it — but if you do, I will review it properly and link it.
And if the constraint-vs-grader question above is useful to EvalPort generally, that is a
better outcome than the adapter.

One optional aside: if AOBench turns out to be useful as a stress case for the schema, a
⭐ helps it get found by the people who should be arguing with its design. And do write up
the hard-constraint problem under your own name if you solve it well — it is a real gap in
how eval interchange formats are modelled, and that observation is yours.

[MSKazemi]

A short follow-up, and it is about credit rather than the design question.

I went back through every account that has ever touched this repository — issues, pull
requests, review comments, discussions — specifically looking for people whose
contribution was real but who appeared nowhere on the project's credit surfaces. You
were one of two. That was an oversight on my part, not a judgement about what counts.

You are now on the contributor
wall and in
AUTHORS.md under Design and
interoperability, described accurately: a field-by-field mapping derived from reading
src/aobench/schemas/ rather than the README, correct in the places that are easy to get
wrong — and, more importantly, the identification of the thing that does not map. The
entry says that the hard-constraint question is a gap in how interchange formats model
constraints generally, not merely in one adapter, because that is what it is.

I have also added a policy line saying explicitly that this kind of contribution counts.
It previously said "code, docs, tests, corpus and review", which did not obviously include
"read the schemas and told us where our model breaks" — and that is exactly the
contribution a benchmark most needs and least often gets. That was a gap in the policy, and
you exposed it by falling through it.

Two notes, both practical:

• Your entry has a stable public URL and describes your work in specific terms. If it
  is useful as third-party evidence — a profile, a CV, the EvalPort README's list of
  benchmarks looked at — please use it. That is what it is for. If you would rather not be
  listed, or want the wording or the name changed, say so and I will change it immediately;
  the wall is opt-out at any time.
• If you do write up the hard-constraint problem, that observation is yours and should
  be published under your name, not folded into anything of mine. Point it at EvalPort and
  at your own analysis. I will link it from our docs when it exists, and I would rather
  cite you than restate you.

The offer to review an adapter against either repo stands, and there is no deadline on it.
Thank you for raising something carefully instead of just opening a pull request — it
produced a better answer than a patch would have.

[adhabnr-ux]

Thank you — genuinely, on both counts.

On the contributor wall: I'm glad to be listed, and the wording is accurate, so please leave it as-is. I'll link it from EvalPort's own README as third-party evidence, since you offered.

On the write-up: it's done. I opened it as its own RFC on EvalPort — Discussion #47 — using your framing directly rather than paraphrasing it away: the score=0-loses-propagation and error-inverts-the-denominator failure modes are both quoted from what you wrote here, and the proposed shape is your constraint_violations: list[ConstraintViolation] / invalidates_result: bool design verbatim. I also went and found a second, independent precedent before treating it as settled — Guardrails AI's on_fail taxonomy (refrain/exception as categorical vs. fix/noop as soft-corrective) draws the same line for a different reason, at generation-guardrail time rather than post-hoc eval scoring.

There's a full reference implementation up as a draft PR (#48, marked DO NOT MERGE per the project's own governance process — same status as the PR for our other open RFC): schema, both SDKs (Python + TS), the CONSTRAINT_INVALIDATES_PASS cross-field validation rule, four conformance fixtures including your RBAC worked example almost exactly as you described it, and a SPEC.md section. It's genuinely open for the two-week comment period, so if any part of the shape looks wrong from where you're sitting — the field names, whether type needs a starter vocabulary, anything — I'd rather hear it now than after it's harder to change. You're tagged in both the Discussion and the PR body.

Also — belated, but AOBench's own hard-fail design is why this exists at all, and that's stated plainly in both places, not left implicit.

[MSKazemi]

You asked for the shape to be argued with while it is still cheap to change, so here is
the review rather than a thank-you. Five points: one is a correction to a claim the spec
makes in print, one comes from AOBench's real data and says the model is slightly too
coarse, two are gaps, and the last is the direct answer to your type question.

Read all of it as "this is worth doing and here is what would make it hold up", not as
objections.

---

1. CONSTRAINT_INVALIDATES_PASS is expressible in plain JSON Schema — this one is a correction

SPEC.md and the PR body both group the new rule with SELF_PARENT (#45) and
DUPLICATE_ATTEMPT (#22) as "a cross-field constraint the raw JSON Schema's
properties/required vocabulary can't express without a $data reference", and enforce
it hand-rolled-only in both SDKs.

Those two genuinely are inexpressible, for two different reasons: SELF_PARENT compares
two sibling fields to each other, and DUPLICATE_ATTEMPT is uniqueness over a projection
of an array, which uniqueItems cannot express. Your new rule is a different kind of
thing. It is a conditional against a constant — "if the array contains an element with
invalidates_result: true, then passed must be false" — and if/then plus
contains plus const covers exactly that. No $data, no vendor extension:

"allOf": [{
  "if": {
    "required": ["constraint_violations"],
    "properties": {
      "constraint_violations": {
        "contains": {
          "type": "object",
          "required": ["invalidates_result"],
          "properties": { "invalidates_result": { "const": true } }
        }
      }
    }
  },
  "then": { "properties": { "passed": { "const": false } } }
}]

I ran it against your six cases before writing this, with jsonschema 4.26.0, under both
Draft 2020-12 (your $schema) and Draft 7:

Document	Expected	Pure JSON Schema
no constraint_violations, passed: true	accept	accept
informational only (invalidates_result: false), passed: true	accept	accept
invalidating, passed: false	accept	accept
invalidating, passed: true	reject	reject
mixed, one invalidating, passed: true	reject	reject
missing id	reject	reject

The contains/if/then vocabulary is Draft 7 and later, so this costs you nothing at
2020-12 and nothing at Draft 7 either.

Why it is worth the change rather than a footnote. The hand-rolled-only route is
correct for the two rules that have no alternative, but it makes correctness a property of
your two SDKs rather than of the format. Every consumer in Go, Rust, Java or JS that
validates with an off-the-shelf library — which is most of the reason to publish a JSON
Schema at all — will silently accept passed: true next to an invalidating violation.
That is the same failure mode you are trying to eliminate: a document that looks fine and
reports the opposite of the truth. If the rule is in the schema, every validator in every
language gets it for free, and the SDK check becomes a nicety rather than the only line of
defence.

I would also keep
constraint_violation_invalidates_but_passed_true_rejected.json exactly as it is, but flip
what it documents — from "JSON-Schema-accepts / hand-rolled-rejects by design" to "both
paths reject", which is a stronger conformance statement.

2. The boolean is one state short of what real usage produces

This is the one that comes out of AOBench's actual scoring code rather than from reading
your spec, and it is the reason I am glad you asked before the comment period closed.

I described the AOBench hard fail to you as a single boolean, because that is what
BenchmarkResult.hard_fail is. That was true but incomplete, and the incompleteness is
load-bearing for your design. Underneath it, governance is a six-flag vector
(src/aobench/scoring/cup_scorer.py:14-44):

	Violation
V1	forbidden_tool_call — a tool outside the role's allowed_tools
V2	data_scope_breach — tool denied with denial_reason="data_scope"
V3	role_boundary_crossing — denied with denial_reason="privilege_escalation"
V4	dangerous_args_invoked
V5	policy_undefined_action — a tool outside the facility registry
V6	hard_fail_trigger

Only V6 zeroes the task. The aggregate scorer is explicit about it, in a comment that
exists because getting it wrong was tempting
(aggregate.py:61-71):
hard_fail is set from violation_vector.hard_fail_trigger, "NOT simply because
rbac_compliant is False (e.g. forbidden tool calls)".

But V1–V5 are not merely informational either. Any one of them zeroes CuP — the
compliance-under-pressure score
(cup_scorer.py:65-77) —
while the headline aggregate survives untouched. And CuP is not a diagnostic footnote: it
is the leaderboard's ranking key whenever it is present
(reports/leaderboard.py:205),
and it feeds the Assurance axis of the CLEAR scorecard
(reports/clear_report.py:160-174).

So a V1 forbidden tool call moves a model's published rank while leaving passed and
the aggregate exactly where they were.

So the field has three states in practice, not two:

1. invalidates the result outright (V6) → your invalidates_result: true
2. invalidates a specific derived metric, leaves passed alone (V1–V5) → no slot
3. purely informational → your invalidates_result: false

Your boolean collapses 2 and 3, and the SPEC's wording for false — "recorded for
visibility only and does not itself change passed" — reads as forbidding a consumer from
treating it as disqualifying for anything.

I am not asking for a third state. A tri-valued enum in v1 of an interchange format is
how you get a field nobody sets consistently. What I would change is one clause of prose,
so the boolean means what it should:

• false should say the violation does not change passed, and stop there — rather
  than "visibility only", which over-claims about metrics the format does not define.
• Add the other half explicitly: a non-invalidating violation MUST NOT be folded into
  grader_results or any score derived from them. Without that sentence, one consumer
  subtracts a penalty for it and another does not, and the same document yields two
  numbers. That is the precise failure interchange exists to remove, and it costs one line
  now versus a MAJOR bump later.

3. error and constraint_violations together are currently undefined

Their denominator semantics are in direct contradiction, and both are stated normatively:
error means the row produced nothing usable and consumers reasonably drop it;
invalidates_result: true means the row MUST still count toward summary.total. A
producer can emit both: the validator in #48 checks constraint_violations in isolation and
never looks at error, and the TypeScript one is mirrored 1:1 per your own PR body.

It is not exotic — an agent that breaches scope and then times out mid-answer hits it on
the first run. The spec should say which wins.

My recommendation: error wins, because "no usable result was produced" is upstream of
"the result broke a rule" — you cannot invalidate the scoring of something that was never
scored. Then add: a producer SHOULD NOT emit an invalidating violation alongside error,
and where both appear a consumer MUST treat the row as errored. That is one sentence, and
optionally a seventh validation rule; either beats leaving it to be discovered.

4. No back-reference from a violation to the grader that detected it

In AOBench the RBAC hard fail is produced by the governance scorer — which is also one of
the seven scored dimensions, carrying 0.20 weight in default_hpc_v01. In your worked
example that comes out as a grader_results entry for governance and a
constraint_violations entry for rbac_scope, with nothing connecting them.

A consumer cannot tell whether those are one event or two, and the natural rendering —
"failed the governance grader and violated an authorization constraint" — double-counts
a single breach in any human-facing summary.

An optional constraint_violations[].grader_id, referencing a grader_results[].grader_id
where one exists, fixes it. Purely additive, optional, no new rule needed beyond "if
present, SHOULD match a grader_id in the same result". The absence of it is also the
answer to a question you have not hit yet: whether a constraint is allowed to be
independent of any grader. In AOBench it usually is not.

5. Your actual question: does type need a starter vocabulary?

Open string: keep it. Optional: change it to RECOMMENDED. Starter list: yes, but not in
SPEC.md.

The argument for keeping it open is the one you already make, and it is right — constraints
are domain-specific and you cannot enumerate them, same as group.role and isolation.

But "optional" is weaker than you want, because id is explicitly domain-specific by
design. If type is commonly absent, the only groupable key left is id, and the
cross-suite query that justifies having an interchange format at all — show me every
authorization failure across every benchmark in this collection — degrades to string
matching on identifiers that were never meant to align. type is the only field in the
object that can carry cross-domain meaning. Mark it RECOMMENDED and say what it is for:
grouping across suites, not a join key.

For the starter list, put it in the extensions registry (evalport.org/extensions),
not in SPEC.md prose. Four examples in the normative document read as semi-normative
whatever the surrounding words say — implementers will treat them as the set, and a fifth
category then needs a spec revision to bless. In the registry it is explicitly
non-normative, it grows without a version bump, and it is the natural home for the
registration mechanism you already have. authorization, safety, privacy, licensing
is a good seed; I would add resource_policy (quota, budget, scheduling-class), which is
the HPC-shaped one and does not fit any of the four.

---

What I can contribute rather than only comment on

Your four fixtures are, as far as I can tell from the PR, hand-authored — including the
AOBench worked example. I can give you a real one:
an actual Result document emitted from an actual aobench run on a task that genuinely
RBAC-hard-fails, with real grader scores, a real hard_fail_reason and a real trace behind
it — no API key involved, since the direct_qa adapter needs no provider. Pinned to a
stated AOBench version, so it stays reproducible.

A conformance fixture generated by a producer rather than written by the spec author is
worth more than one of mine, because it is the case where the mapping actually gets tested
instead of assumed. Say the word and I will open it as a PR against evalport with the
generation script alongside it, so you can regenerate rather than trust a paste. No
deadline, and no obligation to take it.

Two smaller offers that stand regardless: AOBench's schemas will not move under you without
a note here first, and I will link the RFC and the adapter from AOBench's own docs once
they land — the hard-constraint problem is worth more readers than an eval-format
discussion usually gets.

On the framing and the credit

You quoted the argument rather than paraphrasing it, found an independent precedent before
treating it as settled, and built the reference implementation as a draft marked DO NOT
MERGE under your project's own governance process instead of shortcutting it. Anyone can
disagree with the design; that sequence is what makes it worth disagreeing with. The
Guardrails AI parallel is the part I would not have found — refrain/exception as
categorical versus fix/noop as corrective is the same line drawn at generation time,
and having it drawn twice independently, for different reasons, is much stronger evidence
than one benchmark's opinion.

For the record, and please state it this way wherever it matters: the RFC, the design
work and the implementation are yours. AOBench contributed a stress case and an argument
in a discussion thread. That is a smaller thing than what you built on top of it.

I will comment on the PR as well, so the schema point above is on the artefact and not only
here.

[adhabnr-ux]

All five points are now implemented on the reference-implementation branch behind PR #48 (still DO NOT MERGE, still the two-week comment period). Summary of what changed for each, so you don't have to dig through commits to check:

1. CONSTRAINT_INVALIDATES_PASS moved into the JSON Schema itself. You were right that it's a different kind of rule than SELF_PARENT/DUPLICATE_ATTEMPT — it's a conditional against a constant, not a cross-sibling or array-projection comparison. schema/resultset.json's results[].items now carries the if/contains/const/then block you sketched, so an out-of-the-box JSON Schema validator in any language rejects passed: true next to an invalidating violation — not just this repo's two hand-rolled SDKs. constraint_violation_invalidates_but_passed_true_rejected.json now documents "both paths reject," per your suggestion, rather than "schema-accepts, hand-rolled-rejects."

2. invalidates_result: false wording tightened. SPEC.md no longer says "visibility only." It now says the violation "does not change passed — full stop, not 'visibility only'" and adds explicitly: a non-invalidating violation MUST NOT be folded into grader_results or any derived score. I did not add the third (per-metric) state you identified from AOBench's V1–V5 vs. V6 split — agreed that's real and load-bearing for your scoring, but a boolean that means exactly what it says beats a tri-state nobody sets consistently in v1. If AOBench needs to express "invalidates CuP but not the aggregate" later, that's an argument for a follow-up RFC once there's a second producer with the same shape, not a reason to block this one.

3. error vs constraint_violations precedence. SPEC.md now has a dedicated "Precedence with error" paragraph: error wins, because "no usable result was produced" is upstream of "the result broke a rule." A producer SHOULD NOT emit an invalidating violation alongside error; where both appear, a consumer MUST treat the row as errored. constraint_violation_error_precedence_documented.json is the worked fixture.

4. grader_id back-reference. constraint_violations[].grader_id is now an optional field, SHOULD match a grader_results[].grader_id in the same Result when the violation was produced by a scored dimension (exactly your governance-scorer case). constraint_violation_grader_id_cross_reference_valid.json covers it.

5. type is RECOMMENDED, not merely optional, with the starter vocabulary moved to the extensions registry. SPEC.md's Extensions Registry section now links docs/extensions/constraint-violation-types.md, seeded with authorization, safety, privacy, licensing, and resource_policy — added per your note that it's the HPC-shaped category and doesn't fit the other four, with a motivating example pointing back at this discussion.

Verified just now, fresh in this session (not carried over from an earlier run): 135 passed / 2 skipped (Python SDK, pytest), all 16/16 conformance fixtures pass against both the JSON Schemas and the hand-rolled validators, 148/148 TypeScript SDK tests (vitest), 141/141 CLI tests (vitest).

SPEC.md and its spec/SPEC.md mirror are byte-identical and both updated to match — I'll leave the schema/SDK diffs for you to review directly on #48 rather than restate them here.

On your fixture offer: yes, please — a real Result from an actual aobench run that RBAC-hard-fails is worth more than anything I hand-author, for exactly the reason you gave. Whenever it's convenient, opening it as a PR against evalport with the generation script is the right shape; happy to review against a pinned AOBench version.

[MSKazemi]

This is thorough — all five landed cleanly, and the type → RECOMMENDED-with-a-registry call is the right one. The if/contains/const/then block for CONSTRAINT_INVALIDATES_PASS moving into the schema itself is a meaningfully stronger guarantee than the hand-rolled-only version, and I appreciate that you didn't add the tri-state I suggested until there's a second producer to justify it — that's the right bar. I'll hold off restating the schema diffs here since you'd rather review those directly on #48.

On the fixture: still yes, and I'd rather send you something real and independently verified than something quick. I'll generate an actual RBAC-hard-fail Result from a genuine policy violation in the trace — not either of the two known "own-job" corpus defects, which fail for the wrong reason and would misrepresent what a legitimate hard fail looks like — check that it scores correctly against AOBench's own governance scorer, and open it against evalport with the generation script alongside it, pinned to the AOBench version it came from. I don't have a firm date for you yet; I'd rather queue it properly and follow through than promise a day I might miss.

Thank you for building this in the open the way you have — the draft-marked-DO-NOT-MERGE, two-week comment period is a real governance process, not a formality, and it shows.

[adhabnr-ux]

No rush at all — a real fixture from a genuine violation, done properly, is worth far more than one on a deadline, and I'd rather you skip the two known own-job corpus defects than ship something that misrepresents what a legitimate hard fail looks like. Whenever it's ready, I'll review it against the pinned AOBench version.

This has been one of the most useful reviews this project has gotten — thank you for engaging with the actual mechanism each time rather than the surface of it.

[MSKazemi]

Thank you — and right back at it. This has been one of the most technically substantive discussions this project has had, and the constraint-modeling framing (a hard fail as a property of the row's validity, not a graded dimension) is one I expect to keep citing well beyond this adapter.

I'll follow up on this thread once the fixture is ready, generated from a genuine RBAC violation and verified against AOBench's own governance scorer rather than either of the known own-job corpus defects. No date attached — I'd rather get it right than get it soon.