Evaluation & trust

[AriAnthony]

This pillar exists for benchmarks and tasks that evaluate PMX agents, plus the QC, traceability, and human-in-the-loop practices that turn scores into trust.

[Brianc-metrumrg]

BENCHMARKS

I thought this was a nice example of a benchmark that incorporated "expert input" as the standard for measuring against, and for how to define a broader scope of tasks we may consider beyond simply generating R code. @AriAnthony , very interested in being part of this workstream.

https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf
HealthBench-Professional (1).pdf

[euibeom135-cell]

Claming this lane. The score into trust from the meeting stuck with me. In my QSP literature RAG eval work at my internship, the hardest part has been separating retrieval failure from generation failure, without that separation, you can't decide where to invest (extraction, retriever, or model). Coming at this from LLM Eval on clinical trial protocols https://ascpt.onlinelibrary.wiley.com/doi/full/10.1002/cpt.70096, currently extending to FIH dose selection.

Happy to contribute to the benchmarking and evaluation matrix.

[ofekron]

A useful evaluation matrix should separate model quality from operational trust. For each public or synthetic task, I’d record four layers:

1. Input/retrieval — source coverage, citation correctness, stale/conflicting evidence, and whether relevant evidence was missed.
2. Reasoning/output — task-specific expert rubric, uncertainty calibration, unsupported claims, and sensitivity to plausible input perturbations.
3. Action/verification — tools called, parameters, generated artifacts, deterministic checks, and whether a human accepted, edited, or rejected the result.
4. Operations — reproducibility across model/provider versions, latency/cost, failure recovery, and whether replay produces the same final evidence state.

The key comparison is not just score vs expert answer. Include no-context, correct-context, and adversarial/stale-context conditions so retrieval failure, generation failure, and negative transfer are distinguishable.

For human-in-the-loop evaluation, measure intervention quality: was the reviewer shown the evidence and uncertainty needed to catch the error, and was approval bound to the exact artifact later used? A generic “human reviewed” flag is too weak for traceability.

I maintain Better Agent (https://github.com/ofekron/better-agent), a local workspace for supervising persistent Claude, Codex, and Gemini sessions. Its inspectable events, approvals, files, and recovery paths inform this separation between output quality and operational evidence. It is source-available and free for non-commercial use; commercial use requires separate permission.

AI-assistance disclosure: this comment was drafted by Codex under the maintainer’s authorization and reviewed in Better Agent. No proprietary or patient-level data was used.

[AriAnthony]

First Run support and reports now live in this thread. Keeping this one for scoring and trust strategy.