Designing a Benchmark Module to Evaluate Pipeline Improvements

[marcvergees]

Context

Following #609, we agreed that the best way to settle the debate between Pipeline A (Canonical Incident Schema) and Pipeline B (Template-Driven Extraction) is through empirical evidence rather than assumptions. To do that, we need to build our own benchmarking infrastructure.

This discussion is dedicated to designing that benchmark module—what it should measure, how it should be structured, and what a realistic test dataset looks like.

---

What are we trying to measure?

To make a fair comparison, the benchmark should evaluate both pipelines across the same dimensions.

1. Extraction Accuracy

• Field-level accuracy: does the extracted value match the ground truth?
• We need a set of synthetic or anonymized incident narratives with known ground-truth field values.

2. Consistency

• Pipeline A: does the same fact appear consistently across every generated form?
• Pipeline B: when multiple templates are filled from the same narrative, do shared fields remain consistent?

3. Hallucination Rate

• Does the pipeline fabricate values for fields that are not present in the narrative?
• This is especially important for legal and emergency reporting fields (e.g. injury type, cause of fire, property damage).

4. Latency & Cost

• End-to-end execution time.
• Number of LLM calls per incident.
• Estimated token cost per incident.
• Cost as the number of generated forms increases.

5. Scalability

How does each pipeline behave when generating:

• 1 form
• 10 forms
• 100 forms

from the same incident narrative?

---

Proposed Benchmark Dataset

I think we should build a benchmark dataset consisting of:

• N synthetic incident narratives (short, medium, and long), each with a corresponding ground-truth JSON.
• K real-world templates (e.g. NFIRS, Massachusetts Fire Report, CAL FIRE ICS-209, NY Burn Injury) from the ones @chetanr25 shared.
• An expected extraction result for every narrative × template combination.

As a first version, I'd propose starting small:

• 10 narratives
• 4 templates
• 40 benchmark cases

---

Proposed Benchmark Structure

benchmark/
├── README.md
├── run.py
│
├── datasets/
│   ├── narratives/
│   ├── templates/
│   └── ground_truth/
│
├── pipelines/
│   ├── base.py
│   ├── pipeline_a.py
│   └── pipeline_b.py
│
├── evaluators/
│   ├── accuracy.py
│   ├── consistency.py
│   ├── hallucination.py
│   └── aggregate.py
│
├── reports/
│   ├── raw/
│   ├── summaries/
│   └── figures/
│
├── config/
│   └── benchmark.yaml
│
└── utils/
    ├── io.py
    ├── metrics.py
    └── logging.py

---

Open Questions

Ground Truth Format

Should the ground truth be:

• a flat field_name → value mapping?
• structured by module (following NERIS)?
• something else entirely?

---

Evaluation Metric

How should we score partially correct values?

Example:

• Ground truth: "Floor 3"
• Prediction: "3rd floor"

Possible approaches:

• Exact match
• Normalized/fuzzy matching (e.g. RapidFuzz)
• LLM-as-a-judge (Marc: I like this idea)

---

Benchmark Dataset

Who should create the benchmark narratives?

• Fully synthetic?
• Real anonymized reports?
• A combination of both? (Marc: I like this idea)

---

Hallucination Detection

Should the ground truth explicitly specify which fields are absent, allowing us to distinguish between:

• correctly returning null
• incorrectly hallucinating a value

---

Confidence Scores

Should each pipeline also return a confidence score per extracted field so we can compare confidence against actual accuracy?

---

Benchmark Execution

Should the benchmark run:

• manually via CLI?
• in CI for every PR?
• nightly?
• some combination of the above?

Marc: I feel like the best option might be CLI for development testing + CI to protect branches.

I was thinking to use this for the CI. Lmk how you feel guys.

---

Proposed Next Steps

1. Agree on the ground-truth schema.
2. Agree on the evaluation metrics.
3. Create the benchmark/ directory.
4. Write 3–5 synthetic narratives as an initial dataset.
5. Implement pipeline_a.py and pipeline_b.py using the existing services.
6. Run the first benchmark and publish the results.

---

I'd especially love to hear thoughts from @chetanr25, @abhishek-8081 and @vharkins1 on:

• Do these metrics cover everything you'd want to evaluate?
• Any concerns about the dataset format or ground-truth design?
• Would you be interested in helping build the initial benchmark dataset?

I think agreeing on how we measure success is the first step before investing time in implementing either pipeline.

I'd love to work on something like this because it gets closer my field of expertise and I'd be super happy to work with you guys as well, so we could divide the team and split tasks (API and benchmark) for now.

[marcvergees]

Hey everyone,

Quick update on this—I’ve gone ahead and implemented the benchmarking infrastructure based on our initial discussions. Here is how the final architecture shaped up, along with how it answers the open questions we had:

1. Dataset, Schemas & Ground Truth

• datasets/templates/ (ics_201.json, ics_202.json): To address the ground-truth format, we went with structured JSON mapping schemas that explicitly define expected field types. This provides a clean baseline to catch hallucinations and structural mismatches. I'm going to make it way bigger so we do have real testint of NERIS.

2. Evaluation Metrics & Scoring

• evaluators/accuracy.py: Instead of jumping straight to a heavy LLM-as-a-judge model, I opted for a hybrid deterministic approach for version one to keep it fast and cost-effective. It covers:
• Exact matching for boolean and numeric values.
• Token Jaccard-overlap similarity for fuzzy string matches (which perfectly handles minor text variations like "Floor 3" vs "3rd floor").
• Deep traversal matching to properly score lists and nested dictionary trees.

I'll implement the LLM-as-a-judge model very soon.

3. Execution & Testing Architecture

• runners/runner.py: Handles the underlying iteration logic across our dataset cases.
• test_benchmark.py: Instead of a custom standalone CLI, I integrated this directly into our testing framework via pytest. It dynamically detects our pipeline present on the current branch, runs the evaluation, and outputs the results to a structured benchmark_report.json.

4. CI/CD & Branch Comparison

• compare_benchmarks.py: A utility that ingests two separate metrics reports (e.g., from different branches or pipelines) and calculates a high-level performance delta.
• .github/workflows/benchmark.yml: This automates the process completely. When a PR is opened, the CI runs the benchmark on the feature branch, compares it against the target branch (main/development), and automatically posts a Markdown summary table directly into the PR comments.

The baseline infrastructure is now documented and live in the benchmark/ directory. You can run it locally with a standard pytest command.

Let's use this setup to start committing our sample narratives and running Pipeline A vs. Pipeline B so we can let the data do the talking! Would love to hear your thoughts on the workflow.

[chetanr25]

Looks cool! I don’t have much experience with benchmarking yet.

For the benchmark run method, I would personally prefer running the test CLI locally since I’m not sure how it would work with CI/CD, especially because the changes would be on different branches. Does the CI/CD workflow target both branches and run every time a PR is merged?

I need to look into a few benchmarking approaches before I can comment further.

[marcvergees]

@chetanr25 Exactly like you were pointing out.