Evaluating a Canonical Incident Schema vs. Template-Driven Extraction

[marcvergees]

During the API design discussions, we realized we're actually debating two fundamentally different extraction pipelines. Before we commit to one, I think it's worth having a dedicated architectural discussion rather than coupling it with the API implementation.

The two approaches

Pipeline A – Canonical Incident Schema

Narrative
    ↓
LLM
    ↓
Canonical Incident JSON
    ↓
Template Mapper
    ↓
PDF / JSON

The LLM extracts a single, general-purpose incident representation that is later mapped to different forms.

Pipeline B – Template-Driven Extraction (current approach)

Narrative
    ↓
Template / Field Groups
    ↓
LLM
    ↓
Field Values
    ↓
PDF

The template drives what is extracted, without requiring a universal incident schema.

My proposal

I think we should continue developing the API around Pipeline B, since that's the approach we've been designing so far, and avoid changing the API while we're still debating the underlying architecture.

In parallel, we can create a separate branch and implement Pipeline A as a prototype. Then we can compare both approaches on real reports and decide based on evidence instead of assumptions.

Changing the extraction pipeline is a major architectural decision, and I'd rather validate it experimentally before restructuring the API around it.

My concerns about Pipeline A

• Schema explosion. A canonical Incident JSON would eventually need to represent every possible field from every current and future form. It could easily become a large ontology that's difficult to maintain and might reduce the generality we're trying to achieve.
• Manual mapping. Every Incident JSON field still needs to be mapped to every PDF field, creating additional maintenance whenever new templates are added.
• Confidence scoring. We'd need a reliable way to compute field-level confidence and completeness, likely introducing additional validation or ML components.
• Overengineering. The architecture becomes significantly more complex, but it's not yet clear whether the improvement in extraction accuracy justifies that complexity.

Questions

• Has anyone benchmarked these two approaches on the same dataset?
• How much improvement does the canonical-schema approach actually provide in terms of accuracy and hallucination reduction?
• Does the added complexity pay off enough to justify changing the API and overall architecture?

I think this is an exciting direction to explore, but I'd prefer we validate it in isolation before making it the foundation of the project.

[chetanr25]

Thanks for writing this up. Before we go further, I want to clear up one thing, because I think the way it is framed as "schema vs no schema" is where our confusion is. The way I see it, both approaches need a schema. The only difference is what schema should contain and where that schema is applied.

Let me first explain what I actually proposed, because I think calling it a "canonical incident schema" makes it sound heavier than it is.

What I proposed

A firefighter describes the incident once, by voice or text. Convert that into text using whisper if voice input API is used. The LLM reads that once and pulls out every piece of information it can find. It writes those values into one agreed JSON shape, which is the incident contract (or what ever we want to call the schema as). Almost every field in it is optional, so it only fills what the input actually contains. There is no requirement to fill everything.

That extraction runs one time per incident. After that, filling forms needs no LLM at all. Each template we register already says, box by box, which field from the incident JSON goes into it and where on the page it sits. So filling a form is just a lookup and a draw. This is why it can fill any number of forms in seconds and supports bulk filling. The story is read once, the paperwork is generated many times from the same stored data.

You can see the JSON shape here: https://fireform-gsoc.chetanr25.in/incident-contract and the template registration flow where each box is bound to a field here: https://fireform-gsoc.chetanr25.in/create-form

The part I want to clarify

The concern about a canonical schema is fair on its own. But I don't think Approach B avoids the schema. It just moves it.

When you register a template in B, you still stand on each box and say "this box holds the address, this one holds the alarm time, this one holds the cause." That list of "what goes here" is a vocabulary of field names. That vocabulary is a schema. It might live inside one template instead of in a shared file, but it exists. If it did not, there would be nothing for the LLM to aim at and nothing to tell the filler which box gets which value.

So to me the real question is not "schema or no schema." It is "one shared schema that every template maps to." The shared one is what lets a single extraction feed many forms.

The reason a shared middle layer matters so much here is that no two forms name their fields the same way. These are all real forms we need to support, and they share almost no field names but need to insert same data:

• NFIRS complete reference: https://www.usfa.fema.gov/downloads/pdf/nfirs/nfirs_complete_reference_guide_2015.pdf
• Massachusetts sample fire report: https://www.mass.gov/doc/samplefirereportdoc/download
• CAL FIRE ICS-209: https://training.fema.gov/emiweb/is/icsresource/assets/ics%20forms/ics%20form%20209,%20incident%20status%20summary%20(v3).pdf
• New York Burn Injury Report: https://www.dhses.ny.gov/system/files/documents/2022/12/burn-injury-report_form_0.pdf
• UN SSIRS: https://unsmin.dss.un.org/DesktopModules/SSIRS_Incident_Details_V2/GetDocument.aspx?uri=QrW0SXoP2AI%3D

Same incident, five different field layouts and five different naming styles. A shared schema is the one neutral thing all of them can map to. Without it, the data extracted for one form does not transfer to the next.

Where the schema comes from, and what it is not

The schema is inspired by NERIS data framework: https://github.com/ulfsri/neris-framework. NERIS is the National Emergency Response Information System, built by UL's Fire Safety Research Institute with DHS S&T and USFA, and it is the official successor to NFIRS. Departments will be submitting to it going forward, so it is one of the most important forms we have to support. What I took from it is the idea of a modular schema: it is split into modules (incident, fire, structure, medical, hazard, rescue, and shared groups like location and weather) plus a separate set of value sets, which are the allowed choices for each field. There are 187 value sets in there today. That modular shape, plus mostly-optional fields and shared enums, is the same organizing idea our contract uses.

NERIS goes far deeper than we do in several places. This incident contract is flatter than that. We also carry whole sections NERIS core does not have, because other forms need them: wildland detail for CAL FIRE ICS-209, arson for NIBRS, EMS patient detail that maps toward NEMSIS, and periodic reporting for the monthly and quarterly rollups. And NERIS asks for things we do not model yet, like exposures, emerging hazards, and displacement cause.

This incident contract is inspired by NERIS and also added few other parameters in order to support most of the other forms as well. The contract does not grow without limit, because its job is to be the mapping target for a known set of real forms. Its size is bounded by those forms. And if we want reduce, then we can miss out few details which might support some other form in future. I still believe that everyone should review this incident contract and propose what's best for FireForm. If we remove the whole schema, then I am not sure how exactly we can proceed.

My read of Approach B, and where I might be wrong

The way I currently understand B, the template drives the extraction, so the LLM runs per form. If one incident needs six forms, that is six passes over the same narrative. That is slow when many forms are needed at once, and each pass can read the same story slightly differently, so the same fact can land differently across forms. It also makes exact placement harder, because there is no stored, agreed value to drop into a known box.

That is the part I want to confirm. Is that the B you have in mind, one extraction per form driven by the template? Or is there a version of B where the extraction runs once and gets reused across forms? Because if it runs once and is reused, then it is already producing some shared structure in the middle, which is the thing I am calling the schema. If that is the case then we may be closer to agreeing than it looks, and the debate is really about how formal that middle structure should be.

There is one more point from your list I want to ask about directly, the manual mapping one. You mentioned that every field would need to be mapped to every PDF field, and that this adds maintenance every time a new template is added. I read that a few times and I am genuinely a little confused about how you are picturing a template being added, so I want to check it against what I had in mind.

The way I was thinking about it is the flow in the POC: https://fireform-gsoc.chetanr25.in/create-form. The template registry service is not built yet, so we can absolutely update this if there is a better way. But in my head the flow is quite simple. You upload the new form, drag a box over each spot that needs filling, pick which data field goes in that box along with a few text settings, and save. That is the whole thing.

A template like this takes about 2 to 10 minutes to set up, depending on how many fields it has. And once it is saved, we do not have to keep maintaining it. The mapping stays with the template, and every incident after that is filled automatically by the later services with no extra work from us. So the mapping is a one-time step per form, done by the user in a simple editor, not something we hand-maintain in code. It also gives the user more control, because they place the boxes themselves instead of relying only on a PDF library to guess where each value should go.

So my honest question is, when you say manual mapping should be avoided, do you have a different way of adding templates in mind than this drag-and-drop step? If there is a way to skip even this one-time setup and still land each value in the right box, It would be great and I would love to listen the approach so that we all are in same page, plus we have to update API contracts.

On the concerns, briefly

On schema explosion, I covered most of this above. The shape is bounded by the real forms it has to map onto, and almost every field is optional. On the mapping work, my understanding is that the mapping happens once when a template is registered, through the visual editor, and not per incident. I may be missing something here, but it also seems like Approach B would need a similar box-to-field step of its own, so I would love to hear how you see that part working.

With all that said, I fully agree with building both on separate branches and comparing them before we confirm with our approach. I just want the B branch to be clear up front on how it handles two things without a shared shape (Incident contract JSON): reusing one extraction across many forms, and placing each value in the exact box. If we pin that down, the comparison will be fair and we will actually be measuring the same thing. This discussion is very important before working on two different pipeline approaches as this could save a lot of time, If we decide to go with it, there should be strong understanding about both the approaches before working on it.

[marcvergees]

My answers to al your concerns:

The way I currently understand B, the template drives the extraction, so the LLM runs per form. If one incident needs six forms, that is six passes over the same narrative. That is slow when many forms are needed at once, and each pass can read the same story slightly differently, so the same fact can land differently across forms. It also makes exact placement harder, because there is no stored, agreed value to drop into a known box.

I feel like both pipelines are valid. We can find arguments to deffend/refute each model. E.g. the same fact than having one shared schema is critical regarding hallucination. Imagine filling 100 PDFs and having a field in all of them, if the LLM hallucinates with that certain field, there might be problems and at the same time calling for each field might at the end of the day reduce the chances of hallucination. However, at the same time we're introducing repeatitive calls, unnecessary duplications and lots of more stuff.

That's why I feel like we should implement both, build our own benchmark technology to test the pipeline and then decide based on real results.

That is the part I want to confirm. Is that the B you have in mind, one extraction per form driven by the template? Or is there a version of B where the extraction runs once and gets reused across forms? Because if it runs once and is reused, then it is already producing some shared structure in the middle, which is the thing I am calling the schema. If that is the case then we may be closer to agreeing than it looks, and the debate is really about how formal that middle structure should be.

There's one extraction per template, so there's no common schema across forms. Each form would have a schema like that
{ "location of the incident": str, "victim's address": str, "victim's name": str, "victim's age": int }
As you might have seen in the /services folder, this schema is used to call the llm field by field.

There is one more point from your list I want to ask about directly, the manual mapping one. You mentioned that every field would need to be mapped to every PDF field, and that this adds maintenance every time a new template is added. I read that a few times and I am genuinely a little confused about how you are picturing a template being added, so I want to check it against what I had in mind.

Mapping is as easy as analyzing which are the fields. When you load the app and click on the "Create Template" tab, you'll see that fields are automatically recognized and you have the chance to rename them or add any kind of information. This process is basically creating this schema I've just pointed out. This schema is stored in the db as a dictionary in the templates table, so the user, when clicks on the "Fill Form" tab and runs the extraction for a certain given input and a template just concatenates the user input to the prompt and calls the LLM for each of this dictionary key, value pairs. Here is where I see the power of everything, we depend 100% on the intelligence of the LLM model, we will never have any restrictions about how many preagreed fields should have the schema, the LLM is intelligent enough to determine is the field already exists in the given input.

What I see based on the discussion we've been having:

1. General agreed schema vs particular schema: pipeline A can lack of generality but if we create a tool spefically for each emergency department I could be convinced that it could perform better based on what you were pointing out of 187 fields.
2. Field mapping: faster in pipeline A than B, though it needs setting up. It shouldn't be a problem because we could work on something like this, though this arguments works for both pipelines.
3. We agree that there's a need to create a benchmark to test pipeline changes/improvements, that's why I propose to start a new discussion here to debate how this new module should work. I wanna hear your opinions @chetanr25, @vharkins1 and @chetanr25.

[abhishek-8081]

Thanks @marcvergees , @chetanr25 - this is a genuinely useful thread, and reading it end to end helped me get clearer on where the real disagreement is. Sharing where I land.

On the architecture, I think A approach framing is the better one. Marc, your reply actually confirmed the key point: B runs one extraction per template, with no shared structure and no reuse across forms. That means an incident needing six forms is six passes over the same narrative, and the same fact can surface differently on each. So the slowness-and-consistency concern isn't speculative — it's how B behaves by design. And as Chetan put it, this isn't really "schema vs no schema": B has a schema too, just held inside each template rather than shared. The question is whether that structure is shared or scattered.

Where I lean, and why: on balance I lean toward A (extract once, reuse across forms), mainly for reasons that are hard to dispute:

• Fewer model calls. A is one extraction per incident regardless of form count; B is one per form. For bulk filling — one incident, many forms — that gap grows fast. On offline-first hardware running local Ollama, minimizing calls matters.
• Consistency by construction. A stores one value per fact, so every form reads the same value and they can't disagree. For official documents describing the same incident, that's a real property, not a nice-to-have.
• Reusable structured data. A leaves behind a clean incident record that's useful for analytics and audits later; B only leaves filled forms.

But I want to be honest about the one thing A has not proven, because it's the thing that actually decides this. A leans on a single extraction filling the whole contract in one pass. If that one pass hallucinates a field, the wrong value propagates to every form — whereas B's narrower per-field calls might contain that risk better. Marc, this is your core concern, and I don't think A can claim victory until it's tested. Fewer calls and better consistency don't matter if the single extraction is less accurate.

I'd just add one caveat: the benchmark settles accuracy and hallucination, which is exactly the open question above. It does not settle scaling — whether the shared schema stays maintainable depends on whether our form set is bounded (a known set of fire/EMS forms) or open-ended (any arbitrary form).

Short version: I lean A on architecture, speed, and consistency — but its accuracy under one-shot load is the real unknown, and that's what the benchmark should prove or break. Happy to help make that test fair on exactly this point.

[chetanr25]

Thanks, that cleared up Approach B for me @marcvergees .
No shared schema, each template carries its own field dictionary, and the LLM is called per field at fill time. I had that wrong earlier, so it helps.

Before we split into two branches, I want to lay both approaches side by side so we are all looking at the same picture.

Where the two approaches differ

The main difference sits in the extraction layer.

In Approach A, the firefighter describes the incident once, by voice or text. If it is voice, we transcribe it. The LLM reads that once and pulls out everything it can into the incident schema. After that we can drop the audio file and the transcript, because the extracted data in the schema is all we keep. So no matter what the input was or which forms we need at the end, the stored shape stays the same. That stored data becomes the single source of truth for the incident.

In Approach B, there is no separate extraction step. We keep the transcribed text, and when a form is needed we pass that text plus the form's field list to the LLM, which returns the values in that form's shape. So B can skip the extraction layer completely.

That is the real fork between the two. A extracts once and stores structured data. B keeps the text and extracts per form when a form is asked for.

The two approaches side by side

Here is how I picture the full flow for each, step by step.

Step	Approach A	Approach B
Input	Voice or text, once	Voice or text, once
Transcription	Transcribe voice to text	Transcribe voice to text
Extraction	LLM reads the text once and fills the incident schema	No separate extraction step
What we store	The structured incident data. Audio and transcript can be dropped	The transcript text, kept for later
When a form is needed	User picks the forms they want from a list	User picks one form to generate
Fill work	Look up each mapped field and draw it into its box. No LLM at fill time	Send transcript plus that form's field list to the LLM, get values field by field, then draw
Source of truth	The stored incident data, same for every form	The transcript, re-read by the LLM for each form

A few quick points on the flow:

• Both start the same way. They only split at the extraction step.
• A does the LLM work once, up front. B does it later, once per form.
• In A the transcript is throwaway after extraction. In B the transcript is the thing we keep.

And here is how I see the tradeoffs lining up.

Thing	Approach A	Approach B
LLM calls at fill time	None. Extraction already happened once	One pass per form, field by field
Filling many forms at once	Fast. Same stored data feeds all of them	Slower. Each form is its own LLM pass over the text
Filling a few forms	Carries the cost of the one extraction pass upfront	Likely faster. Straight from text to the one form
Same fact across forms	Comes from one stored value, so it is consistent everywhere	Extracted per form, so the same fact can land differently
Hallucination	If a stored field is wrong, it is wrong in every form that uses it. Can be corrected while manual review before generating all PDFs	Per field calls may catch more, but at the cost of repeated calls and duplication. Model can make repeated mistakes and that should be fixed everytime for every PDFs
Middle schema	One shared schema every template maps to	No shared schema. Each template holds its own field dictionary
Template setup	Map each box to a schema field path, once per template	Map each box to a field name, once per template
Prompt caching	Fits well, since forms and mappings are known ahead of time	Less room for it, since each form re-reads the text
Periodic reporting and querying type of PDFs	Easy. Data is already structured and stored	Harder and almost impossible. Data lives in free text and per-form outputs
Overall complexity	More upfront design for the schema and mapping	Simpler. No shared schema to agree on or maintain
Adding new fields	Bounded by the schema, so a truly new field needs a schema change	Fully open. The LLM handles whatever the field list asks for

A few quick points on the tradeoffs:

• A pays its cost once, so it scales well when many forms are needed. B pays per form.
• A gives one reviewed value that every form reuses, so the same fact stays consistent and gets checked once.
• B is simpler to stand up and flexible on fields, since there is no shared schema to agree on.
• Reporting and querying is easy in A and very hard in B.

Flow changes and which contract layers each one touches

The two approaches also change which parts of the API contract we build. This is roughly how I see it.

Contract layer	Approach A	Approach B
Input	Needed. Voice or text in, transcript out	Needed. Same
Extraction	Needed. Transcript to incident schema, run once	Skipped. There is no standalone extraction
Incidents	Needed. Stores the incident contract as the single source of truth	Not needed. Nothing canonical is stored
Templates	Needed. Box mapped to a schema field path	Needed. Box mapped to a plain field name
Forms	Fill by lookup and draw, no LLM	Fill by calling the LLM per field, then draw
Reporting	Needed and natural. Rollups query the stored structured data	Hard to support. No structured store to query

A few quick points on the layers:

• Input, templates, and forms are needed in both, just wired a bit differently.
• A adds extraction and incidents on top. B skips both.
• The forms layer is light in A (lookup and draw) and heavier in B (LLM per field).

So the short version is that A adds the extraction and incidents layers and gets reporting almost for free, while B drops extraction and incidents but does more work inside the forms layer and loses the easy path to reporting. Please correct me if I have any of this wrong, since I want the picture to match how you see it before we build.

What I see going for Approach A

Because extraction runs once and every form's boxes are already mapped to fields in the schema, filling a form is just placing known values, with no LLM call at fill time. That makes bulk filling fast. I also had prompt caching in mind here, since the forms and their mappings are known ahead of time, so we are not rebuilding context on every fill.

The bigger win I see is on the review side. All templates are preloaded, so the system already knows exactly what each form needs. Once extraction finishes and the data shows up on screen, we can look at every form and tell right away which ones can be filled from what we have. My idea is to show the forms that have all their data in green, ready to generate, with select and unselect so the user picks which ones to produce in the next step. The forms that are missing a field or two show up greyed out, and hovering or clicking on one tells the user exactly what is missing. From there they can type in the missing value by hand, which unlocks a few more forms. In B, the only way to fix a missing value is to re-prompt with more input or add some outside way to override the data.

The other reason review matters is trust. If the input itself has a mistake, or the model reads something wrong, in B that same wrong value gets extracted fresh into every form, so the user has to check each generated form point by point to be sure nothing false slipped through. These are official reports, so false information is both a real risk and a lot of manual review effort. In A the data is reviewed once, up front, before any form is generated. After that review the forms are considered reliable, because at fill time the LLM is not guessing anything, it is just placing values we already mapped and confirmed.

The last thing A handles well is the reporting side. Not every form is filed after an incident. Some are filed monthly, quarterly, or yearly for state and national statistics, and USFA guidelines already ask for quarterly state submission at a minimum. Because A stores every incident as structured data in Postgres, any range of that data can be queried in seconds with no bottleneck. A full year of incidents comes back fast. If we want the number of wildfire incidents in 2026, that is a simple filter on type and year and it returns in milliseconds. If we want fire incidents per month, say 10 in January 2026, 20 in February, and so on, that is one query too. So any stat we need is easy to pull, which helps both for analysis and for filling those periodic per-month and per-year forms. It keeps FireForm ready for these reporting tasks instead of us bolting them on later. In B this is much harder, because the data lives inside free text and per-form outputs and there is no structured store to query.

What I see going for Approach B

B shines when only a few forms are needed at a time. There is no schema to agree on and no mapping target to maintain in the middle, so the overall setup is simpler. For a small number of forms it could well be faster, since we skip the extraction pass and go straight from text to the one form we want. It's simple to work on and faster for less number of forms. Form fields are more flexible here i believe.

Would love to hear feedback from @marcvergees @abhishek-8081 @vharkins1

[chetanr25]

But I want to be honest about the one thing A has not proven, because it's the thing that actually decides this. A leans on a single extraction filling the whole contract in one pass. If that one pass hallucinates a field, the wrong value propagates to every form — whereas B's narrower per-field calls might contain that risk better. Marc, this is your core concern, and I don't think A can claim victory until it's tested. Fewer calls and better consistency don't matter if the single extraction is less accurate.

@abhishek-8081
Thanks for asking, I had thought about this before GSoC proposal submissions and I do have the flow ready in my mind (theoretical with few research).

Yes, this is the concern which even I thought of. And I agree, this is the part that actually decides it. Fewer calls and better consistency mean nothing if the extraction itself is less accurate.

But I want to clear up one thing, because I think A is being seen as one single big pass, and that is not how I plan to build it. (I have mentioned that Approach A would convert data only once in extraction layer, what I ment is we would convert raw text to data in one round or iteration)

The incident contract is too large to pass to the model in one call and expect a clean result. This is a known problem, not just my guess. The general advice on structured output is that very large schemas make the output worse, and once you cross around 50 fields you are told to split it into smaller extraction calls instead of one big one. Our contract is well past that, so a single pass would be the worst version of A, not the real one.

So the plan is the opposite of one pass. We break the schema into groups and the backend handles it step by step:

• send a small group of fields to the model
• get that part back and validate it against the schema, including the data types
• retry that group if it does not match
• once all groups are done, stitch the validated pieces into one JSON

So A ends up making narrow calls too, just like B does per field. The difference is A checks each piece against a known schema and stores it once, while B re-extracts into a fresh shape for every form.

And this is not just my own idea of how to do it. There is recent research on exactly this that argues you should turn extraction from one joint prediction into a controlled sequence of narrower decisions, each with a validation gate and a local retry. Their reasoning is the same point being raised here about hallucination: when you split it up, a mistake in one field does not corrupt the others, and a bad value can be retried on its own without redoing the whole object. In their tests this path reaches around 90 percent object-level accuracy, and schema-aware prompting with careful field naming hits about 95 percent field accuracy. So the grouped, validated approach is not a shortcut to dodge the accuracy problem, it is the approach that is meant to handle it.

Honestly, this makes A more complex than B. More moving parts and more orchestration. It is also slower upfront, because there are several calls before we get clean structured data, where B goes straight from text to the one form.

But the speed part is where I think we have a real solution. These grouped calls do not depend on each other, so we can hit Ollama in parallel instead of one after another. Ollama supports this directly through OLLAMA_NUM_PARALLEL, which lets one model take several requests at once, and reports put throughput at around 3 to 4 times serial at 4 parallel requests, with about 20 to 40 percent more latency on each single request. The model also stays warm for 5 minutes between calls by default, so we are not paying the load cost every time.

The one thing to be careful about is memory. Ollama's own docs say the RAM needed scales as OLLAMA_NUM_PARALLEL times the context length, so parallel calls are not free. Each parallel slot loads the model once and then adds its own KV cache on top, and that KV cache grows with the context length. So the total is roughly the model weights plus the number of parallel calls times the memory of one context.

The good news is that our default models are small, so this fits well. We are not running 70B models, our range is roughly 1GB to 5GB. They all use grouped query attention, which keeps the KV cache small, and that is what makes running several calls at once cheap. Here is how the resources map out, using the Q4 builds Ollama ships and about a 4K token context per call. The KV cache column is worked out from each model's own config, its layers times key-value heads times head size:

Model	Size on disk	VRAM to run (GPU)	RAM to run (CPU)	KV cache per parallel call (~4K ctx)
qwen2.5:1.5b	986MB	~1.6GB	~3GB	~0.11GB
qwen2.5:3b	1.9GB	~2.6GB	~4GB	~0.15GB
llama3.2:3b	2.0GB	~2.9GB	~4.5GB	~0.46GB
mistral:7b	4.4GB	~5.4GB	~8GB	~0.52GB
qwen2.5:7b	4.7GB	~5.5GB	~8GB	~0.23GB

On a GPU it spends VRAM, on a CPU-only box it spends about the same in system RAM plus some headroom for the OS, and Ollama's practical floor is 8GB of system RAM. The KV cache column is the important one for parallelism, because that is what each extra parallel call adds on top of the loaded model.

One thing I want to be careful about here. Memory would technically allow a lot of parallel calls for the small models, but that is not the number we should actually run. Two reasons. First, compute is the real limit, not memory. Past a handful of parallel calls each one gets a smaller slice of the GPU and slows down, so the gain flattens out. This is why Ollama itself defaults to 4 parallel rather than some large number, and why the 3 to 4 times throughput figure was measured at 4, not beyond. Second, we should not assume this service owns the whole card. The OS and anything else on the machine need room too, so budgeting all the memory to one service is not realistic.

So instead of a memory ceiling, here is a conservative parallel setting I would actually start with, leaving headroom on the card:

Model	Parallel on 8GB	Parallel on 12GB
qwen2.5:1.5b	4	4 to 6
qwen2.5:3b	4	4 to 6
llama3.2:3b	3 to 4	4 to 6
mistral:7b	2	3 to 4
qwen2.5:7b	3	4 to 6

These are deliberately modest. The point is not to squeeze the machine, it is that even at a safe 2 to 4 calls at once, the upfront extraction is already several times faster than doing the groups one after another, which is enough to make A practical. If a machine has more room, we can raise the cap, but I would rather start low and let the benchmark find the real sweet spot on actual hardware than assume a big number up front.

These are conservative estimates, not hard limits. The safe move is to cap parallel calls below what the card could technically take, tie the cap to the model and the VRAM on the machine, and start low and scale up while watching memory instead of maxing it out and risking a crash. The exact numbers are something the benchmark will pin down on real hardware.

So my point is simple. A is more complex and slower upfront, that is true. But the accuracy worry comes from the single pass picture, and grouped extraction with validation is actually the recommended way for a schema this big. Whether it lands more accurate than B's per form calls is exactly what the benchmark should tell us. I am not saying A wins on accuracy yet, I am saying the version of A worth testing is the grouped one.

---

Sources for the numbers:

• Ollama concurrency and memory scaling (OLLAMA_NUM_PARALLEL, RAM scales as parallel times context, models kept warm 5 minutes): https://docs.ollama.com/faq
• Throughput around 3 to 4 times at 4 parallel, the latency tradeoff, and the "start at 1 to 2 on limited VRAM" advice: https://www.glukhov.org/llm-performance/ollama/how-ollama-handles-parallel-requests/
• Splitting large schemas into multiple calls past ~50 fields, and validating output against a schema with retry: https://techsy.io/en/blog/llm-structured-outputs-guide
• Exact Ollama download sizes for these models (qwen2.5 1.5b 986MB, 3b 1.9GB, 7b 4.7GB): https://ollama.com/library/qwen2.5
• Ollama size for llama3.2:3b (2.0GB) and mistral:7b (4.4GB): https://ollama.com/library/llama3.2 and https://ollama.com/library/mistral
• Ollama system RAM floor of 8GB for CPU inference: https://localaimaster.com/blog/ollama-system-requirements
• How KV cache adds to memory per context and how to compute it from a model's config: https://mbrenndoerfer.com/writing/kv-cache-memory-calculation-llm-inference-gpu
• Research on splitting extraction into narrow, validated, retryable steps and the accuracy it reaches: https://arxiv.org/html/2604.27906
• Grouped query attention and how it shrinks the KV cache so the same hardware serves more requests: https://zeroentropy.dev/concepts/grouped-query-attention/

[abhishek-8081]

A small suggestion on the validation gate, in case it's useful. It confirms a field is well-formed and matches the schema, which is really valuable - though I think it's checking shape rather than truth. A made-up address, for instance, would still be a valid string and pass the check. So it seems to catch malformed output more than invented output, and since invented output is the concern Marc raised, it might be worth having the benchmark speak to that directly.

On that note, one thing I'd gently suggest we pin down before splitting branches is what the benchmark actually measures. Since the open question is mostly about fields that aren't in the input, it might help to include inputs where a field is genuinely absent and see how often each pipeline fills it in anyway. Generic field accuracy probably won't surface that on its own. And on the ~90/95% figures — they read as a good sign the approach is worth building, though since they're from a different dataset, maybe we treat them as encouraging rather than something to lean on for fire reports.

One more small thing worth keeping in mind: the benchmark can settle accuracy, but consistency across forms and reporting feel more like design outcomes than things we'd measure - A tends to get them naturally, B less so. So the genuinely open question really is accuracy, and it might be enough to test that with a thin extraction harness per pipeline rather than building both out fully.

Happy to help define the absent-field test if that'd be useful.

@marcvergees @chetanr25 @vharkins1

[chetanr25]

All valid points. And yes, having a validation layer after the extraction layer is a must. I’d consider that non-negotiable.

[vharkins1]

@chetanr25

These grouped calls do not depend on each other, so we can hit Ollama in parallel instead of one after another. Ollama supports this directly through OLLAMA_NUM_PARALLEL, which lets one model take several requests at once, and reports put throughput at around 3 to 4 times serial at 4 parallel requests, with about 20 to 40 percent more latency on each single request. The model also stays warm for 5 minutes between calls by default, so we are not paying the load cost every time.

I think this is a great point. I'd like to migrate to llama.cpp for the long term but we 100% should be batching.

For the topic on hallucinations I'm not sure why given the same base information about how to fill the form they'd be any different. It seems to me that the difference between approach A and B is that A is simplified for the user on setup. They wouldn't need to give the exact matching specification for the fields for each and every form, rather they'd just need to describe it. That being said I don't think it's that much of an ask for the user to define the specific fields if they already need to define it in quite a bit of detail.

To ease the burden on the firefighters creating these templates, we might want to create a method for sharing the form with all the information attached already. I think for both approaches this would be quite helpful.

@abhishek-8081

A small suggestion on the validation gate, in case it's useful. It confirms a field is well-formed and matches the schema, which is really valuable - though I think it's checking shape rather than truth. A made-up address, for instance, would still be a valid string and pass the check. So it seems to catch malformed output more than invented output, and since invented output is the concern Marc raised, it might be worth having the benchmark speak to that directly.

I totally agree with both the need for a JSON validation layer and a verification layer to check for mistakes.

In all I think the both of you @abhishek-8081 @chetanr25 have made a very compelling argument for why the mapping approach is better for the firefighters. What do you think? Am I missing any major things?

A couple things on the benchmark since that's what Marc and I are working on. I think the absent-field case @abhishek-8081 brought up should definitely be one of the main metrics. I'd expect hallucination per call to look pretty similar between A and B's per-field calls. The thing worth measuring is what happens after: in A a mistake gets made once and reviewed once before anything generates, in B every form re-extracts so the review work scales with the number of forms. And for the form sharing idea, that overlaps with #610 so we can continue that part there.