Incident contract: lock the schema, settle naming, and define extraction accuracy rules

[abhishek-8081]

Umbrella thread for everything that has to be settled before AI extraction (#550) can be
built. #550 is blocked until these are resolved. Three parts.

1. Lock the incident contract

#550 is built entirely around the extracted JSON, so the schema must be final before any
extraction code. The last community meet decided some fields need to change to match
FireForm's current requirements.

@chetanr25 ,Can you please list the agreed changes here? Once they're
written down, we lock it and treat that as final for #550.
If an issue or PR already
tracks the schema edits, link it and we'll use that.

Current draft for reference: https://fireform-gsoc.chetanr25.in/incident-contract

2. Naming — this is a structural question, not just a word choice

Grepping the codebase, both terms are in heavy use, and they look like they may be two
different things:

• "incident contract" / incident_contract — used in extraction.yaml and
  extraction-record.yaml (the extracted JSON output). Schema file: incident-contract.yaml.
• "incident record" / incident-record — used in incidents.yaml (~13 times) and
  openapi.yaml. Separate schema file: incident-record.yaml.

So there are two schema files and two terms, split across the extraction path vs the
incidents/CRUD path.

@chetanr25 -are these the same object with two names, or two different things (e.g. the
extraction output vs the stored incident from #555)?

• If same: pick one name, merge/delete the duplicate schema file.
• If different: give them clearly distinct names so they stop getting mixed up.

3. Accuracy rules (assigned to me and @vharkins1)

These decide how extraction behaves, and they can't be set in the abstract — they depend
on what the local model can actually do. Open questions:

• If the model is unsure about a field: leave it empty, or guess?
• What rate of wrong/hallucinated values is acceptable?
• If a field is low-confidence: does form generation stop, or does the field get flagged
  for a human to review later?

I'll run a viability test before we decide any of these: real sample narratives against
the current draft schema, measuring valid-JSON rate, per-field accuracy, hallucination
rate, and whether one-shot extraction of 100+ fields holds or whether we need to split
into smaller calls. That last result changes the architecture, so it comes before code.

I'll post the numbers here so these rules get decided on evidence.

@vharkins1 — input welcome on all three, especially the accuracy rules.

[chetanr25]

Thanks for putting this together. Replying to all three.

1. Locking the incident contract

The draft schema and the link were proposed by me, and in the last community meet @marcvergees said a few things need to change to match FireForm's current requirements.
One thing to note though, we did not actually go through the list of exact fields that need to change in that meet. We agreed changes are needed, but we did not write down what those changes are. Since these are @marcvergees's points, I think he is the right person to list what he wants changed. Once we have his input we can draft a first version, and then we lock that as final for #550. There is no issue or PR tracking the schema edits yet as we had to discuss what should be changed in the JSON.

2. Naming

This has been on my mind for a while too. We never picked a proper name for this JSON, so both terms ended up in the codebase. I have wanted to bring this up for a while, so this is a good place to discuss and finalise.

• The incident contract is the big extracted JSON. It holds every field any form could need, and it is what the extraction layer fills in. This is what AI extraction layer - API Contracts #550 produces.
• The incident record is the stored incident from the incidents/CRUD layer (Incident CRUD operations - API Contracts #555). It is a small tracking object with the id, status, tags, which forms were generated, and timestamps. It does not hold the extracted data itself, it just points to the contract that has it.

When I built the contract, these two names felt too close, but I did not want to pick the final name by myself, so I left them as is at that time. I wanted us to decide the naming together later, and I think this is the best time to do.

So they should stay as two separate things, but we need to agree on clearer names for both and then update them everywhere in the contracts and the codebase.

---

3. Accuracy rules

Small correction, I (or we) did not assign this to @vharkins1. In the meet he showed interest in these kinds of issues, so I just thought he would be a better person to address this issue than me. Looking forward to the numbers. The one-shot vs split-calls result is the big one, since it decides the architecture before any code gets written.

[marcvergees]

Hey @abhishek-8081, @chetanr25 and @vharkins1.

1. Incident contract

I agree guys with the fact that something has to be changed, but basically is gonna be getting rid of everything. We don't need to store any field in the incident rather than which was the input given to the LLM. It looks like @chetanr25 tried to create a structured schema of what an incident was, but we don't need that based on how we approach FireForm. Furthermore, since FireForm is going to be something bigger and emergency-department-agnostic soon, we don't need to add specific fields of fire departments in the incident contract.

2. Naming

For easiness, could we call them:

• Incident schema: to refer to the schema that the LLM has to match (that basically is the JSON with keys the fields and values the type of field that has to be generated by the LLM, i.e. float, string, boolean, etc)
• Incident record: record in the db of the job/task created, so that we can track the action that the client did

3. Accuracy rules

Based on my experience in this project I can tell:

1. It's almost impossible to fill multiple fields at once. That's why we implemented a call per each field. You can test by yourself, but it's too complicated with this low-weight models. Since soon we're migrating to something bigger, maybe is something we can leave for later.
2. The idea of confidence score is not bad but I wouldn't implement it. Definitely lots of GSoC candidates proppose it but I'm not extremely keen because the fact of AI hallucination again. Who is going to determine which is the confidence score and which is going to be the threshold? I feel like is solving a problem using heuristics.
3. In my opinion, the hallucination fact should be reviewed by the end-user, i.e. the firefighter or any user of FireForm. We cannot guarantee lack of hallucination because we know that exists, so I feel like we should fill every field regardless if there's an hallucination and then the final user doublechecks.