FireForm
Replies: 2 comments
|
Hi guys! When Vincent says "Some of the limitations of this approach include: Long context causing reduced accuracy" means literally MB of .txt of narrative. |
Benchmark Accuracy Root-Cause Analysis
📊 Top-Level Summary
🔴 Part 1 — Structure Failures (4 cases, forced to 0% accuracy)These four cases never reached the value-accuracy step because the extracted JSON did not match the ground-truth schema. ❌
|
| Problem | Detail |
|---|---|
| Missing key in extracted | 8_current_and_planned_actions_strategies_tactics |
| Extra key in extracted | (none) |
| Sub-key mismatch | Key present but sub-keys differ inside resources_summary |
Root cause: The model dropped the actions_tactics section entirely — likely because the form narrative was long and the field appeared near the end of the prompt context window.
❌ ics201_3
| Problem | Detail |
|---|---|
| Missing key in extracted | 8_current_and_planned_actions_strategies_tactics |
| List item schema mismatch | Inside resources_summary: extracted items lack kind sub-key |
Root cause: Same pattern as ics201_2 — the actions table is truncated/dropped by the model, and the resource list schema is inconsistent (missing kind field per item).
❌ ics209_2
| Problem | Detail |
|---|---|
| Top-level key mismatch | Extracted contains extra wrapper key vs. GT flat structure |
| Sub-key mismatch | 2_approval_and_location.location → extracted lacks short_description, has extra county |
Root cause: The ICS 209 schema is complex and nested. The model invented a slightly different nesting for the location sub-object, causing the structural comparator to reject the whole case.
❌ ics204_5
| Problem | Detail |
|---|---|
| List item schema mismatch | assignment_list items: extracted lacks reporting_location sub-key |
| Type mismatch | One scalar field extracted as list |
Root cause: The model collapsed multi-row assignment list entries and omitted reporting_location from each row's schema.
🟡/🟢 Part 2 — Value Accuracy Issues (52 cases that passed structure check)
Recurring Error Patterns (cross-case)
These patterns account for the bulk of accuracy loss across all structure-valid cases:
🔁 Pattern A — Date format mismatch (affects ~30 cases, ~3–5% loss each)
ext='2026-07-07' ≠ gt='07/07/2026'
ext='2026-10-14 06:15' ≠ gt='10/14/2026 06:15'
The model always outputs ISO-8601 (YYYY-MM-DD) while ground truth is in US MM/DD/YYYY format. The date normalizer in AccuracyCalculator catches some cases but not all (particularly combined date+time strings and timezone-suffixed strings like 07:15 EDT).
Fix: Extend try_parse_datetime to handle timezone suffixes and combined datetime strings more aggressively, OR normalize the GT to ISO before comparison.
🔁 Pattern B — Role prefix leaked into name field (affects ~20 cases)
ext='Safety Officer Captain Thomas Wright' ≠ gt='Captain Thomas Wright'
ext='Communications Unit Leader J. Myers' ≠ gt='J. Myers'
ext='Planning Section Chief Commander Richard Croft' ≠ gt='Commander Richard Croft'
The model copies the role title from the form label into the name field instead of extracting just the person's name/callsign. This affects prepared_by, approved_by, and all organization_chart name fields.
Fix: Prompt engineering — explicitly instruct the model to extract only the name, not the preceding role label.
🔁 Pattern C — Incident name suffix appended (affects ~10 cases)
ext='Whispering Pines Derailment Operational Period' ≠ gt='Whispering Pines Derailment'
ext='Silverwood Reservoir Crude Oil Pipeline Rupture Operational Period' ≠ gt='...'
ext='Port of Houston Sulfuric Acid Tanker Leak Incident' ≠ gt='Port of Houston Sulfuric Acid Tanker Leak'
The model appends section context ("Operational Period", "Incident") to the incident name field.
Fix: Add explicit prompt instruction: "Do not append form section names to the incident_name value."
🔁 Pattern D — iap_page_number format (affects ~6 cases)
ext='Page 6' ≠ gt='6'
Model outputs the full label "Page 6" instead of just the numeral.
Fix: Add explicit schema hint: iap_page_number should be a bare integer string.
🔁 Pattern E — Phone number formatting (affects ~5 cases)
ext='555-456-7890 / Ch 1' ≠ gt='(555) 456-7890 / Ch 1'
Model strips parentheses from phone numbers. Fuzzy matcher scores this high but it doesn't reach 100%.
Fix: Normalize phone numbers before comparison (strip all non-digit characters from both sides), or accept both formats in the evaluator.
🔁 Pattern F — Sentence trailing punctuation (affects ics201 family heavily)
ext='Maintain life safety and enforce the 500-foot exclusion perimeter by 08:00 hours'
gt= 'Maintain life safety and enforce 500-foot exclusion perimeter by 08:00 hours.'
Objectives and actions strings are missing trailing periods or have minor paraphrasing. Fuzzy matcher catches most (>83%), but the LLM judge sometimes marks these PARTIAL.
🔁 Pattern G — ICS 207 org-chart role prefix in section names
ext='Vessel Salvage & Entry Branch' ≠ gt='Vessel Salvage & Entry Branch Director'
ext='Operations Section Chief Battalion Chief A. Ross' ≠ gt='Battalion Chief A. Ross'
Opposite of Pattern B in some fields: the model omits the role suffix (Director/Supervisor) from branch titles, and prefixes the role name into person fields.
🔴 Low-Accuracy Outlier Cases
| Case | Accuracy | Main cause |
|---|---|---|
ics201_1 |
40.6% | Near-complete mismatch on map_sketch sub-fields + actions table paraphrasing |
ics201_2 |
0% | Structure failure |
ics201_3 |
0% | Structure failure |
ics209_2 |
0% | Structure failure |
ics204_5 |
0% | Structure failure |
ics201_4 |
84.4% | Heavily paraphrased map_sketch + date format + summary verbosity |
ics209_1 |
84.6% | report_version/report_number fields swapped + location format |
🟡 Medium Cases (70–85%)
| Case | Key issues |
|---|---|
ics201_1 |
Map sketch fields: model gives abbreviated values, GT has full descriptive text |
ics209_1 |
report_version / report_number fields swapped — model put "ICS 209" in version and "4" in number, GT is reversed |
ics207_3 |
Role prefixes in org chart (Pattern B + G combined) — systemic across all 30+ org-chart fields |
✅ Part 3 — What's Working Well
- Boolean & numeric fields: Near-perfect accuracy where applicable
- Structured datetime fields with clear separation (date+time as separate keys): well-handled
- Free-text safety messages (ICS 208): 99–100% on most cases
- ics208_2: 100% accuracy — clean template, no role-prefix issues
🛠️ Recommended Fixes (Priority Order)
| Priority | Fix | Est. Impact |
|---|---|---|
| 🔴 1 | Extend date normalizer to handle MM/DD/YYYY HH:MM and timezone-suffix strings |
+3–5% avg accuracy |
| 🔴 2 | Prompt: strip role prefix from name fields — "extract only the person's name, not their role title" | +5–8% avg accuracy |
| 🟡 3 | Prompt: incident_name should not include form context words | +2–3% avg accuracy |
| 🟡 4 | Normalize phone numbers in evaluator before string comparison | +1% avg accuracy |
| 🟡 5 | Fix iap_page_number schema hint — bare integer, not "Page N" |
+1% avg accuracy |
| 🟢 6 | Fix ics209 location schema (short_description vs county) in template |
Unblocks 1 structure failure |
| 🟢 7 | Fix ics201 actions/tactics section truncation (longer context / chunked prompt) |
Unblocks 2–3 structure failures |
Uh oh!
There was an error while loading. Please reload this page.
As we've discussed in our weekly meetings the benefits of one shotting filled JSON templates has proved extremely successful. For anyone unfamiliar, this means we send the full form schema and narrative to the LLM in a single call rather than a multi-step pipeline. We've seen higher accuracy filling out ICS forms, as well as increased speed (~30s).
Me and @marcvergees takeaway was that these models performed much better when given a straightforward task.
Some of the limitations of this approach include:
For field descriptions, I discovered that most ICS PDF form widgets have /TU tooltip attributes, these are author-written screen-reader strings baked into the PDF itself. On ICS 201 for example, 174 out of 190 widgets had them. This means we can pull accurate field descriptions directly from the PDF. I tried to use an LLM to make a first pass to give descriptions for each field as a potential solution, but using the /TU tooltip provided just as good information.
Table creation was very important to increase accuracy. Using PyMuPDF's ruling-line detection to find table boundaries had pretty high accuracy. Being able to recreate the tables in the JSON template increases the accuracy substantially so I think this is very important to include and focus on for any implementation we end up with.
I've spent a lot of my time over the past couple weeks ideating on the user experience. It's really important that the FireFighter (and any user) has the easiest onboarding path possible. By this I mean they shouldn't have to manually type field descriptions and field names if it can be at all avoided.
One solution that I've come up with is to have a list of supported forms. This would allow us to do three things. The first is that we can custom design the field descriptions to help the LLM as much as possible. Secondly we could design custom prompts form by form increasing the accuracy. Third, it would allow us to benchmark and refine issues by removing edge cases.
While I think it's a path worth considering, I think we'd all prefer to have a more general solution. One that's form agnostic and still has good performance. The tooltip and geometry work has actually closed a lot of the gap here, most of what "supported forms" would give us, these automated methods provide for free on any well-authored PDF.
The second is a hybrid approach, we can ingest forms the same way we currently are and provide a frontend to edit the field descriptions and use an LLM as the first pass for template creation. This would look like inserting descriptions using the LLM (or pulling them from tooltips where available) and allowing the firefighter to edit it manually where issues arise. @chetanr25 you've made a really compelling front end for this, and I think it's worth moving forward with.
All reactions