Skip to content

Add requires_review flag to detect incomplete LLM extraction#349

Open
Arijit429 wants to merge 6 commits intofireform-core:mainfrom
Arijit429:add-requires-review-flag
Open

Add requires_review flag to detect incomplete LLM extraction#349
Arijit429 wants to merge 6 commits intofireform-core:mainfrom
Arijit429:add-requires-review-flag

Conversation

@Arijit429
Copy link
Copy Markdown

@Arijit429 Arijit429 commented Mar 25, 2026

Closes #450
Closes #40

Overview

This PR enhances the reliability and safety of the FireForm extraction pipeline by introducing a validation layer that detects incomplete or low-confidence LLM extraction results.

In the current workflow, form submissions proceed even when the language model fails to extract critical fields (e.g., returning placeholder values like "-1" or missing entries). This change introduces a structured mechanism to flag such cases for human review.

Key Changes

Extraction Validation Utility

  • Added a reusable validation helper (src/utils/validation.py) to evaluate extracted JSON data against required template fields.
  • The validator checks for:
    • Missing keys
    • Empty string values
    • Placeholder extraction outputs such as "-1"

requires_review Flag Integration

  • Extended the extraction pipeline to compute a requires_review boolean after LLM-based extraction.
  • Updated FileManipulator.fill_form() to return both:
    • Generated PDF path
    • Review requirement status

Controller and API Flow Updates

  • Propagated the validation flag through the Controller layer.
  • Updated /forms/fill route to persist the review state alongside form submission data.

Database Model and API Contract Extension

  • Added requires_review column to the FormSubmission model.
  • Extended FormFillResponse schema to expose review status to API consumers.

Motivation

LLM-driven extraction is inherently probabilistic and may produce incomplete outputs in real-world emergency scenarios. By surfacing a review indicator, this change supports safer operational workflows and enables future human-in-the-loop validation interfaces.

Impact

  • Non-breaking enhancement to submission pipeline.
  • Improves transparency and reliability of automated form generation.
  • Establishes groundwork for advanced validation, confidence scoring, and UI review flows.

Future Work

  • Field-level confidence metrics
  • Structured extraction schema enforcement
  • Frontend review dashboards for flagged submissions
  • Retry or fallback extraction strategies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant