Skip to content

feat: add weighted confidence scoring for extraction validation#424

Open
Arijit429 wants to merge 9 commits intofireform-core:mainfrom
Arijit429:weighted-confidence-scoring
Open

feat: add weighted confidence scoring for extraction validation#424
Arijit429 wants to merge 9 commits intofireform-core:mainfrom
Arijit429:weighted-confidence-scoring

Conversation

@Arijit429
Copy link
Copy Markdown

@Arijit429 Arijit429 commented Apr 12, 2026

Closes #60
Closes #450

🚀 Summary

This PR enhances the extraction validation workflow by introducing field-level weighted confidence scoring instead of using a flat deduction model.

The goal is to make confidence evaluation more representative of real-world data importance and improve the reliability of the requires_review decision.


✨ What Changed

Updated the ExtractionValidator logic to use weighted importance for each field:

FIELD_WEIGHTS = {
    "location": 30,
    "time": 20,
    "severity": 30,
    "description": 20
}

Each missing field now reduces confidence based on its relative importance instead of applying a uniform deduction.


💡 Why This Helps

In the previous approach, all missing fields contributed equally to confidence reduction.

This could under-represent critical missing information.

For example:

  • missing location should have higher impact
  • missing severity should strongly affect review decision
  • missing description may be comparatively less critical

The new weighted model makes confidence scoring more realistic and production-aligned.


🔍 Example

Previous behavior

all fields = equal impact

New behavior

location → 30
severity → 30
time → 20
description → 20

This allows the validator to make smarter review decisions.


🧪 Testing

Added dedicated unit test coverage for weighted confidence scoring.

Validated scenarios include:

  • multiple high-weight missing fields
  • partial extraction completeness
  • review threshold correctness
  • confidence floor protection

Executed locally using:

PYTHONPATH=. pytest test/test_extraction_validator.py -q

and verified successful execution.


🎯 Impact

This improves:

  • confidence score realism
  • human review routing accuracy
  • downstream PDF fill reliability
  • production readiness of extraction workflow

@Arijit429
Copy link
Copy Markdown
Author

Hi maintainers, this update improves the review workflow by making confidence scoring field-aware and more aligned with real extraction importance—happy to refine the weighting logic based on project needs.

@Arijit429 Arijit429 force-pushed the weighted-confidence-scoring branch from cf4ec51 to ab7f503 Compare April 19, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant