Change-Order Extractor

A Python pipeline for extracting structured data from construction change orders into validated JSON, with field-level confidence scores, supporting evidence, and human-review warnings.

Built for the Sledge AI Engineer assignment.

All sample documents and evaluation labels in this repository are synthetic and were created for development and testing.

Construction change orders can contain inconsistent formatting, ambiguous values, multiple monetary amounts, and incomplete information. Extracting these fields reliably requires more than matching numbers and labels.

This project implements a layered extraction pipeline that combines deterministic rules, value normalization, validation, evidence tracking, heuristic confidence scoring, and an optional large language model (LLM) fallback.

The central design principle is trustworthy extraction over forced completion: when the available evidence is insufficient, the system can return null and flag the document for review instead of inventing a value.

• Multi-format input: Processes text files and PDFs with extractable text.
• Structured JSON: Extracts nine change-order fields into a consistent schema.
• Evidence tracking: Preserves supporting source text for individual field values.
• Field-level confidence: Assigns heuristic confidence scores to extracted values.
• Value normalization: Handles supported date formats, monetary amounts, credits, and approval statuses.
• Validation and warnings: Flags missing information, ambiguous values, competing amounts, and potentially unsafe input.
• Conservative PDF handling: Flags scanned PDFs without a usable text layer for OCR rather than pretending extraction succeeded.
• Optional LLM fallback: Supports configured Gemini or Anthropic providers for eligible unresolved fields.
• Automated testing: Includes unit, regression, safety-related, CLI, and mocked LLM fallback tests.
• Evaluation tooling: Measures extraction results against labelled synthetic datasets.
• Reproducible workflow: Includes sample documents, evaluation labels, dependency files, and generated JSON outputs.

• Python
• pip

Open PowerShell or a terminal:

git clone https://github.com/salikhussain71-code/change-order-extractor.git
cd change-order-extractor

On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install -r requirements-dev.txt

python -m pytest -q

Verified result: 49 tests passed in the reported local run.

python -m change_order_extractor data

The command processes the supported sample files and writes generated JSON outputs to outputs/.

python -m change_order_extractor data\co_003.txt

python -m change_order_extractor.evaluate data evaluation\labels.json

python -m change_order_extractor.evaluate data\heldout evaluation\heldout_labels.json

Install dependencies in the environment where you run the project. In particular, pdfplumber is required for PDF text extraction.

The pipeline follows a layered workflow:

Input document
      |
      v
Document reading
      |
      v
Text normalization
      |
      v
Rule-based extraction
      |
      v
Value normalization
      |
      v
Validation and safety checks
      |
      v
Confidence scoring
      |
      v
Optional LLM fallback
      |
      v
Structured JSON + evidence + warnings

The LLM fallback is optional. The default rule-based workflow does not require an external model API key.

Text files are read directly. PDFs are processed using pdfplumber to extract their text layer.

If a PDF has no usable text layer, the pipeline flags it as requiring OCR. A complete OCR extraction workflow is not currently implemented.

The pipeline normalizes formatting differences such as whitespace, unusual spaces, and dash variations to improve matching consistency.

Field-specific patterns are applied, prioritizing explicit labels over less reliable matches. Where implemented, tolerant patterns can handle certain label variations and typographical errors.

Rules are used first because many change-order documents contain recognizable labels and semi-structured fields. Deterministic rules are easy to inspect, reproduce, and debug.

Supported values are converted into consistent representations:

• Dates to ISO format (YYYY-MM-DD) when interpretable.
• Monetary amounts to numeric values.
• Credits to negative amounts when supported by the source notation.
• Approval statuses to supported values.
• Unresolved values to null.

The pipeline checks for issues such as missing required fields, implausible values, competing amounts, ambiguous dates, and potentially adversarial instructions embedded in the source text.

Warnings and review flags help identify documents that should not be accepted automatically.

Each extracted field receives a heuristic score based on the strength of its evidence and relevant validation concerns. A weighted overall score is also calculated.

Confidence scores are not calibrated probabilities. They represent the implementation's assessment of evidence strength, not a statistically established probability that a prediction is correct.

When configured, an LLM can attempt to resolve eligible missing or low-confidence fields. Its output remains subject to the pipeline's validation approach.

The fallback is covered by tests using a mocked model. The automated suite does not make live requests to Gemini or Anthropic, so real-provider authentication, network behavior, provider-specific errors, and actual model responses have not been verified.

Each field contains the following metadata:

The document-level result also includes:

• source_file
• warnings
• overall_confidence
• needs_review

The following shortened example illustrates the output structure for data/co_003.txt:

{
  "source_file": "co_003.txt",
  "fields": {
    "change_order_number": {
      "value": "031",
      "confidence": 0.95,
      "evidence": "CHANGE ORDER NUMBER: 031",
      "method": "rule"
    },
    "amount": {
      "value": -3200.0,
      "confidence": 0.88,
      "evidence": "Credit Amount: ($3,200.00)",
      "method": "rule"
    },
    "schedule_impact_days": {
      "value": null,
      "confidence": 0.0,
      "evidence": null,
      "method": "none"
    }
  },
  "warnings": [
    "Amount interpreted as a credit (negative).",
    "Schedule impact not found."
  ],
  "overall_confidence": 0.79,
  "needs_review": false
}

This example is shortened for readability. The generated output files in outputs/ contain the actual results produced by the pipeline.

A missing value can remain null while other fields are extracted. This supports partial results without requiring the system to invent unsupported information.

The latest reported local test run completed with:

49 passed, 0 failed.

The test suite covers extraction behavior, normalization, regression cases, validation and safety-related behavior, CLI processing, and LLM fallback behavior using a mocked model.

Passing the tests confirms the covered behaviors for the tested environment. It does not prove that every possible document format or real-world failure mode is handled.

The held-out evaluation exposed two issues:

1. Handling the Sept date abbreviation.
2. Parsing the $(7,500.00) credit format.

After fixes, the held-out result improved from 44/48 to 47/48 correct fields.

Because the held-out examples informed these fixes, the updated result is no longer a fully untouched evaluation. The initial result is the more conservative reference; the updated result documents the observed improvement.

The remaining mismatch is the project_name field in h_01.txt.

• Expected value: Hillcrest Senior Living
• Actual rule-based output: null
• Cause: the project name appears in a title line rather than a pattern recognized by the current project-name extraction rules.

Returning null avoids presenting an unsupported project name as a confirmed extraction. Title-aware rules or a validated LLM fallback are possible future improvements.

The latest held-out evaluation reported:

• Mean confidence on correctly extracted fields: 0.89 across 47 fields.
• Mean confidence on wrong predictions: unavailable (nan) because there were no incorrect non-null predictions in that run.

This diagnostic does not establish confidence calibration.

The datasets contain 17 labelled documents in total and are synthetic. They are useful for regression testing and demonstrating known behavior, but they are too small to establish reliable production accuracy.

Performance on representative real-world documents must be measured before making production-level claims.

Problem: The phrase level 1 appeared in descriptive text containing the word credit. An overly broad pattern interpreted the number as a monetary credit.

Fix: Restrict amount extraction to supported labelled amount lines.

Lesson: A number should not be treated as a monetary value without sufficient contextual evidence.

Problem: A document contained a new contract sum of $1,312,300 alongside the actual change amount. The extractor initially selected the wrong figure.

Fix: Exclude recognized original, previous, new, or revised contract-sum lines when selecting the change amount.

Lesson: Correct extraction requires distinguishing the role of each value, not merely matching numeric patterns.

Problem: An instruction embedded in a document attempted to change the extracted amount to $1.

Fix: Constrain extraction to supported evidence patterns and add a warning for suspicious instruction-like text.

Lesson: Document content should be treated as input data, not trusted instructions that override extraction logic.

Problem: The initial held-out evaluation exposed failures involving Sept and $(7,500.00).

Fix: Update the relevant normalization and extraction rules, then verify the changes with evaluation and regression tests.

Lesson: Differently phrased examples reveal assumptions that development examples may not expose.

The potential improvements listed above are future work, not claims that these capabilities are already implemented.

The default rule-based pipeline does not require an API key.

To use the optional Gemini fallback in PowerShell:

$env:GEMINI_API_KEY = "your-key"
python -m change_order_extractor data --llm gemini

To use the optional Anthropic fallback:

$env:ANTHROPIC_API_KEY = "your-key"
python -m change_order_extractor data --llm anthropic

Replace your-key with your own valid credential. Do not commit API keys, credentials, or other secrets to the repository.

The fallback has mock-based test coverage but has not been verified through live-provider testing.

change-order-extractor/
├── .github/workflows/       # Workflow configuration
├── change_order_extractor/  # Extraction, normalization, validation, CLI and evaluation
├── data/                    # Synthetic documents and held-out samples
├── evaluation/              # Evaluation labels and PDF-generation helper
├── outputs/                 # Generated JSON results
├── tests/                   # Automated tests
├── .gitignore
├── LICENSE
├── README.md
├── TECHNICAL_WRITEUP.md     # Detailed technical approach and failure analysis
├── pyproject.toml
├── requirements.txt
└── requirements-dev.txt

1. OCR support: Add extraction for scanned PDFs and images, with validation of OCR output.
2. Larger evaluation dataset: Expand the labelled dataset with permitted, representative documents.
3. Confidence calibration: Compare heuristic scores with observed correctness and evaluate calibration methods such as isotonic regression when sufficient labelled data is available.
4. Layout-aware extraction: Improve support for tables, multi-column layouts, and values split across lines.
5. Document segmentation: Separate multiple change orders within multi-page PDFs.
6. Evidence review interface: Highlight supporting text and allow reviewers to correct uncertain values.
7. Live LLM evaluation: Measure accuracy, latency, cost, and failure handling for configured providers.
8. Expanded safety tests: Add more cases involving misleading instructions, ambiguous amounts, unusual formats, and unexpected document structures.

See TECHNICAL_WRITEUP.md for the detailed design rationale, confidence-scoring approach, evaluation methodology, debugging examples, failure modes, and future work.

Released under the MIT License. See LICENSE for details.