A production-grade pipeline for extracting structured data from unstructured oil & gas reports (10-Ks, earnings PPTs, press releases, AIFs) into a 39-column analyst-ready schema — powered by a versioned rule registry, blocking pre-flight checks, and a two-layer eval loop.
What this repo is really about: a case study in getting an LLM-based extraction pipeline from "usually works" to demonstrably stable. The domain (upstream O&G filings) is the harness; the interesting part is the engineering pattern.
Financial and reserves data in oil & gas filings is scattered across dozens of tables, footnotes, charts, and narrative paragraphs. Human analysts spend hours per report copying numbers into a fixed 39-column schema so the data becomes queryable. A naive LLM pass looks great on the first PDF and quietly regresses on the fifth, because:
- Rules drift — corrections made in one session don't reach the next.
- Context skips — critical registries get dropped from long-context runs and nobody notices until QC.
- No eval loop — "the output looks right" isn't a test.
This pipeline addresses all three with an architectural pattern, not a bigger model.
| Layer | File | Purpose |
|---|---|---|
| WHAT to extract | docs/SOP.md |
The domain SOP — what a "byte" is, what counts, what to skip |
| HOW to run | skill/SKILL.md |
The pipeline stages: extract → identify → normalize → validate → emit |
| WHICH rules apply | skill/rules/rules.yaml → rules.md |
Versioned rule registry, single source of truth |
The SOP describes the human process. The SKILL describes the pipeline. The rules describe the enforceable corrections learned across runs. No file restates another. Rules are edited only in rules.yaml; rules.md is regenerated and is the file the model reads at extraction time.
Every rule earns enforcement. A new pattern lands as candidate (do not apply — just observe). After it holds for a second run it becomes confirmed (flag deviations). After a third run it becomes locked (auto-fix by validator). This prevents overfitting a single report's quirks into the global registry.
See docs/rules-evolution/ for archived snapshots of rules.yaml showing how the registry grew.
Every run begins with a required-reading checklist. Extraction cannot start until four files are confirmed loaded: SOP, SKILL, rules.md, and the memory index. This gate exists because it was skipped once and cost 20 percentage points of accuracy on the same PDF.
scripts/validate_output.py— enforces LOCKED rules against the output XLSX. Auto-fixes what it can, flags what it can't.scripts/audit_coverage.py— the miss-catcher: re-scans the source PDF and flags numbers the extraction skipped. Every flag must be resolved individually. Bulk-dismissal is forbidden (that failure mode caused 28 misses on one run).docs/QC_prompt.md— a manual re-audit prompt used as a third-layer safety net.
| Milestone | Metric |
|---|---|
| Prior run (Company A, Feb 2026 PR, 9-page WCSB reserves + ops) | 268 / 274 = 97.8% loose-match vs gold |
Regression (same PDF, rules.md skipped) |
215 / 274 = 78.5% |
| Post-fix (STOP gate enforced) | Restored to ~97% |
Full write-up: docs/results.md. The regression story itself is the single most useful artifact in this repo — it's a worked example of how eval loops earn their keep.
skill-driven-extraction/
├── README.md ← you are here
├── ARCHITECTURE.md ← deeper dive on the pattern
├── docs/
│ ├── diagram.svg ← architecture picture
│ ├── SOP.md ← the domain SOP (what to extract)
│ ├── walkthrough.md ← end-to-end on a public 10-K
│ ├── results.md ← accuracy numbers + the regression story
│ ├── QC_prompt.md ← manual re-audit prompt
│ ├── engineering-notes/ ← anonymized doctrine memories (the "why" behind every rule)
│ └── rules-evolution/ ← archived rules.yaml diffs
├── skill/
│ ├── SKILL.md ← pipeline definition + STOP gate
│ ├── rules/
│ │ ├── rules.yaml ← source of truth
│ │ ├── rules.md ← generated, loaded by the model
│ │ └── history.yaml ← version log
│ ├── references/ ← domain reference docs
│ └── examples/ ← few-shot extraction samples
├── scripts/ ← validator, auditor, transforms
├── tests/ ← regression harness + scoring
├── templates/ ← blank 39-column skeleton + parameter window
└── examples/public_10k_sample/ ← runnable example on an SEC-EDGAR filing
The pipeline is designed to be invoked as a Claude skill. To run it against your own PDF:
# 1. Drop a US or Canadian upstream O&G report PDF into examples/
# 2. Point Claude at skill/SKILL.md and let the STOP gate run
# 3. Validate the output
python3 scripts/validate_output.py \
--xlsx examples/public_10k_sample/output.xlsx \
--report-type "AR" \
--pdf-pages 120 \
--country USA \
--auto-fix
# 4. Audit coverage
python3 scripts/audit_coverage.py \
--pdf examples/public_10k_sample/source.pdf \
--xlsx examples/public_10k_sample/output.xlsxFull walkthrough on a public Devon Energy 10-K: docs/walkthrough.md.
The three-layer separation, the rule lifecycle, the STOP gate, and the two-layer eval loop generalize to any high-accuracy structured-extraction task where:
- The output schema is fixed and non-negotiable
- Ground truth exists (or can be produced by an expert)
- The cost of a silent miss > the cost of an extra flag
Insurance claim processing, medical coding, contract abstracting, and regulatory filing intake all fit the shape.
Apache 2.0 — see LICENSE.
Built iteratively across ~40 real-world extraction runs. Every rule in rules.yaml traces back to a specific miss on a specific report. Every engineering note in docs/engineering-notes/ is scrubbed doctrine from those runs. Client and operator names have been anonymized.