Skip to content

perf(image-evidence): collapse Stage C OCR attempt ladder from 3 tiers to 2 (issue #677) - #698

Merged
WilfordGrimley merged 3 commits into
feat/extractor-ownership-visibilityfrom
feat/collapse-ocr-ladder-677
Aug 5, 2026
Merged

perf(image-evidence): collapse Stage C OCR attempt ladder from 3 tiers to 2 (issue #677)#698
WilfordGrimley merged 3 commits into
feat/extractor-ownership-visibilityfrom
feat/collapse-ocr-ladder-677

Conversation

@WilfordGrimley

@WilfordGrimley WilfordGrimley commented Aug 5, 2026

Copy link
Copy Markdown

Summary

  • Collapses _collector_line_ocr_attempts (issue Stage D no-text bucket: OCR preprocessing/crop recovery (88.8% of skips are fixable, not a coverage ceiling) #259's Stage C OCR attempt ladder) from 3 tiers/8 tesseract calls to 2 tiers/6 calls: tier 3 (a re-try of tier 1's own variants under ALTERNATE_TESSERACT_CONFIG/PSM 11) is removed. Two fresh real-production probes (worktree-only analysis scripts, not committed to the repo — consistent with how this repo already cites other uncommitted probe tooling) walked the old 3-tier ladder to full completion over 450 forced-escalation cards (300 currently-blank + 150 currently-resolved, both freshly re-derived live 2026-08-05) and recorded, per card, the first tier at which a candidate-validated genuine match appeared (local_ocr.validate_against_candidates against the card's own real name-scoped candidates, not merely a lexicon-shaped parse). Tier 2 produced 2 genuine matches across the 450-card sample; tier 3 produced zero — only more lexicon-valid-but-uncorroborated noise (6/300 in the blank pool).
  • Cross-checked the collapse against all 30 golden_set.GOLDEN_CARD_IDS cards the same way: none of the 30 ever resolved uniquely at tier 3 (genuine match, lexicon-valid parse, or "best invalid" fallback) — this collapse changes zero golden-set expectations.
  • Worst case now pays for 6 tesseract calls, down from 8 (pre-Stage C OCR passes: collapse the 4x attempt ladder and the duplicated bottom-band read (4 passes -> 2) #677) and up from Stage D no-text bucket: OCR preprocessing/crop recovery (88.8% of skips are fixable, not a coverage ceiling) #259's original 2.
  • COLLECTOR_LINE_OCR_EXTRACTOR_VERSION/COLLECTOR_LINE_TSV_EXTRACTOR_VERSION/ARTIST_OCR_EXTRACTOR_VERSION bumped v2→v3 (all three read through the same escalation loop). ARTIST_OCR_EXTRACTOR_VERSION jumps to v4 instead: fix(image_evidence): wire the artist-crop fallback into live extraction #685 (merged separately, unrelated modern_artist_credit fallback wiring) already claimed v3 for that extractor, and stamping two different behaviours under one version string would defeat MANIFEST_EXTRACTOR_CURRENT_VERSIONS's staleness filter. LEGAL_LINE_EXTRACTOR_VERSION is unchanged.
  • Investigated and explicitly did NOT ship the issue's second named restructuring — unifying collector_line_ocr's narrow crop (DEFAULT_CROP_BOX) and legal_line's full-width crop (LEGAL_LINE_CROP_BOX, same y-band) into one OCR pass. A real-image accuracy probe on 150 currently-successful production cards (same worktree-only, uncommitted convention as above) found that re-reading tier 1 from the wider crop with the same preprocessing regresses 46/150 (30.7%) of currently-correct collector_line_set_code/collector_line_collector_number reads — mostly proxy/watermark text and artist full names getting picked up by the "search after the number" set-code fallback in place of the real code, and in some cases the wider image causing tesseract to drop the collector-number line entirely. Zero improvements found (0/150).
  • Lifts the EXTRACTOR_OWNERSHIP EXCLUDED_HELPERS entry for the OCR ladder that feat(image-evidence): make evidence-affecting helpers version-visible + CI totality #690 left in place while this branch was in flight — _collector_line_ocr_attempts and preprocess_fallback_variants now carry real ownership entries ({"collector_line_ocr", "collector_line_tsv", "artist_ocr"}).
  • docs/features/catalog-completion-plan.md: replaces the now-stale "OCR ladder exclusion" paragraph feat(image-evidence): make evidence-affecting helpers version-visible + CI totality #690 wrote with a dated entry carrying the full measured evidence above, including the crop-unification negative finding. golden_set.py's collector_line_ocr comment gets a matching re-verification note (2026-08-05); no GOLDEN_EXPECTATIONS value changed.

Stacking

Cut from, and based on, feat/extractor-ownership-visibility (#690) — this PR is stacked on top of it and should be reviewed/merged in that order. The diff shown is scoped to this PR's own changes.

Verification

  • cardpicker/tests/test_image_evidence.py (111 tests, 7 updated for the new 6-attempt/2-tier ladder shape), together with test_golden_set.py + test_run_image_evidence_cohort.py (207 tests combined) — green.
  • cardpicker/tests/test_local_ocr.py + test_ocr_engine_ab.py (65 tests, untouched files) — green.
  • Full backend suite: 3598 passed, 8 skipped, 0 failed.
  • .github/scripts/tests/test_check_extractor_ownership_totality.py + test_check_extractor_manifest_sync.py (40 tests, 4 updated for the lifted exclusion + 3 new) — green.
  • python3 .github/scripts/docs_lint.py — clean. This caught a real finding after the initial push: two backticked citations to the two probe scripts above (catalog-completion-plan.md lines 1921/1945) resolved to paths that don't exist in the repo, since those scripts are deliberately uncommitted. Fixed by rephrasing both citations as prose rather than backticked paths, matching how this repo already cites its other uncommitted probe scripts (e.g. docs/reports/2026-07-23-ocr-preprocessing-probe.md) — the measured evidence itself is unchanged, only the machine-checkable path reference is gone.

…s to 2 (issue #677)

_collector_line_ocr_attempts drops tier 3 (a PSM-11 re-try of tier 1's own
variants) - measured, not guessed. Two fresh real-production probes
(450 forced-escalation cards combined) walked the old 3-tier ladder to
full completion and recorded, per card, the first tier at which a
candidate-validated genuine match appeared: tier 2 produced 2 genuine
matches across the sample, tier 3 produced zero - only more lexicon-valid
-but-uncorroborated noise, consistent with the 2026-07-23 preprocessing
probes' own ~99% hopeless-art-noise finding for this population.
Cross-checked against all 30 golden_set.GOLDEN_CARD_IDS cards: none ever
resolved uniquely at tier 3, so this collapse changes zero golden-set
expectations.

Worst case drops from 8 tesseract calls to 6 (down from #259-#677's own
4x pre-#259 cost to 3x).

collector_line_ocr/collector_line_tsv/artist_ocr extractor versions
bumped v2->v3 (all three read through the same escalation loop, per
EXTRACTOR_OWNERSHIP's own 'bump every listed key together' convention);
legal_line unchanged.

Investigated and explicitly did NOT ship the companion 'unify the
collector-line/legal-line OCR reads into one full-width pass' idea named
in this issue's own brief: a real-image accuracy probe found it regresses
46/150 (30.7%) of currently-correct collector-number reads (mostly proxy/
watermark text getting picked up by the set-code regex in place of the
real code, or the wider crop causing tesseract to drop the collector-
number line outright), with zero improvements. Reported as a legitimate
negative finding rather than shipped anyway.

Lifts the EXTRACTOR_OWNERSHIP EXCLUDED_HELPERS entry for the OCR ladder
(check_extractor_ownership_totality.py) that PR #690 left in place while
this branch was in flight - _collector_line_ocr_attempts and
preprocess_fallback_variants now carry real ownership entries.

Verification: full backend suite (3598 passed, 8 skipped, 0 failed),
ownership-totality + manifest-sync scripts clean with their own unit
tests green (40/40), docs_lint --strict clean, pre-commit clean on all
tracked files.
…'s v3

ARTIST_OCR_EXTRACTOR_VERSION was set to "artist-ocr-v3" on this branch for
issue #677's ladder-collapse dedup, but PR #685 (merged to master
separately) already claims "artist-ocr-v3" for an unrelated change
(wiring modern_artist_credit's crop fallback into live extraction). Two
different behaviours under one version string would defeat the staleness
filter MANIFEST_EXTRACTOR_CURRENT_VERSIONS exists to enforce.

Bumps artist_ocr straight to v4, updates the matching manifest entry in
run_image_evidence_cohort.py, and documents the collision in both the
version constant's own history comment and catalog-completion-plan.md's
#677 section. collector_line_ocr/collector_line_tsv stay at v3 (master
still has both at v2, no collision).

Also fixes golden_set.py's #677 re-verification comment, which cited
MPCAutofill/scripts/experiments/ocr_ladder_tier_attribution.py - a
worktree-only analysis script never committed to this repo, per the
established convention (see docs/reports/2026-07-23-ocr-preprocessing-
probe.md's own LIVE STATE) of not committing one-off probes. Points at
catalog-completion-plan.md's already-committed #677 section instead,
which documents the same 450-card probe's methodology and results.
Both scripts are deliberately uncommitted (worktree-only analysis
tooling, same convention as the 2026-07-23 probe reports). Backticked
citations to their paths tripped docs_lint.py's path-reference rule
since MPCAutofill/scripts/ doesn't exist in the repo. Rephrase as
prose so the provenance stays legible without a machine-checkable
path reference to a file that was never meant to be committed.
@WilfordGrimley
WilfordGrimley merged commit c5eee1c into feat/extractor-ownership-visibility Aug 5, 2026
14 checks passed
WilfordGrimley added a commit that referenced this pull request Aug 5, 2026
…s to 2 (issue #677) (#698) (#702)

* perf(image-evidence): collapse Stage C OCR attempt ladder from 3 tiers to 2 (issue #677)

_collector_line_ocr_attempts drops tier 3 (a PSM-11 re-try of tier 1's own
variants) - measured, not guessed. Two fresh real-production probes
(450 forced-escalation cards combined) walked the old 3-tier ladder to
full completion and recorded, per card, the first tier at which a
candidate-validated genuine match appeared: tier 2 produced 2 genuine
matches across the sample, tier 3 produced zero - only more lexicon-valid
-but-uncorroborated noise, consistent with the 2026-07-23 preprocessing
probes' own ~99% hopeless-art-noise finding for this population.
Cross-checked against all 30 golden_set.GOLDEN_CARD_IDS cards: none ever
resolved uniquely at tier 3, so this collapse changes zero golden-set
expectations.

Worst case drops from 8 tesseract calls to 6 (down from #259-#677's own
4x pre-#259 cost to 3x).

collector_line_ocr/collector_line_tsv/artist_ocr extractor versions
bumped v2->v3 (all three read through the same escalation loop, per
EXTRACTOR_OWNERSHIP's own 'bump every listed key together' convention);
legal_line unchanged.

Investigated and explicitly did NOT ship the companion 'unify the
collector-line/legal-line OCR reads into one full-width pass' idea named
in this issue's own brief: a real-image accuracy probe found it regresses
46/150 (30.7%) of currently-correct collector-number reads (mostly proxy/
watermark text getting picked up by the set-code regex in place of the
real code, or the wider crop causing tesseract to drop the collector-
number line outright), with zero improvements. Reported as a legitimate
negative finding rather than shipped anyway.

Lifts the EXTRACTOR_OWNERSHIP EXCLUDED_HELPERS entry for the OCR ladder
(check_extractor_ownership_totality.py) that PR #690 left in place while
this branch was in flight - _collector_line_ocr_attempts and
preprocess_fallback_variants now carry real ownership entries.

Verification: full backend suite (3598 passed, 8 skipped, 0 failed),
ownership-totality + manifest-sync scripts clean with their own unit
tests green (40/40), docs_lint --strict clean, pre-commit clean on all
tracked files.

* fix(image-evidence): bump artist_ocr to v4 to avoid colliding with #685's v3

ARTIST_OCR_EXTRACTOR_VERSION was set to "artist-ocr-v3" on this branch for
issue #677's ladder-collapse dedup, but PR #685 (merged to master
separately) already claims "artist-ocr-v3" for an unrelated change
(wiring modern_artist_credit's crop fallback into live extraction). Two
different behaviours under one version string would defeat the staleness
filter MANIFEST_EXTRACTOR_CURRENT_VERSIONS exists to enforce.

Bumps artist_ocr straight to v4, updates the matching manifest entry in
run_image_evidence_cohort.py, and documents the collision in both the
version constant's own history comment and catalog-completion-plan.md's
still has both at v2, no collision).

Also fixes golden_set.py's #677 re-verification comment, which cited
MPCAutofill/scripts/experiments/ocr_ladder_tier_attribution.py - a
worktree-only analysis script never committed to this repo, per the
established convention (see docs/reports/2026-07-23-ocr-preprocessing-
probe.md's own LIVE STATE) of not committing one-off probes. Points at
catalog-completion-plan.md's already-committed #677 section instead,
which documents the same 450-card probe's methodology and results.

* fix(docs): cite OCR ladder probe scripts as prose, not backticked paths

Both scripts are deliberately uncommitted (worktree-only analysis
tooling, same convention as the 2026-07-23 probe reports). Backticked
citations to their paths tripped docs_lint.py's path-reference rule
since MPCAutofill/scripts/ doesn't exist in the repo. Rephrase as
prose so the provenance stays legible without a machine-checkable
path reference to a file that was never meant to be committed.

(cherry picked from commit c5eee1c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant