Skip to content

Test image & detection architecture overhaul #1219

@colefoster

Description

@colefoster

Problem

The current detector/reader testing system has fundamental limitations:

  1. Labels in filenames — can't encode multi-dimensional ground truth (species + detector bools + sprites on one image)
  2. 1 image = 1 detector directory — prevents reuse across detectors/readers, forces duplication
  3. No comprehensive screen model — detectors added ad-hoc, no shared screen graph for auto-ladder state machine
  4. Re-labeling is destructive — new detector/reader means re-labeling everything from scratch

Solution

Replace with a manifest-based, screen-centric architecture:

  • screens.yaml — canonical screen graph defining all 13 screen states, transitions, detector/reader registration with typed field schemas
  • Per-screen image directories with manifest.json storing all reader labels
  • Implicit negative testing — images not in a detector's registered screens are automatic negatives
  • Web-based labeler on ash dashboard with OCR pre-population via ColePC job runner API

Key Decisions

Decision Choice
Image storage Per-screen directories (13 screens + 1 overlay)
Labels manifest.json per directory
Negative tests Implicit from directory membership
Screen graph screens.yaml with transitions, detectors, readers, field schemas
Singles/doubles Separate directories for in-battle screens only
C++ test runner Add yaml-cpp + nlohmann/json, parse new structure
Labeler Web UI on ash dashboard
OCR suggestions Job runner API on ColePC (port 8422)
Migration Rip and replace CommandLineTests/

Full PRD

See plans/test_image_architecture.md for complete specification including:

  • Screen inventory and state machine transitions
  • screens.yaml and manifest.json schemas
  • Directory structure
  • Negative test model
  • C++ test runner changes
  • Web labeler design (inbox, screen, validation views)
  • OCR suggestion API spec

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions