W5: streaming bulk — illustration calc, version-aware resume, bulk-scale test - #511
Merged
Conversation
Phase 1: Import Scryfall illustration_id into CanonicalPrintingMetadata - Add illustration_id UUIDField to PrintingMetadataRow and model - Add resolved_illustration_id property for single/double-face parsing - Migration 0089 for the new field Phase 2: Stage D illustration deduction calculator (issue #507) - IllustrationIndex: in-memory (artist_pk, card_name) → illustration_id → printing_pks - calculate_illustration_verdict: 0→abstain, 1→vote@0.85, N>1→union@0.85/N - run_illustration_calculator: batch runner with eligibility, gate, purgeability - Wired into management command between fallback and slow-path - Tests for index, verdict logic, integration, dry-run, gate check, purgeability
Source bug: - IllustrationIndex.artist_by_pk was keyed by artist_pk but match_artist expects card_pk keys; separated into a dedicated query over ALL CanonicalCards so artists are matchable even without illustration metadata Test bugs: - CanonicalArtistFactory.objects.get() — factory class, not a model - candidates=candidate (bare) instead of candidates=[candidate] - _join_key_no_hit_card set both printing and is_no_match=True, violating cardprintingtag_printing_xor_no_match constraint - _eligible_card had content_phash=None, causing all cards to be skipped - purge_machine_votes called with unsupported interactive kwarg
…ale test, docs
- Wire run_illustration_calculator into stage_e_dispatch._run_stage_d
(after fallback, before slow-path; lazy import, card_ids-scoped)
- Add MANIFEST_EXTRACTOR_CURRENT_VERSIONS single-source version map in
run_image_evidence_cohort.py; fix resume filters in both callers to
use __contains (version-aware) instead of __has_keys (presence-only)
- Add bulk-scale (1200-card) selection test verifying version-aware
resume filter re-processes stale versions and skips current ones
- Add Bulk run preconditions subsection to stage-e-operations.md
- Update all test helpers to use real version strings instead of
synthetic {key-v1} placeholders
- Idempotency audit: confirmed sound (no code change needed)
…w5-streaming-bulk
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the illustration deduction calculator (#507) into the Stage E streaming conveyor and makes the resume filter version-aware, preventing stale-version rows from being silently skipped.
Stacked on #509 (illustration-id-deduction).
Changes
stage_e_dispatch._run_stage_d):run_illustration_calculatorcalled after fallback, before slow-path; lazy import,card_ids-scoped,dry_run=False. Ledger counters:stage_d_illustration_votes,stage_d_illustration_already_voted.MANIFEST_EXTRACTOR_CURRENT_VERSIONSdict (single-source, inrun_image_evidence_cohort.py) maps each extractor key to its current version string. Both callers (_select_micro_batchand_run_stage_cinstage_e_dispatch.py, plusrun_image_evidence_cohort's own bulk filter) now use__contains(PostgreSQL@>) instead of__has_keys, so a card with old version tags is re-processed rather than skipped.docs/features/stage-e-operations.md: streaming disabled/paused, no open trip, version-aware filter explanation, git SHA, illustration calculator.created=True), evidence-change echo suppression prevents dispatch-internal cascades, illustration calculator writesPrintingrows (notImageEvidence), no new echo path.Test plan
pytest cardpicker/tests/test_stage_e_dispatch.py— 52 passedpytest cardpicker/tests/test_evidence_transfer.py cardpicker/tests/test_stage_e_shakedown.py cardpicker/tests/test_run_image_evidence_cohort.py— 93 passed