feat(data ingest): preflight parity — the dry-run's promise becomes a checked contract

[LukasWodka]

Summary

P3 of backend#828 — closes cli#69, cli#71, cli#72, cli#73 (+1 rule no ticket had). Stacked on #145 → #142; retargets down the stack as they merge.

Every local check now previews a named data-ingestors validator with matching semantics, so preflight passed ⇒ in-cluster validation passes — failures land before the upload, not after. And the claim isn't taken on faith: a parity harness pins it against the real Python validators.

The previews (internal/push/preflight.go, each cites its source rule)

• dataset push: --label-column is never checked against the CSV header — wrong name writes orphaned rows then fails registration #69 label column exists — LabelColumnValidator semantics (exact, then case-insensitive+trimmed; never stricter than the cluster)
• dataset push: UTF-8 BOM not stripped from CSV header — Excel-exported CSVs get a corrupt first column name #71 BOM — split by what the cluster actually does: tabular rejected pre-upload (the in-cluster stdlib schema probe falsely rejects BOM'd CSVs — filed as data-ingestors#338), image/text accepted + stripped (the pandas paths are BOM-immune)
• dataset push: image pre-flight is too shallow — missing-image refs, undecodable/zero-byte images, and mixed resolutions all pass dry-run #72 every image decoded (header-only): zero-byte, corrupt, resolution-vs-target; labels↔images cross-check with the ingestor's _has_extension naming rules (dotted stems like photo.2024 resolve, not reject)
• dataset push: tabular pre-flight silently collapses duplicate header columns and accepts zero-data-row CSVs #73 duplicate headers (stripped, case-sensitive like the probe), zero data rows — plus gaps the grounding surfaced: --schema columns ⊆ header, and the CSV encoding gate (UTF-8 + no NUL — the first thing the cluster checks)
• label diversity (≥2 classes; NA-sentinel drop + numeric collapse for schema-typed tabular labels; empty string is a class for image/text) — discovered by the harness's own first run; was in no ticket
• object_detection images↔annotations stem pairing

🚨 Fixes a pre-existing ship-blocker

The adversarial pass empirically proved spec.go's [H,W] swap on target_size emission wrong (the schema + ImageResolutionValidator compare PIL's (W,H) verbatim) — every non-square dataset failed in-cluster post-upload, 100%. Emission is now [W,H], pinned end-to-end by the imgc-nonsquare / imgc-nonsquare-swapped parity pair (8×4 images: [8,4] passes both sides, [4,8] rejects both sides — verified against the real validator).

The parity harness (the durable part)

• 23 fixture cases + goldens.json generated from the real Python validators (scripts/gen-validator-goldens.py, sync-validator-goldens.sh --check for verdict-level drift)
• parity_golden_test.go runs push.PreflightDataset — the production dispatch, shared with runDataIngest, so test and reality cannot drift (deleting/rewiring a check fails parity)
• Deliberate divergences (read-/transfer-time failures the ingestor's preflight can't see but the CLI previews) are explicit manifest notes — e.g. imgc-dup-header where the real validators accept and the rejection is read-time-only
• When the ingestor's rules change: regenerate goldens → the Go test fails until manifest + preview are consciously reconciled

Verification

23/23 parity green · full go test ./... green · live dry-runs on a real cluster (missing file / bad label column / single-class / Latin-1 all caught pre-upload; a 320×200 non-square dataset — previously doomed — sails through) · adversarial review (2 lenses): all findings folded, including de-masked fixtures and a vacuous kubeconfig test revived with decodable fixtures.

Refs: backend#828 (P3) · backend#879 · data-ingestors#338.

🤖 Generated with Claude Code

[LukasWodka]

👋 Heads-up — Code review queue is at 35 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#156 — feat(strategies): FedMedian Byzantine-robust aggregation (CLI: a mistyped subcommand (e.g. dataset pus) prints help and exits 0 instead of erroring #75) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#157 — feat(strategies): FedTrimmedAvg Byzantine-robust aggregation (CLI: a mistyped subcommand (e.g. dataset pus) prints help and exits 0 instead of erroring #75) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#158 — feat(strategies): Krum / MultiKrum Byzantine-robust selection (CLI: a mistyped subcommand (e.g. dataset pus) prints help and exits 0 instead of erroring #75) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#159 — feat(strategies): FedProx aggregation strategy (dataset push: --category help lists 5 of 9 categories, and unknown-vs-empty category errors give two different lists (one leaks unsupported segmentation) #74) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#160 — feat(strategies): FedOpt family — FedAdam / FedYogi / FedAdagrad (dataset push: --category help lists 5 of 9 categories, and unknown-vs-empty category errors give two different lists (one leaks unsupported segmentation) #74) · author: @shujaatTracebloc · reviewer: @divyasinghds
• backend#927 — feat(validators): schema check for tabular/text/token classification (#924 G2) · author: @divyasinghds · no reviewer assigned
• backend#931 — feat(validators): generic all-category checks + validation matrix (#924 Phase 0/G12) · author: @divyasinghds · reviewer: @saqlainsyed007
• backend#934 — feat(validators): column-level tabular/text schema diagnostics (#924 G3/G4) · author: @divyasinghds · no reviewer assigned
• backend#935 — feat(validators): time-series & survival temporal checks (#924 G8/G9) · author: @divyasinghds · reviewer: @saqlainsyed007
• backend#936 — feat(validators): text/NLP language + alignable-text checks (#924 G10) · author: @divyasinghds · reviewer: @saqlainsyed007

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)