fix(ingest): mirror semseg-mask + missing DataValidator parity in preflight (cli#352)

[LukasWodka]

What

Closes two data ingest preflight parity gaps where a dataset accepts locally then rejects after the full upload (cli#352):

1. Semantic-segmentation mask resolution. data-ingestors composes a second ImageResolutionValidator for semseg — name "Mask Resolution Validator", subdir="masks", same target_size/min_size as the image one (modalities/validators.py:168-173). The CLI only validated images/, so a corrupt or mis-sized mask sailed through preflight and failed in-cluster. Now previewed by a new ValidateMaskResolution over masks/.

2. DataValidator per-value type check. For tabular / time-series schemas the ingestor validates every value against its declared SQL type (validators/data_validator.py); the CLI only checked that the schema's columns exist (CheckSchemaColumns). A non-numeric or fractional value in an INT column, or a non-numeric value in a FLOAT column, passed preflight and rejected in-cluster after the table was created (orphaning it). Now previewed by a new CheckColumnValueTypes.

Why this scope (and what's deliberately left out)

preflight.go's contract is never out-strict the ingestor — an over-reject burns a valid upload. DataValidator is a ~1000-line, ~15-type validator built on pandas to_numeric/to_datetime coercion. Reproducing the string-length, BOOLEAN, and DATE/DATETIME/TIMESTAMP/TIME checks (plus integer overflow and non-finite) in Go risks rejecting values the cluster accepts — the same reason perGroupTimeViolation already leaves the date branch a documented under-preview. So this mirrors only the numeric type mismatch (INT-family non-numeric + non-integer; FLOAT-family non-numeric), which is provably never-over-reject (NA handling matches coercion.NA_SENTINELS; ParseFloat accepts a superset of pandas' numeric grammar; the time-series grouping time column is excluded exactly as the ingestor's factory excludes it). The remaining DataValidator surface is left as a safe under-preview — flagged for a follow-up.

Pin

scripts/.data-ingestors-ref is left unchanged. Both validators already exist at the pinned ref (8f89aec), so this is a mirror gap, not a stale pin — no goldens regeneration is required to fix it. (The pinned commit is behind develop and its comment is out of date; bumping it is a separate, deliberate change that would pull in unrelated upstream drift and needs a full goldens regen against a data-ingestors checkout.)

Test plan

• New unit tests: TestValidateMaskResolution, TestCheckColumnValueTypes (17 sub-cases incl. over-reject guards), TestNumericCellBad.
• Parity corpus +4 cases (drive both the Go preflight and the real Python validators): semseg-mask-res-mismatch (reject), tabular-int-nonnumeric (reject), tabular-int-noninteger (reject), tabular-typed-ok (accept — over-reject guard). TestValidatorParity green across all 71 cases.
• Local gates green: go build, go vet, gofmt -s -l (clean), go test ./..., make lint (errcheck/ineffassign/misspell/staticcheck), make fmt-check, make schema-check, make deadcode, make vulncheck, scripts/file-budget.sh (preflight.go 1650 ≤ 1700 — the new validators live in image_resolution.go + value_types.go).
• goldens-drift runs in CI (needs a data-ingestors checkout); the four added golden verdicts were derived from the validator source at the pin and the existing analog goldens.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes ingest preflight gates on tabular CSVs and semseg datasets; scope is limited to under-reject-safe numeric checks and mask parity with images, with strong unit and golden parity coverage.

Overview
Closes two accept-locally, reject-after-upload preflight gaps (cli#352) so failures surface before the full ingest.

Semantic segmentation masks: Preflight now validates masks/ with the same rules as images—corrupt/empty files, --min-size, and exact target_size—via shared scanImageResolutions and ValidateMaskResolution, wired into the semseg branch of PreflightDataset.

Tabular schema values: After column presence checks, preflight runs CheckColumnValueTypes to mirror the ingestor’s numeric DataValidator slice: non-numeric or fractional values in INT columns and non-numeric values in FLOAT/DOUBLE/etc. String, date, and boolean per-value rules are intentionally not mirrored to avoid over-rejecting valid uploads.

ValidateImages delegates decoding to the shared scanner so image and mask resolution previews stay aligned. Four parity cases plus unit tests cover mismatch rejects and an over-reject guard for valid typed data.

^{Reviewed by Cursor Bugbot for commit b9b0fa3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 31 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#204 — chore(release): promote staging → main (week of 2026-07-20) · author: @aptracebloc · no reviewer assigned
• backend#1131 — perf(psr): offload pause/stop edge notify to Celery (#980) · author: @aptracebloc · reviewer: @saqlainsyed007
• backend#1134 — perf(landing): bulk submission counts + team-member fold (#978 pt5 steps 2-3) · author: @aptracebloc · no reviewer assigned
• backend#1147 — chore(release): promote staging → master (week of 2026-07-20) · author: @aptracebloc · reviewer: @saadqbal
• backend#1148 — fix(sb): always close the failed client in reply_to_edge (Bugbot #1147) · author: @aptracebloc · reviewer: @saqlainsyed007
• backend#1149 — chore(models): remove dead Company model (superseded by Account) · author: @saqlainsyed007 · reviewer: @saadqbal
• backend#1155 — chore(deps): bump pillow from 12.2.0 to 12.3.0 · author: @dependabot · no reviewer assigned
• backend#1156 — chore(hygiene): dead deps, dev-deps split, .dockerignore, dead app/scripts/templates (#959) · author: @saqlainsyed007 · reviewer: @saadqbal
• cli#366 — docs(rfc): RFC-0003 — dataset storage & offboard hygiene (draft) · author: @LukasWodka · no reviewer assigned
• cli#369 — fix(doctor): stop requests-proxy readiness false-green — honest neutral until the egress probe (cli#351) · author: @LukasWodka · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[LukasWodka]

@BugBot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b9b0fa3. Configure here.}

[saadqbal]

/fr-pass