fix(moarstats): harden bivariate header read against joined-CSV page-cache race

[jqnatividad]

Summary

Fixes a flaky CI failure in test_moarstats::moarstats_join_type_left_runs_and_writes_bivariate (example run):

left join must merge primary+secondary numeric columns into bivariate output:
bivariate output is missing expected columns ["value2"] (saw {"id", "value1"});
likely silent join corruption

Root cause

The moarstats --join-inputs --bivariate pipeline shells out to qsv join and qsv stats subprocesses that write temp CSVs, then re-reads them. Under heavy parallel CI load a follow-up read intermittently sees a short/stale view of the joined temp file, dropping a secondary column.

join_datasets_internal and the joined-stats block already guard their reads with sync_subprocess_output + header-coverage validation + a retry. The bivariate header read (csv_headers) was the one link in that chain with no guard — and when a stats-CSV column was missing from that header, the field-pair loop silently continued past it, producing "primary-only" join-corrupt bivariate output.

Change

src/cmd/moarstats.rs — for the joined-input path:

• fsync the joined CSV before reading its header;
• validate the header covers every column the stats subprocess produced records for;
• on a mismatch, log a warning, re-sync, and re-read the header once;
• if a column is still missing, fail loud with a clear "silent join corruption" diagnostic instead of silently emitting primary-only output.

The non-joined path is unchanged (no new failure mode for users with a pre-existing stats cache).

Test plan

• cargo test -F all_features test_moarstats::moarstats_join — 6/6 pass, including the previously-failing test.
• cargo clippy -F all_features --tests — no new warnings in the edited region.
• cargo check --features=lite — compiles clean (the feature set of the failing CI job).
• cargo +nightly fmt — applied via the repo's auto-format hook.

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[Copilot]

Pull request overview

Hardens moarstats --join-inputs --bivariate against intermittent short/stale reads of joined/stats temp CSVs under heavy parallel load, turning previously-silent join corruption into validated retries and (if still inconsistent) a clear failure.

Changes:

• join_datasets_internal now returns the validated final joined CSV header to avoid downstream re-reads of the joined temp file for coverage checks.
• Joined-input stats generation now reads and reuses a single coverage-validated stats CSV snapshot (string) instead of performing later unvalidated re-reads.
• Bivariate header read for joined inputs now fsyncs + validates header coverage against stats fields, retries once, and fails loudly if columns are still missing.