Skip to content

#560: port pec_csv to the two-stage design (canonical rename in parse)#605

Merged
jepegit merged 1 commit into
issue-560-parse-pec-customfrom
issue-560-parse-pec
Jul 21, 2026
Merged

#560: port pec_csv to the two-stage design (canonical rename in parse)#605
jepegit merged 1 commit into
issue-560-parse-pec-customfrom
issue-560-parse-pec

Conversation

@jepegit

@jepegit jepegit commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Part of #560. Stacked on #602 — review that first; this diff is against its branch.

What

pec_csv gets parse() and declarations() beside its existing loader(), which is untouched. It was the one loader that couldn't be handled mechanically: a BaseLoader (no config_params), columns identified by alias set rather than fixed strings, and a different unit per column embedded in the header (Voltage (V) vs (mV)).

Design decisions and their rationale are in the plan (§2.6a); the pivotal one — canonical rename in parse() → static declarations — was a maintainer call.

How

  • parse() reads → matches alias sets → renames matched columns to canonical vendor-neutral names (voltage, current, …) → scales each to its canonical unit → shifts 0-based cycles. The unit scaling is not optional: harmonize() has no per-column unit scaling, so an unscaled mV column would be tagged as V. Silent corruption — and the parity oracle catches it (a mutation removing the scaling shows a 3795 mV diff on potential).
  • declarations() does not hand-transcribe the canonical→native map. It reuses derive_column_maps by inverting _COLUMN_KEY_TO_CELLPY_HEADER and composing with cellpy-core's legacy→native map — so pec inherits the provenance rule (vendor test dropped, not mapped onto the framework test_id) and the 0.2.3 energy mapping from the same code the config loaders use. Only the passthrough set is per file: the export's other columns (temperatures, OCV, peak power) are preserved under sanitised names, matching loader().

A false negative it fixed in the oracle

pec carries sparse per-measurement columns that are all-null in the fixture (cell_id, station_temperature_degc, the internal resistances). The numeric diff of two all-null columns is None, which the oracle had been counting as a mismatch — and which would equally have hidden a real values-vs-null difference. The comparison now checks null masks first, then non-null values. Mutation-checked both ways: the neware PER_STEP mutant still fails; an all-null-vs-all-null column agrees.

Result

  • pec reaches value parity on every mapped and passthrough column; only exclusion is the shared date_time representation gap.
  • Five parity cases now (maccor ×2, neware, custom, pec).
  • tests/test_pec_two_stage.py pins the decisions directly.
  • Full suite under PYTHONUTF8=1: 1205 passed, 0 failed.

Next on #560

arbin_res (ODBC — CI can only check the committed fixture) and the tier-2 arbin_sql family, then the switchover. Config-driven maccor_txt_zero and batmo_bdf_bdf remain without parity coverage.

🤖 Generated with Claude Code

pec_csv is not configuration-driven - it identifies columns by alias set and
carries a different unit per column in the header - so its parse()/declarations()
are hand-written rather than derived. loader() is untouched; this adds the two
stages beside it.

parse() reads the file, matches the alias sets, renames matched columns to
canonical vendor-neutral names (voltage, current, ...), and scales each column
to its canonical unit. That last step is not optional: harmonize() has no
per-column unit scaling, so leaving a mV column unscaled would tag it as V -
silent corruption. The parity oracle catches exactly that (a mutation removing
the scaling shows a 3795 mV diff on potential).

declarations() does not hand-transcribe the canonical->native map. It reuses
derive_column_maps by inverting _COLUMN_KEY_TO_CELLPY_HEADER and composing with
cellpy-core's legacy->native map, so pec gets the provenance rule (vendor `test`
dropped rather than mapped onto the framework test_id) and the 0.2.3 energy
mapping for free - the same code the config loaders use. Only the passthrough
set is per file: the export's other columns (temperatures, OCV, peak power) are
kept under their sanitised names, matching loader().

Building it surfaced a false negative in the parity oracle, now fixed: pec
carries sparse per-measurement columns that are all-null in the fixture, and the
numeric diff of two all-null columns is None, which the oracle counted as a
mismatch. That rule would also have hidden a real values-vs-null difference. The
comparison now checks null masks first, then non-null values. Mutation-checked
both directions (the neware PER_STEP mutant still fails; an all-null column
agrees).

pec reaches value parity on every mapped and passthrough column; the only
exclusion is the shared date_time representation gap.

tests/test_pec_two_stage.py pins the decisions (units normalised in parse, static
canonical->native map, passthroughs preserved, provenance dropped).
Full suite under PYTHONUTF8=1: 1205 passed, 0 failed.

Loader plan §2.6a updated with the execution record.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 20, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@jepegit
jepegit merged commit 51eced2 into issue-560-parse-pec-custom Jul 21, 2026
1 check passed
@jepegit
jepegit deleted the issue-560-parse-pec branch July 21, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant