Skip to content

docs: document the CSV data contract and the repeng consume-not-extract boundary - #36

Open
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:docs/data-contract
Open

docs: document the CSV data contract and the repeng consume-not-extract boundary#36
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:docs/data-contract

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #33.

Adds docs/data-contract.md and links it from the README's Scope section. Prose only — no code changes beyond a one-line pointer in each of the two module docstrings that previously held this material.

What the page covers

The three CSV schemas, each with a required/ignored column table, a real example row, and the shared rules (header required, extra columns tolerated, missing ones fail before any row is parsed, no missing-value sentinel):

  • dose_response.csv — one row per (coeff, seed); required {coeff, seed, formality, repetition, ppl}, alpha_norm ignored.
  • layer_sweep.csv — one row per (layer, seed); same tail keyed on layer, with layer_pos / dir_norm / resid_norm / coeff / alpha_norm ignored, and a note that the x column is a parameter.
  • side_effects.csvbenchmark,unsteered_acc,steered_acc, all required; documents that it is optional at the CLI, that an unspecified one becomes a header-only stub while a named one that does not exist is an error, and that delta is signed so negative means degraded capability.

The raw-per-seed convention, stated as the rule it is ("emit one raw row per seed, do not pre-aggregate") along with what the reader actually computes: fmean for the centre, population pstdev for the spread (0.0 at one seed), n_seeds counting distinct seed values, grouping by x and sorting by x. It also warns that handing the reader a pre-averaged file is not detected — it silently reads as one sample per x and every error bar collapses to zero. That seemed worth writing down given the 3+ seeds rule in CONTRIBUTING.md.

The coherence directions from COHERENCE_DIRECTIONppl lower-is-better, repetition higher-is-worse — plus the fact that no sign convention is imposed on the producer, and the actual thresholds each analyzer applies: analyze_dose at perplexity_tol=0.5 relative to the nearest-to-zero baseline, analyze_layers at the deliberately more generous 1.0 relative to the best perplexity anywhere in the sweep, both with repetition_cap=0.5, both axes checked because the cliff can appear in either.

The repeng boundary, stated plainly: steerbench consumes vectors and does not reimplement extraction. Covers what load_vector accepts (repeng's native .gguf — arch controlvector, the two KV fields, direction.{layer} tensors — plus a plain .pt fallback), why save_vector writes a superset with steerbench.concept / steerbench.repeng_version (repeng's import_gguf ignores unknown KV, so files stay natively loadable), and what that means for a contributor: a new concept is prompts and a vector, a new model is a sweep, and PCA over hidden states belongs upstream.

Accuracy

Every column, default and threshold was cross-checked against _SWEEP_COLUMNS, _read_sweep_rows, _combine, _aggregate, load_side_effects, COHERENCE_DIRECTION, analyze_dose, analyze_layers, _ensure_side_csv and vectors.py. The example rows are copied verbatim from the committed artifacts/dose_response.csv rather than invented.

One thing the cross-check turned up

Documenting the effect column against the real files surfaced a live drift, so the page carries it as an explicit callout rather than papering over it: the reader hardcodes formality, but 14 of the 24 sweep CSVs committed in artifacts/ name that column effect — the cross-model sentiment and verbosity runs and the redosed layer sweeps — so steer-report cannot render any of them as committed:

ValueError: artifacts/dose_response_sentiment_qwen.csv is missing columns: ['formality']

That is exactly #30, and #34 fixes it. The callout links the issue and flags that this page needs a one-line update when that lands. Say the word if you would rather the page not mention it at all.

Checks

ruff check ., ruff format --check ., mypy src, pytest -q all pass, unchanged (no behaviour touched).

The sweep CSV schemas, the raw-one-row-per-seed convention and the coherence
directions lived only in the report.py module docstring, and the
consume-not-extract boundary only in vectors.py, so a contributor writing a
sweep producer had to read source to find either.

Add docs/data-contract.md covering all three CSVs (exact required columns,
which extras are ignored, how the missing-columns error reads), why the
producer must not pre-aggregate, the two coherence axes and the coherence
thresholds both analyzers apply, and what load_vector/save_vector do and do not
accept. Link it from the README Scope section and point both module docstrings
at it.

Prose only, cross-checked against _SWEEP_COLUMNS, load_side_effects,
COHERENCE_DIRECTION, _combine and the analyzer defaults; no invented columns.

Closes bamdadd#33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document the CSV data contract and the repeng consume-not-extract boundary

1 participant