docs: document the CSV data contract and the repeng consume-not-extract boundary - #36
Open
dchaudhari7177 wants to merge 1 commit into
Open
docs: document the CSV data contract and the repeng consume-not-extract boundary#36dchaudhari7177 wants to merge 1 commit into
dchaudhari7177 wants to merge 1 commit into
Conversation
The sweep CSV schemas, the raw-one-row-per-seed convention and the coherence directions lived only in the report.py module docstring, and the consume-not-extract boundary only in vectors.py, so a contributor writing a sweep producer had to read source to find either. Add docs/data-contract.md covering all three CSVs (exact required columns, which extras are ignored, how the missing-columns error reads), why the producer must not pre-aggregate, the two coherence axes and the coherence thresholds both analyzers apply, and what load_vector/save_vector do and do not accept. Link it from the README Scope section and point both module docstrings at it. Prose only, cross-checked against _SWEEP_COLUMNS, load_side_effects, COHERENCE_DIRECTION, _combine and the analyzer defaults; no invented columns. Closes bamdadd#33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #33.
Adds
docs/data-contract.mdand links it from the README's Scope section. Prose only — no code changes beyond a one-line pointer in each of the two module docstrings that previously held this material.What the page covers
The three CSV schemas, each with a required/ignored column table, a real example row, and the shared rules (header required, extra columns tolerated, missing ones fail before any row is parsed, no missing-value sentinel):
dose_response.csv— one row per (coeff, seed); required{coeff, seed, formality, repetition, ppl},alpha_normignored.layer_sweep.csv— one row per (layer, seed); same tail keyed onlayer, withlayer_pos/dir_norm/resid_norm/coeff/alpha_normignored, and a note that the x column is a parameter.side_effects.csv—benchmark,unsteered_acc,steered_acc, all required; documents that it is optional at the CLI, that an unspecified one becomes a header-only stub while a named one that does not exist is an error, and thatdeltais signed so negative means degraded capability.The raw-per-seed convention, stated as the rule it is ("emit one raw row per seed, do not pre-aggregate") along with what the reader actually computes:
fmeanfor the centre, populationpstdevfor the spread (0.0at one seed),n_seedscounting distinct seed values, grouping by x and sorting by x. It also warns that handing the reader a pre-averaged file is not detected — it silently reads as one sample per x and every error bar collapses to zero. That seemed worth writing down given the 3+ seeds rule inCONTRIBUTING.md.The coherence directions from
COHERENCE_DIRECTION—ppllower-is-better,repetitionhigher-is-worse — plus the fact that no sign convention is imposed on the producer, and the actual thresholds each analyzer applies:analyze_doseatperplexity_tol=0.5relative to the nearest-to-zero baseline,analyze_layersat the deliberately more generous1.0relative to the best perplexity anywhere in the sweep, both withrepetition_cap=0.5, both axes checked because the cliff can appear in either.The repeng boundary, stated plainly: steerbench consumes vectors and does not reimplement extraction. Covers what
load_vectoraccepts (repeng's native.gguf— archcontrolvector, the two KV fields,direction.{layer}tensors — plus a plain.ptfallback), whysave_vectorwrites a superset withsteerbench.concept/steerbench.repeng_version(repeng'simport_ggufignores unknown KV, so files stay natively loadable), and what that means for a contributor: a new concept is prompts and a vector, a new model is a sweep, and PCA over hidden states belongs upstream.Accuracy
Every column, default and threshold was cross-checked against
_SWEEP_COLUMNS,_read_sweep_rows,_combine,_aggregate,load_side_effects,COHERENCE_DIRECTION,analyze_dose,analyze_layers,_ensure_side_csvandvectors.py. The example rows are copied verbatim from the committedartifacts/dose_response.csvrather than invented.One thing the cross-check turned up
Documenting the effect column against the real files surfaced a live drift, so the page carries it as an explicit callout rather than papering over it: the reader hardcodes
formality, but 14 of the 24 sweep CSVs committed inartifacts/name that columneffect— the cross-model sentiment and verbosity runs and the redosed layer sweeps — sosteer-reportcannot render any of them as committed:That is exactly #30, and #34 fixes it. The callout links the issue and flags that this page needs a one-line update when that lands. Say the word if you would rather the page not mention it at all.
Checks
ruff check .,ruff format --check .,mypy src,pytest -qall pass, unchanged (no behaviour touched).