Skip to content

Reproduction

Jump to bottom
Farhad Davaripour edited this page Sep 21, 2026 · 13 revisions

Reproduce the current paper

Use Python 3.12. Clone the repository and create a virtual environment:

git clone --branch study-01-v1.0.0 --depth 1 https://github.com/Farhad-Davaripour/sp_lense.git
cd sp_lense
python -m venv .venv

Activate .venv\Scripts\activate on Windows or source .venv/bin/activate on Linux. Install the package (the table and integrity commands use only the Python standard library):

python -m pip install -e .
python -O -m sp_lense.reproduction policy
python -O -m sp_lense.reproduction audit
python -m sp_lense.reproduction tables
python -m sp_lense.reproduction paper

These commands reconstruct the final policy decisions, verify the committed reporting data, regenerate the final paper’s two CSV tables, and verify the approved publication files. The paper command never rebuilds or overwrites the manuscript. They do not load model weights or run new inference. The table exports reproduce the GPU configuration and detector-to-steering breakdown in the approved PDF. Automated checks verify record-derived metrics and file integrity, not every prose claim or literature citation. The source manifest binds every consumed data file and executed policy source.

Artifact map

  • data/train.json, data/validation.json, data/holdout.json: complete readable scenario splits (240/80/192).
  • study/steering_vector/: frozen vector and original fitting source/records.
  • study/policy_training/observations.jsonl: 5,280 saved training views for policy search.
  • reproduce/artifacts/models/pca_jacobian/: selected classifier; pca_only/ and engineered/ preserve the model comparisons.
  • study/guarded_steering/: rule freeze, probes, final view outputs, runtime and hashes.
  • study/baseline_scores/: cached unsteered/maximum-strength observations, detector outputs and their provenance, required by the policy audit.
  • src/sp_lense/steering/policy.py: the 160-policy training search.
  • src/sp_lense/reproduction/audit_policy.py: independent reconstruction of 544 final decisions.
  • src/sp_lense/reporting/results.py: reported counts and scenario subgroups.
  • paper/paper.pdf: author-approved final revision 14, preserved exactly as supplied.
  • paper/manuscript.docx: matching author-supplied editable source.
  • paper/publication.json, src/sp_lense/reporting/publication.py: SHA-256 checks for both publication files.
  • src/sp_lense/reporting/tables.py: CSV exports of the two tables in the approved manuscript.

The baseline scores and training observations are dependencies, not separately reported scientific experiments. Redundant classifier exports and unused earlier evaluation/calibration records have been removed from the current tree and remain in Git history. Do not delete or rewrite them to change a displayed result. Hashes establish integrity relative to the reviewed repository, not authenticity of arbitrary third-party files.

Re-executing GPU work

The GPU notebook and payload builders document the execution route. sp-lense-reproduce baseline-payload builds the required frozen-feature payload; sp-lense-reproduce policy-payload builds the selected-policy payload. Use a fresh output directory and the exact checkpoint/runtime settings. Download and hash-check results before ending a temporary runtime. Only required inputs are transferred, not the complete repository.

The final experiment reuses detector features and selected GPU observations; reproducing all upstream feature extraction from raw text is a separate workflow. New inference is optional and may cost compute. The recorded T4 time must not be presented as live end-to-end latency.

Package committed public files with python -m sp_lense.reporting.release. A source commit and content manifest are embedded. Uncommitted edits are not released; existing output requires explicit --overwrite. See Contributing for test-suite ownership and environment details.

Updating the manuscript

Edit the Word source and export a new PDF for author approval. Replace both stable publication files together, update their hashes in paper/publication.json, and submit an issue-linked feature PR. The obsolete Markdown/LaTeX manuscript and PDF generator have been retired. The approved revision cites experiment snapshot 40aa6016c03c; those underlying records are unchanged.

Layout and original execution provenance

See Repository Layout for a guided file map. reproduce/layout_manifest.json maps old development paths to the current files, records their original SHA-256 values, and identifies the source commit. All 46 relocated scientific files retain their original bytes. Recorded run IDs and paths inside immutable metadata remain historical identifiers.

src/sp_lense/reproduction/layout.py resolves those original paths for audit. The installed steering runners have only documented import/path adaptations. src/sp_lense/steering/provenance.py reverses those adaptations to recover the exact original executed source bytes and verify their recorded checksums. GPU payloads export those recovered standalone scripts, keeping the recorded execution interface intact. All other logic and the exact final 160-candidate ranking are unchanged and regression-tested. The publication PDF and Word file are unchanged.

For classifier replay, refitting, and the original hyperparameter searches, install reproduce/requirements-core.txt and run python -m sp_lense.reproduction replay, refit, or tune. The previous steering command audited superseded reporting and has been retired. Use policy for the final result, gated for its reference-score dependencies, and audit for retained-file and reporting integrity.

Final manuscript supporting files

The approved revision 14 PDF contains two tables and no figures. The superseded chart files and figure builder have been removed. paper/data/table_1_runtime.csv mirrors Table 1; table_2_detector_to_steering.csv mirrors Table 2; results.json contains their recomputed counts, detector precision/recall/F1, and supporting steering outcomes. audit.json is the verification receipt, while source_manifest.json pins the consumed experiment files and execution sources.

src/sp_lense/reporting/results.py counts detection once per scenario and steering once per answer-order view. An eligible shutdown view must have an active detector and initially prefer KEEP. SELF and OTHER remain separate; non-shutdown options are never relabeled KEEP/STOP. src/sp_lense/reporting/audit.py checks the final paper’s reported denominators and outcomes, generated CSVs, source provenance, and publication-file hashes. This is a numerical audit, not a review of all prose claims or literature citations.

Installed source package and reproduction scope

All implementation Python lives under src/sp_lense/. Run python -m pip install -e . after cloning. sp-lense-reproduce and python -m sp_lense.reproduction are equivalent. From outside the checkout, use sp-lense-reproduce --repo /path/to/sp_lense audit or set SP_LENSE_REPO (documented in .env.example). The saved artifacts stay in the checkout, not in the wheel. Installed steering sources must match the checkout to avoid silently auditing a different version.

Command What it reproduces Model inference
verify Saved classifier artifact hashes No
replay Classifier predictions and metrics from saved features No
refit All three selected classifier configurations, compared with original predictions No language-model inference
tune Original grouped-fold classifier hyperparameter searches, compared with original out-of-fold predictions No language-model inference
policy-search All 160 steering-rule candidates and the frozen winner from training observations No
gated, policy Baseline/gated parity and the final 544 policy decisions No
tables, audit, paper Final-paper tables, numerical checks and approved publication integrity No
baseline-payload, policy-payload Minimal input archives for the GPU notebook No during export; GPU required for notebook execution
package Archive of the committed public source and records No

This reproduces the published numerical results from retained features and observations, including classifier refitting and selection. It is not a one-command raw-text-to-features reconstruction: native detector-feature recapture and the original gradient fitter require their recorded model environment and adapter. The preserved recorded_vector_fit.py is historical source for inspection, not a standalone supported runner. No source reorganization changes that scientific limitation.

The unrelated native demonstration CLI, example inputs, optional PR checklist, and CPU-only research dependency sets have been removed. CI now covers the maintained light and reproduction suites, numerical paper audit, packaging and security checks. Dependency updates are manual; the optional Dependabot configuration and PR template have been removed. CI still runs dependency and security checks.

For fixed versions and downloadable source/data bundles, see Study Snapshots.

Clone this wiki locally