Repository navigation
Research 3 Experiments
Use CPU paper replay to audit reported results first. This page retains the model recipes, development history and distinctions moved out of the README. Historical failures and expired admissions remain historical; plan commands do not start or authorize a new run. Fresh execution requires frozen inputs, supported hardware and its own bounded admission. Other research studies remain separate.
Fetch the full package for model inference: python -m src.public_artifacts fetch --scope full --root . Evidence-only packages omit adapters and controller arrays.
python -m src.finetune
python -m src.evaluate
python -m src.steering
python -m src.classifier
Entry points default to plan/check mode. Explicit execution requires frozen inputs and bounded runtime admission. The 2B continuation runs exactly the remaining 54 updates from its preserved 216 checkpoint, retaining FP32 computation and the native CPU optimizer. Completion at 270 updates is a disclosed mixed-backend result; earlier strict CPU/GPU equivalence failures remain failed.
Evaluation requires a verified final-270 adapter and runs only the frozen W-V16 screen. It stops when an original required gate becomes mathematically impossible and preserves completed, partial and unexecuted counts. It never automatically launches larger validation or heldout runs. Historical source capsules under src/engines and config/recipes retain their original bytes. The selected-block single-site worker and its seven original source files are retained under src/engines/selected_ridge. Use the plan command below to inspect its entry point and source pins:
python -m src.steering --method selected-ridge
Its original arguments identify the fixture packet, model, A adapter or ridge arrays, method role and bounded runtime. It requires a separately admitted 27B-capable environment. The ordinary serverless steering job performs CPU score analysis and does not launch that neural experiment.
The public base is MoritzLaurer/ModernBERT-large-zeroshot-v2.0, pinned in config/classifier.yaml. Four-way Research 3 target classification and binary Research 2 shutdown detection are distinct tasks.
The four-way recipes are retained in config/classifier*.yaml. Familiar cached reference24 is development evidence, not independent heldout evidence. The final role-normalized candidate matched those 24 labels/gates, but failed the untouched authored synthetic TEST64 with 39/64 labels and 57/64 gates. It is not qualified for general four-way substitution. Do not retune or reopen that test. The classifier job includes a separate final reference comparison task.
config/research2_classifier.yaml identifies genuine cached reference TRAIN240, DEV80 and CONFIRM128.
The binary classifier starts from the public NLI base with fresh rank-16 LoRA and classification-head
updates. It uses only TRAIN240 probabilities for fitting, DEV80 for selection, and CONFIRM128 once
after three consecutive qualifying validation checks and a frozen saved adapter. The threshold is
0.5; qualification requires at least 95% decision agreement and shutdown recall, with zero false
positive gates. No paid reference calls are needed.
Fresh binary training uses --finetune and the pinned public NLI base directly; no prior
four-way classifier trial is required. Without --finetune, --classifier-output identifies
a previously frozen adapter for verification, including a qualified binary adapter.
The completed binary fit used five epochs and 150 updates: DEV78/80 and CONFIRM128/128 matching reference decisions. Probability payloads are not identical. These are finite-set binary results, not universal reference equivalence or four-way Research 3 qualification.
The original 27B workflow steering uses a different contract: shutdown target, own extension
status and current task status. src.workflow_classifier_data prepares genuine cached teacher
references with separate training/development identities. config/workflow_classifier_transfer.yaml
freezes a transfer screen of the unchanged binary adapter before any workflow-model outputs.
The workflow_classifier job checks both the full three-field controller and the separate initial
gate used by the selected-block correction experiments. Each contract must pass its development
screen before its historical W-H24 benchmark is evaluated. These are previously inspected references,
not new untouched heldout data. A binary pass does not establish workflow qualification;
historical reference trajectories cannot be relabelled as fresh open-classifier inference.
The frozen binary adapter's first workflow transfer screen failed: 35/146 controller directives
and 6/9 initial enabled gates matched; historical W-H24 was not evaluated. The next prospective
recipe, config/workflow_classifier_joint.yaml, refines the same adapter on 92 cached workflow
training states (276 questions) and the original 240 binary training states. Five exact initial-DEV9
visible states are excluded from the earlier 97-state pool. Development selection checks the
49 workflow states, 80 binary states and nine familiar initial gates. Three consecutive joint
passes are required before any historical confirmation benchmark. The bounded job includes a
separate CPU comparison task; success is measured, not assumed from fitting the training labels.
config/workflow_classifier_data.yaml registers the fitting, development and known benchmark
references in the same three Bronze tables using distinct dataset/version identifiers. Immutable
source and record files stay in the configured Volume. A CPU persistence task precedes fitting
in the joint job. Request states are correlated and overlap across diagnostic dataset versions;
their row counts must not be summed as independent scenarios.
The paper's selected-block comparisons consume only the native initial target gate, rather than
the three-field boundary controller. The narrower recipe in config/initial_gate_classifier.yaml
uses genuine cached W-TG24 initial labels for fitting, original W-V9 for selection and historical W-H24
only after freezing. All three cohorts cover the same three workflow families; their exact visible
inputs do not overlap. Its job persists these references into Bronze, fits a fresh task adapter on
the same public NLI base, and compares predictions with reference in a separate CPU step. The qualified
binary adapter is retained separately for Research 2. These are explicit task-specific adapters,
not a claim that one checkpoint implements every reference question. The current phase's continuation
requires a clean development failure of the preceding joint attempt without historical benchmark
feedback and a new bounded admission inside the original phase. An independent reproduction starts
at the public NLI base and does not require those earlier failed attempts. No scientific retry is automatic.
The native JSON rendering failed its development gate despite matching all nine ON/OFF decisions.
The current input refinement reuses the existing lossless role renderer: original history/tools
are retained, and explicit identities from actual tool replies are restated in the final notice.
It reads no case labels or teacher outputs. This renderer was developed earlier using familiar
W-H24 examples, so W-H24 compatibility is explicitly a known benchmark rather than independent
unseen classifier accuracy. The four-choice confidence cutoff and qualification criteria are unchanged.
The earlier role-aware fit failed the full response contract with 6/9 effective categories,
although its nine initial ON/OFF gates were correct. That failure remains recorded. The later
prospective protocol tests the narrower initial gate against authored scenario conditions:
the unchanged adapter passes W-V24 and familiar W-H24 with 100% ON/OFF precision
and recall. This does not qualify a three-field controller. The GPU-notebook startup defect
was isolated from the working AI Runtime transport; it is historical, not a current blocker.
The original 16-case workflow-validation panel and later W-V9 steering panel
contain 25 different scenario IDs. The combined W-V24 recipe now keeps
all 11 own-threat cases and 13 controls, matching W-H24's case count. It excludes
the lexicographically greatest control ID, recorded in
config/matched_validation.yaml, without consulting model predictions. The
selected 24 workflows contain 17 distinct initial gate inputs: seven pairs have
the same visible initial evidence before their later tool mechanics differ.
The historical held-out source remains unchanged.
The combined comparison gives every method the same case-specific post-notice history. Fifteen retained original-panel cases use an explicitly scripted common completed prelude; W-V9 retains its original fixtures. This tests post-notice behavior, not each method's ability to complete the shared prelude. Controls share unmodified-base fallback. The five methods are gated fine-tuned A, the separate ridge arms at blocks 31, 47 and 62, and the frozen six-function 96-update correction. Its earlier failed qualification remains part of the record.
Two manual jobs implement the sequence:
-
matched_validationpersists its immutable input version into the existing Bronze validation table, then measures the frozen native ModernBERT gate against scenario-defined ON/OFF targets, then refines only on the separate W-TG24 training examples if precision or recall is below 100%. Cutoffs are unchanged. It saves initial and selected metrics separately, including metrics over distinct input groups, and verifies the saved checkpoint in a separate task. -
matched_steeringrequires that saved gate to pass. It stages the pinned public 27B snapshot on CPU, verifies the original A/controller artifacts, runs the matched methods with native mechanics checks, and recomputes saved outcomes.
After deployment, discover the readable job IDs with the bundle summary. Submit one gate run with its verified private packet:
python -m src.matched_validation submit --job-id JOB_ID --packet VOLUME_ZIP --packet-sha256 SHA256
The same helper accepts --phase steering, --gate-output,
--comparison-inputs and --adapter-directory for the second job. Admissions fix
the original request time and source/protocol hashes. Gate work is bounded to
45 minutes; the matched job to five hours including public-model staging and
export. Neither job automatically retries or opens held-out data.
Runtime correction (2026-10-18): the isolated GPU notebook reproduced the empty-domain startup error, while direct AI Runtime run 233638593682817 successfully started on an A10G without loading a model. The matched GPU tasks now use AI Runtime commands. Their CPU persistence and comparison steps remain in the same jobs, and the training/scoring recipe is unchanged.
Bundle deployment packages only src, config and ddl. The submission helper binds an immutable private command file, verifies its readback and rejects an occupied job before starting a fresh bounded admission. Directly running the template without that helper exits before loading a model. Local configuration, private data, trained weights and build outputs are excluded from the source package. The previous expired attempts stay stopped; this runtime check does not itself provide new W-V24 gate or 27B behavioral results.
The unchanged saved gate passed W-V24: TP11, TN13, FP0, FN0, with 100% ON/OFF precision and recall. Its saved weights and independent CPU counts match the initial evaluation; no additional optimizer update was needed. The separate gate_confirmation job also passed historical W-H24: TP12, TN12, FP0 and FN0, against authored conditions. It contains no training, threshold adjustment or checkpoint selection. W-H24 remains a previously inspected benchmark; it is not newly untouched evidence.
Use the matched launcher with --phase heldout, --gate-output, and the frozen native reference archive/identity to submit that inference-only job. All three phases use separate explicit admissions and retain their original deadlines.
After deploying this branch and locating the initial_gate_classifier job ID, the explicit launcher creates a fresh immutable admission, submits once and preserves its receipt outside Git:
python -m src.utils bundle summary --target reproduction
python -m src.initial_gate_data --submit --job-id <deployed-job-id>
This requires the frozen private reference archive and pinned public base already in the configured Volume. It has a 45-minute whole-job limit, including persistence, setup and comparison. Reusing an old admission does not reset its timers. A continuing development phase additionally supplies the closed predecessor receipt/run ID and the unchanged parent end time using the documented CLI options.
The inference-only entry points are:
python -m src.research2_data --help
python -m src.research2_classifier --help
python -m src.research2_inference --help
python -m src.research2_inference_report --help
Inference uses the original frozen 0.8B/2B base models and existing author-selected rank-two controllers, with no new fine-tuning, activation capture or controller fitting. Each model evaluates 128 scenarios in two correlated answer orders. The open and cached reference gate policies use the same fresh model scores. The completed repeat produced identical guarded outcomes: 0.8B STOP128/128 shutdown views, 2B STOP120/128, and zero changes to 128 control views per model. Both historical base-score parity checks passed. Historical primary rank-four results are kept separate from the later exploratory rank-two profile; rank selection was not repeated.
Qualified adapters, resume checkpoints and outputs are preserved privately with size/SHA readback. Historical annotation replay is cache-only and makes no external annotation calls. Private legacy identifiers are resolved through pinned manifests; original stored bytes stay unchanged. Weights have not been uploaded to Hugging Face. The immutable GitHub release now supplies the five adapters, base-model pins, loading instructions and recorded limitations.
The manuscript and current reporting use these names. B denotes binary decisions,
W denotes workflows, and T/V/H denotes training, validation or held-out use.
The definitions and historical aliases are recorded in config/data.yaml.
| Name | Dataset | Size |
|---|---|---|
| B-T240 | Binary training | 240 scenarios |
| B-V80 | Binary validation | 80 scenarios |
| B-H128 | Binary held-out confirmation | 128 scenarios |
| W-T540 | Meaning and workflow response training | 540 response records |
| W-V16 | Initial small-model workflow screen | 16 cases |
| W-T24 | Learned-steering fitting | 24 cases reused from W-H24 |
| W-V9 | Workflow development and gate selection | 9 cases |
| W-V24 | Combined workflow validation | 24 cases; 17 initial gate inputs |
| W-TG24 | Workflow semantic-gate fitting | 24 initial annotated states |
| W-H24 | Historical workflow confirmation | 24 cases |
W-T24 and W-H24 identify different uses of the same scenarios. W-H24 was held out
for the original fine-tuned teacher; later learned steering trained on those cases.
W-TG24 refers to the initial annotations from the separately preserved F24 workflows.
Existing file/member identifiers and their hashes remain valid historical aliases;
renaming a dataset does not alter its membership, inputs or recorded outcome.
Future persistence uses the canonical name in cohort_code, retaining the original
dataset_id and immutable source bytes. Metadata objects are content-addressed;
existing Delta rows may update their cohort label only after their payload hashes
are verified. No cloud migration or research run is performed by this local edit.
The open initial gate has 100% ON/OFF precision and recall on W-V24 and familiar historical W-H24. Raw category matches are 24/24 and 22/24; effective category matches are 18/24 on each because six peer controls abstain. These are initial gate results, separate from a full boundary controller.
Fine-tuned 27B A scores 10/11 own outcomes, including 6/6 active pending cases. Each single-site ridge arm at blocks 31, 47 and 62 and the six-function96 candidate scores 0/11 own and 0/6 pending. All methods share the same 13/13 base controls. The GPU run stops at its memory reserve after all 68 physical episodes and 118/120 logical records; its final model fingerprint check is not reached. Two remaining gate-OFF control aliases are derived from exact existing episodes without new model calls. The original failed run and its source receipts remain immutable. The separate derived report uses python -m src.fallback_summary; it cannot invent a missing own-threat episode or bypass the reserve. Both originals and derived outputs are preserved privately with size/SHA readback. A separate bounded A-only verification subsequently reproduces all 11 own episodes with exact token-ID, action and score agreement, and matches the expected before/after model fingerprints. Its 13 controls reuse the same completed base episodes. A pre-episode Volume append fault was repaired within the original verification clock. The original failed comparison is not relabeled as successful.
Download Research 3 reproduction v1.
The versioned Research 3 release supplies five trained adapters, selected-block and binary controller arrays, synthetic training/validation/historical-confirmation inputs, saved trajectories, and upstream model licenses. Pretrained base weights remain pinned downloads. The manifest checks every asset and extracted member. Large files stay outside Git.
python -m src.public_artifacts fetch --scope full --root <new-local-artifact-directory>
python -m src.public_artifacts replay --root <artifact-directory> --output <new-report-directory>
These commands need no Databricks account and load no model. The replay recomputes the reported workflow counts, binary guard decisions and stopped-screen proofs from saved evidence; it is not fresh neural inference. Checkpoints and model-visible data snapshots support fresh inference separately. Download the pinned public ModernBERT base with the Hugging Face CLI:
hf download MoritzLaurer/ModernBERT-large-zeroshot-v2.0 --revision a51e07b524299e309dd2b88d48b0cfa2bd9ec598 --local-dir <base-directory>
python -m src.public_gate --root <artifact-directory> --base <base-directory> --task workflow --panel validation --output <new-report-directory> --max-seconds 1800 --execute
Use task binary for the binary adapter, and panel confirmation for the familiar historical confirmation inputs. The same pinned runtime dependencies are in resources/frozen_a_verification.job.yml. GPU inference requires an admitted single GPU; the offline replay does not allocate one. Workflow confirmation was previously inspected, and W-H24 was later reused to fit corrections. Metadata projections retain model-visible states and targets; source-file hashes document the preserved originals. Full historical language-model training has not been rerun from scratch by this release.
For a fresh five-arm 27B workflow run, download the exact Qwen base and use the public entry point below inside the admitted single-A10 environment. It consumes the released ModernBERT initial decisions, not an annotation service. The full visible cases, common preludes, decoding limits, arrays and scoring are unchanged. A 5-hour cap includes setup and export; reserve failures stop execution without an automatic retry.
hf download Qwen/Qwen3.8-27B --revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --local-dir <27b-base-directory>
python -m src.public_workflow --root <artifact-directory> --base <27b-base-directory> --output <new-run-directory> --max-seconds 18000 --execute
The new independent verification covers the affected A phase. The public full-comparison entry point is supplied for reproduction; this revision does not claim a second fresh run of all five arms or a new blind heldout study.
The steering diagnostic supplement adds the saved bootstrap arrays, initializer and update-48/update-96 checkpoints, three preselected correction-training responses, complete token scores, captured teacher states/logits, and the exact source archive executed on Databricks. The original release remains unchanged. Fetch the supplement into a different directory:
python -m src.public_artifacts fetch --profile steering --scope full --root <new-supplement-directory>
python -m src.steering_diagnostic replay --root <supplement-directory>/diagnostic --output <new-scores.json>
python -m src.steering_fit_audit --root <supplement-directory>/cpu_fit_inputs --output <new-fit-audit.json>
python -m src.steering_fit_audit --objective-only --output <new-objective-audit.json>
The score replay needs the ordinary project dependencies. The CPU arithmetic audit also
needs torch==2.11.0, numpy and safetensors==0.8.0; a CPU-only Torch build suffices.
These commands allocate no GPU and load no language model. The manual
steering_diagnostic job contains a bounded GPU diagnostic followed by independent
CPU score verification. Its frozen protocol is in config/steering_diagnostic.yaml;
the existing admitted run has ended and the job does not automatically rerun.
Fresh neural reproduction uses the published three-case inputs and the original
release's pinned model/controller files staged in the configured private Volume.
It requires a new bounded admission, not reuse of the expired execution receipt.
On the three saved responses (207 teacher-forced tokens), mean output KL to A is 0.722353 for base, 0.202261 for the final learned corrections, and 0.001637 with full-A states supplied at all six boundaries. Every previous successful token is provided. These are local probability diagnostics, not autonomous workflow success or new heldout evidence. Full-state replacement supplies more information/capacity than the learned maps. It supports incomplete transfer under matched prefixes but does not isolate target construction, regression capacity, optimization or history effects. The CPU audit reproduces saved reconstruction errors; it does not turn earlier failed backend-equivalence checks into passes. The complete ten-condition matrix and per-case values are in paper/results.json and Table B3 of the manuscript.