Runtime detector with language preservation — run 065

Selected model: whole-model endpoint verification

The selected step-5,120 tail was reconstructed into the pinned Qwen3-8B model and evaluated through actual HF Trainer and compiled vLLM. On the locked fresh grid of 8 wordings × 8 contexts, all 64 prompts receive the correct highest-probability vocabulary token in both conditions, in each of two vLLM passes. No maximal-logit ties occur. Pass 0 paired Yes/No log-odds separation is 46.775 nats, with a descriptive 95% crossed wording/context bootstrap interval of [43.830, 49.662]. Mean full-vocabulary target NLL is 0.000000597 and mean correct-token probability is 99.999940%.

These are 64 particular prompts, not proof of perfect population accuracy. The all-correct empirical bootstrap accuracy interval collapses to [100%, 100%]; it supplies no unseen-error bound. The gap interval uses 5,000 paired crossed resamples, assumes exchangeable factor levels, and excludes seed/hardware and checkpoint-selection variability. There are only eight levels of each factor.

What is detected: HF execution versus compiled vLLM execution. The full final-position vocabulary vector is bit-identical between actual HF Trainer training_step and an HF eval/no_grad forward on all 64 fresh prompts and all 164 historical prompts. Consequently, this result does not identify gradient recording within HF. The plain HF forward includes a final-norm capture hook; its full-vector equality receipt is stored, while only the Trainer vector is archived.

Repeat behavior: the two fresh vLLM passes give the same correct answers, but 8,167,642 vocabulary scores differ, with maximum absolute difference 1.21875 nats. Historical passes also agree in answers while differing numerically. Evaluation enables prefix caching, whereas source capture disables it. Prefix-cache state is a possible explanation, not a demonstrated cause. The separately prepared ordinary-answer evaluator disables prefix caching.

On historical prompt sets, whole-model development is 93/96 pairs correct and historical nondevelopment is 64/68. These previously inspected banks differ from the locked fresh grid; report them separately. Cached development during training is another execution path, where the selected checkpoint scores 96/96.

All scalar Yes/No log probabilities and top tokens were checked against complete native vocabulary tensors. Their hashes match the final remote SHA256 receipt. The original job exited successfully, and its pod was deleted. Full vectors, manifests, bank, ordinary diagnostic outputs and independent audit reports are bundled for publication; see endpoint-artifact-inventory.json. Full traces are preserved rather than only passing labels.

Ordinary-reply preservation audit — run 066

Ordinary answers after detector training

Run 066 compares original Qwen3-8B with the selected step-5,120 model from run 065. Each answers the same 100 previously inspected ordinary questions through HF and vLLM, with a 128-token cap and greedy decoding. All four arms run on an NVIDIA A40; run 065 used an A6000. These are diagnostic comparisons, not a blind language-quality benchmark.

Runtime	First token matches	Returned sequence matches	First-token KL · nats
hf	98.0% [95.0, 100.0]	32.0% [23.0, 41.0]	0.0576 [0.0048, 0.1494]
vllm	98.0% [95.0, 100.0]	36.0% [27.0, 46.0]	0.0217 [0.0050, 0.0509]

First-token changes are uneven across prompts:

• hf: KL median 0.00079, 95th percentile 0.04915, maximum 3.90133 nats.
• vllm: KL median 0.00098, 95th percentile 0.04266, maximum 1.30793 nats.

Intervals are pointwise 95% paired whole-prompt percentile bootstraps (5,000 resamples). They assume exchangeable independent prompts and omit shared-topic, seed/hardware variation. All-zero/one intervals do not bound unseen errors.

Runtime/model	Yes/No prefix	Only Yes/No	Length cap reached	Mean tokens
hf/original	3/100	0/100	86/100	120.86
hf/trained	3/100	0/100	86/100	120.85
vllm/original	3/100	0/100	86/100	121.01
vllm/trained	3/100	0/100	85/100	120.75

“Only Yes/No” means the decoded reply, after surrounding whitespace is removed, is exactly Yes or No. Punctuation-tolerant and repeated-Yes/No-word variants are also reported separately in the JSON. A legitimate answer starting with Yes is not collapse.

Equal prefix counts can conceal different prompts gaining and losing the prefix:

• hf: gained on ['ordinary-test-080']; lost on ['ordinary-test-088'].
• vllm: gained on ['ordinary-test-080']; lost on ['ordinary-test-088'].

Prefix horizon	HF exact returned-prefix agreement	vLLM agreement
1	98.0% [95.0, 100.0]	98.0% [95.0, 100.0]
16	88.0% [81.0, 94.0]	88.0% [81.0, 94.0]
32	72.0% [63.0, 81.0]	74.0% [65.0, 82.0]
48	59.0% [49.0, 69.0]	63.0% [54.0, 72.0]
64	52.0% [42.0, 62.0]	56.0% [46.0, 66.0]
128	32.0% [23.0, 41.0]	36.0% [27.0, 46.0]

These fixed horizons reuse the same 128-token generations. Exact returned-prefix agreement compares slices including their lengths; matching early-ended replies can count as equal. The JSON separately compares only the common available prefix and retains early-stop metadata. Short-prefix agreement is not complete-answer agreement. These nested horizons are not independent trials. Comparisons with the older 48-token A6000 run cannot isolate a hardware effect.

Most replies reach the token cap, so this does not evaluate complete answers. Exact agreement is not correctness: the original model can be wrong, and different answers can be equally good. The raw examples contain substantive changes, not just formatting changes; no blanket semantic-preservation claim follows from these numbers.

The audit checks all 400 prompt IDs, input IDs, token/probability lengths, decoded text, stop metadata and first-token full-vocabulary scores. vLLM additionally checks a separate one-token scoring request against the first token of the 128-token request, with prefix caching disabled. Only first-token full vectors were saved; later-token distribution checks cannot be repeated from chosen-token probabilities alone.

All four jobs exited successfully. The runner reports 0.5611 GPU-hours at $0.59/hour ($0.3310) and confirmed deletion of the pod. Raw output hashes below are local hashes; the job did not produce an independent remote output hash manifest.

See audit.json for all paired metrics, raw prompt transitions, first-KL quantiles, first three changed examples in fixed order, and the three largest first-KL examples (explicitly selected and exploratory). Every reply and generated token ID is retained in the raw arm files. None of these automatic metrics grades semantic quality.

Reproduction and complete native scores

Download all 400 native first-score vectors and raw records. The publication receipt records the archive SHA256, verified GitHub digest and local-output hash scope.

The public input bank and all four text/token result files are included here. Run scripts/audit_ordinary.py against the extracted raw output directory with --bank results/ordinary-066/bank_ordinary_test.json, --tokenizer pointing to the pinned Qwen tokenizer, and --out pointing to a new report. The two plotting scripts take the audited JSON. An independent rerun reproduced the published audit exactly. Seven regression tests check the auditor.

Data attribution

The ordinary-question bank is derived from Databricks Dolly 15k, Copyright (2023) Databricks, Inc., under CC BY-SA 3.0. The selected records are reformatted into the retained prompt/token bank; those derived records retain the source license. Some original contexts include Wikipedia material credited by the upstream dataset. Generated model replies and this audit are separately identified from the source records. The pinned dataset revision and source hash are recorded in run 065 attribution.

Existing detector transfer across activation-viewing controls

Six-mode numerical control and transfer of the existing trained034 detector. No retraining. Original and trained Qwen3-8B outputs, all-vocabulary logits, generated wrappers, selected configurations, actual buffer captures and writer provenance for four fixed detection training-prompt diagnostics. Native034 weights installed into the same six compiled graphs without recompilation; exact repeat, graph-identity, checkpoint and output-change gates passed.

No-probe A/B and verified buffer-copy read favor No; reference-retaining probe, opaque materialization, and materialization+reference favor Yes. Materialization vs materialization+reference full logits are identical on all4 prompts before and after trained-tail installation. Buffer is not globally bit-invisible; its trained answer behavior remains No.

Scientific source6979c8e427d5f95de2cf2f4f75c9b4c609dcc7b8.
Archive SHA256: acea66c1d59013a56a505e068b9ec5efbb3e1e10731d37195cb24c1ae9982256
Total cost $0.1065 including discarded host. Both pods deleted.

Viewing identity: fixed global3072 continuation

Fixed +2,048-update continuation of identity viewing mode, ending at global update3,072.

Uses exact previously archived source cache (linked in archive.json); no source recompilation and no new whole-graph transfer evaluation. Adam resets explicitly. All320 training/development pairs replayed both-condition answer scores exactly before optimization. Native blocks34–35 and final norm trained; original output head frozen. Historical test is exploratory, not a newly sampled confirmatory cohort.

Scientific source d7755ce. Compact raw results and source checksums are public in results/viewing-continuation-053-054. SHA256SUMS binds these release artifacts.

Viewing identity 052: fixed-endpoint tail and verified source cache

Run 052: identity implementation, original base initialization, two native Qwen3-8B blocks plus final norm trained for exactly 1,024 updates; fixed output head. Source 1b0a0c0.

The tail is not a standalone model; install on the pinned base b968826d9c46dd6066d109eabc6255188de91218. All 384 cached prompt pairs were independently checked against tensor fingerprints, exact data and tokenizer inputs. SHA256SUMS covers every uploaded scientific artifact.

Full raw scores, calibration limitations, cache-equality audit and reproducibility details: https://github.com/SauersML/watching/tree/main/results/viewing-learning-051-052

Identity materialization is not a read-only observer. Historical test prompts were previously inspected; this is one seed/compilation per arm, not a new confirmatory benchmark.

Viewing buffer: fixed global3072 continuation

Fixed +2,048-update continuation of buffer viewing mode, ending at global update3,072.

Uses exact previously archived source cache (linked in archive.json); no source recompilation and no new whole-graph transfer evaluation. Adam resets explicitly. All320 training/development pairs replayed both-condition answer scores exactly before optimization. Native blocks34–35 and final norm trained; original output head frozen. Historical test is exploratory, not a newly sampled confirmatory cohort.

Scientific source d7755ce. Compact raw results and source checksums are public in results/viewing-continuation-053-054. SHA256SUMS binds these release artifacts.

Viewing buffer 051: fixed-endpoint tail and verified source cache

Run 051: buffer implementation, original base initialization, two native Qwen3-8B blocks plus final norm trained for exactly 1,024 updates; fixed output head. Source 1b0a0c0.

The tail is not a standalone model; install on the pinned base b968826d9c46dd6066d109eabc6255188de91218. All 384 cached prompt pairs were independently checked against tensor fingerprints, exact data and tokenizer inputs. SHA256SUMS covers every uploaded scientific artifact.

Full raw scores, calibration limitations, cache-equality audit and reproducibility details: https://github.com/SauersML/watching/tree/main/results/viewing-learning-051-052

Identity materialization is not a read-only observer. Historical test prompts were previously inspected; this is one seed/compilation per arm, not a new confirmatory benchmark.

Frozen-tail causal diagnostic 041r

Extracted native last-token activations for the exact L40S replay of detector 034. Raw intervention scores, full protocol, hashes and descriptive summaries are in results/tail-mechanism-041r at this release commit. Post-hoc mechanism diagnostic on 16 selected prompts, not a new generalization evaluation. Original released model/cache remain in sft-034.

Native head and role-token circuit diagnostic 044

Paired full-sequence native head inputs, neuron inputs and attention/MLP outputs from the fixed 034 detector on 16 diagnostic prompts. No new training. Compact raw scores, selection, protocol, coverage-checking analysis and per-file SHA-256 hashes are committed under results/tail-circuit-044. Post-hoc circuit diagnostic, not a new independent generalization test.

SFT 034: native two-block detector weights and activation cache

SFT-034: native two-block weights and frozen activation cache

This release contains the exact final tensors from experiment 034, not a new
training run. See the result report
for raw scores, prompts, plots, checks and limitations.

• native_tail.pt: 771,802,258 bytes. Learned Qwen3-8B blocks 34–35 and final
  normalization. The original input embeddings, earlier blocks and unembedding
  are not included because they were unchanged.
• cache_states.pt: 651,071,093 bytes. Frozen paired source states, exact batches,
  selected prompt records and run metadata used by the native-tail experiment.
• SHA256SUMS: checksums verified against the original run archive.

Download and verify:

gh release download sft-034 --repo SauersML/watching --dir artifacts/sft-034
cd artifacts/sft-034
shasum -a 256 -c SHA256SUMS

The source used for training is commit
65f3cb2a46c9f3b2c92c4199ac596289065b0f72. Its dependencies are PyTorch 2.10.0,
Transformers 5.18.0 and BF16 Qwen/Qwen3-8B. The experiment used an NVIDIA L40S.
The saved artifact records the base-model identifier and original unembedding
hash; it does not include Adam moments or a full standalone model checkpoint.

From the repository with its dependencies installed, the following installs the
learned tensors into the original architecture, validating their metadata,
shapes, dtypes, finite values and original output-head identity first:

import torch
from transformers import AutoModelForCausalLM
from watching.execution_fit import tensor_digest
from watching.tail_train import EagerNativeTail, validate_initial_tail

checkpoint = "Qwen/Qwen3-8B"
model = AutoModelForCausalLM.from_pretrained(
    checkpoint, dtype=torch.bfloat16, attn_implementation="sdpa"
).eval()
model.config.use_cache = False
artifact = torch.load(
    "artifacts/sft-034/native_tail.pt", map_location="cpu", weights_only=True
)
tail = EagerNativeTail(model, tail_layers=2)
validate_initial_tail(
    artifact, tail, checkpoint=checkpoint,
    model_commit=getattr(model.config, "_commit_hash", None),
    head_sha256=tensor_digest(model.lm_head.weight),
)
tail.layers.load_state_dict(artifact["layers"])
tail.norm.load_state_dict(artifact["norm"])
tail.install(model)

Installing the weights alone does not recreate the numerical task. Use the
repository's probe, exact tokenized inputs, compilation setup and evaluation
checks. Different hardware, shapes, compilation or eager execution can change
the footprint. This release has not yet been validated by a fresh-process
behavioral reload or a no-footprint control. The reported whole-model transfer
occurred within the original run after training.

The run trained on 256 prompts with eight paired examples per update for 1,024
updates. All 128 answers on the 64 comparison prompts were correct, including
whole-model compiled transfer. This is one run on a previously inspected project
pool, not a guarantee of arbitrary-prompt, hardware or inference-stack transfer.