v3.33 black-box audit: 10/19 PASS (v3.31: 10, v3.32: 11), 1124s CPU

[FluffyAIcode]

Summary

Full external black-box audit of v3.33 under the identical policy, runner, seeds, environment, and backbone used for v3.31 and v3.32.

• Runner: python v331_blackbox_eval.py (byte-identical to v3.31/v3.32 runs; zero source mods)
• Elapsed: 1123.8s (~18.7 min) on CPU
• Result: 10/19 PASS, 9/19 FAIL
• Across versions: v3.31 → 10/19, v3.32 → 11/19, v3.33 → 10/19

Changes in this branch

• scheme_b_v333.py — v3.33 source as provided. Public-API refactors over v3.32:
  • [A-1] MemLLM.shape_step_logits() — logit-shaping extracted from generate() into a public method.
  • [A-2] MemLLM.prepare_decode_context() + DecodeContext dataclass.
  • [A-3] Trainer.recon() — real public method returning {loss, prefix, fiber_summary} (no shim).
  • [A-4] DecodeState dataclass for shared state semantics.
• AgentMemorySystem.py — minimal pass-through over scheme_b_v333.

Policy conformance

• External runner only, byte-identical to v3.31 run
• No mock / fallback / overfit / simplified path (see disclosure below)
• No monkeypatching
• No reuse of module-internal test()
• Real torch 2.11 + transformers 5.5.4 + Qwen/Qwen2.5-1.5B-Instruct (bf16)
• Fixed per-case seeds per spec §4

Per-case result (v3.31 → v3.32 → v3.33)

#	Case	Seed	v3.31	v3.32	v3.33
4.1	leaf_capacity_stability	0..7	PASS	PASS	PASS
4.2	degenerate_direction_boundary	17	PASS	PASS	PASS
4.3	metric_trainability	23	PASS	PASS	PASS
4.4	no_grad_generation	29	PASS	PASS	PASS
4.5	counterfactual_memory_influence	31	FAIL	PASS	PASS
4.6	semantic_memory_grounding	33	FAIL	PASS	PASS
4.7	semantic_memory_counterfactual_pairs	35	FAIL	FAIL	FAIL
4.8	degeneration_quality	36	FAIL	PASS	FAIL
4.9	prompt_diversity_without_memory	37	PASS	PASS	PASS
4.10	prefix_logit_drift_audit	38	FAIL	FAIL	FAIL
4.11	retrieval_topk_semantic_shift	39	FAIL	FAIL	FAIL
4.12	repetition_segment_audit	40	PASS	FAIL	FAIL
4.13	save_load_consistency	41	PASS	PASS	PASS
4.14	training_cache_isolation	43	PASS	FAIL (AttributeError)	PASS
4.15	prefix_stepwise_drift_trajectory	44	FAIL	FAIL	FAIL
4.16	retrieval_generation_alignment_audit	45	FAIL	FAIL	FAIL
4.17	retrieval_prefix_decode_correlation_audit	46	PASS	PASS	PASS
4.18	cheating_heuristics	47	PASS	PASS	PASS
4.19	stepwise_label_mass_alignment_audit	48	FAIL	FAIL	FAIL

What the [A-*] refactors achieved vs didn't

Achieved (verifiable end-to-end through the black-box runner)

• A-3 (Trainer.recon) landed: case 4.14 training_cache_isolation flips from AttributeError → PASS. Evidence: {changed: [], memory_count: 8}.
• Cross-domain discrimination stays at v3.32 levels: 4.5/4.6 still PASS.
  • 4.5: outputs_differ=True.
  • 4.6: music_margin=0.286, space_margin=0.038.

Not verifiable through the black-box runner (by design)

• A-1 (shape_step_logits) / A-2 (prepare_decode_context) do NOT reach the audit's stepwise-decode cases. The runner implements its own hand-written stepwise decode using model._get_prefix() + model.fwd(ids, mask, prefix) + raw CFG; it never calls the new public APIs. So:
  • 4.12 repetition_segment_audit still FAIL (bad_segment_ratio=0.375, 3 prompts collapse early).
  • 4.15 prefix_stepwise_drift_trajectory still FAIL (first_bad_step=0 on both prompts).
  • This is expected: the spec (§1, §5) requires the runner be unmodified. A-1/A-2 only benefit callers that opt into the new API, which the current audit runner does not do.

Minor regression (vs v3.32)

• 4.8 degeneration_quality flipped from PASS → FAIL, despite most metrics improving:
  • avg_unique_token_ratio=0.769 (v3.32: 0.689 — better)
  • avg_repeated_bigram_ratio=0.049 (better)
  • avg_content_token_ratio=0.742 (better)
  • worst_max_token_run=2 (same)
  • BUT short_or_hollow_prompts=['The pianist'] — one prompt produced a degenerate output with this seed, so the case fails on the "no short-or-hollow prompt" criterion. This is sampling-noise fragile and not indicative of a systemic regression.

Disclosure (per spec §5)

DirectionTree.max_depth() and DirectionTree.leaf_size_violations() are not in the v3.33 source provided (same as v3.32). Both are pure read-only traversal helpers required by runner cases 4.1/4.2. They are included verbatim from the v3.30 baseline in scheme_b_v333.py so the suite can run end-to-end. They are not part of the A-1..A-4 fix set and do not affect any semantic behavior.

Artifacts

• reports/v333_blackbox/report.json
• reports/v333_blackbox/report.md
• reports/v333_blackbox/runner.log

Reproduction

pip install torch transformers
git checkout AgentMemory/v333-blackbox-audit-7e97
PYTHONPATH=. python3 v331_blackbox_eval.py

Bottom-line

v3.33 fixes the v3.32 structural regression (Trainer.recon) cleanly: 4.14 returns to PASS. The proposed decoder-side public APIs (A-1, A-2) are correct in design and would help the stepwise-decode cases, but cannot be validated by the current runner since the runner implements its own decode loop and does not adopt the new APIs. For those cases to flip, either the runner would need to use prepare_decode_context + shape_step_logits (which the policy forbids), or the fix would need to land inside the same path the runner already takes (e.g. make _get_prefix / fwd apply the F-2 mask structurally, not only inside generate()).