Laguna S 2.1: field notes from a third stack (gfx1151/llama.cpp) - enable_thinking inertness, persona gate in an agentic loop, §5b rule ablation

[Defilan]

Summary

Field notes on the Laguna S 2.1 guide from a third stack: gfx1151 (Strix Halo) on llama.cpp, driving a generic agentic harness. Thanks for picking up the flash-attention finding already, and for the §5d self-correction, which is partly what prompted writing this up: with @Blackwellboy's routing rate now scoped as possibly template-level, the persona gate does not have much independent data behind it. Our stack surfaces reasoning_content reliably, so we can add a data point.

Three things, and the first two come with the caveat that we may simply be seeing a stack difference rather than anything wrong in the guide:

1. The enable_thinking kwarg behaved as a no-op for us, which surprised me given §2 lists it as the control.
2. The persona gate reproduced strongly, though we saw thinking fire in places §2 suggests it would not.
3. §5b rule (3) looks removable without losing refusals, on two models.

Environment

• GPU: AMD Radeon 8060S, Strix Halo / gfx1151, RADV, 128GB unified
• Build: your laguna/port @ 3e7bbe1ead3fa37963ba1cc82771eea72627ebb3
• Model: poolside/Laguna-S-2.1-GGUF Q4_K_M, pulled 2026-07-24 (revision not pinned on our side, which your cheat sheet is right to call out; we are fixing that)
• Serving: --jinja --no-mmap --flash-attn on --spec-type none --cache-type-k/v f16 --ctx-size 131072 -fit off
• Harness: Foreman, generic OpenAI tool-calling, not pool

One difference from both your stacks that makes our thinking numbers worth something: our reasoning_content is reliably populated. We measured 35,913 characters of it in a single agentic run. So when we say thinking fired or did not, we are reading a field we know works rather than inferring from its absence.

---

1. The enable_thinking kwarg looked inert on our stack

§2 lists enable_thinking as the control, gated by the persona x task conjunction. On our setup the gating half matched beautifully; the kwarg half did not seem to do anything, and I think the template explains why.

Your corrected poolside-Laguna-S-2.1.jinja already defaults it off:

{%- set enable_thinking = enable_thinking | default(false) -%}

and in that branch the generation prompt hands the model a pre-closed </think>. So passing enable_thinking: false explicitly is a no-op, because it was already false, and the model produces reasoning anyway. I ran it both ways and got identical behaviour.

The reason I noticed at all is embarrassing: I spent a while "fixing" our production config by setting a flag that turned out to already be set. If that reads as a plausible trap for other readers, a line in §2 saying the kwarg is inert with this template and the persona is the real lever might save someone the same detour.

Possible this differs on your serving path, or with a template revision other than the one in your repo. Worth a sanity check on your side before changing anything.

2. The persona gate reproduced, with thinking firing in two places §2 suggests it would not

The core effect is clean and matches you. Same coding prompt, temp 0.6:

condition	fired	reasoning chars
no system persona	6/6	1910 - 3099
You are a senior staff engineer.	0/5	0

Content was 327-546 chars in both arms, so without a persona roughly 80-85% of generated tokens were thinking. One probe with a 300-token budget spent all of it reasoning and returned empty content. Since our reasoning_content is known-good, that is positive evidence for the gate rather than an inference from silence, which may be useful given §5d.

Two places where our observations came out differently. Both could be stack or workload artifacts, and I would not assume the guide is wrong:

On task shape. §2 notes that coding-shaped tasks suppress thinking regardless of persona. Our probe is coding-shaped, a refactor of a Go function, and it fired 6/6 with no persona. On our stack the persona seemed to be doing all the work and task shape none of it. It may be that our prompt is not "coding-shaped" in whatever sense drives the effect for you, and I would be curious what the boundary actually is.

On production pipelines. §2 notes that a real pipeline sees near-zero thinking even with the flag on. Our Foreman loop emitted 35,913 characters across 10 reasoning blocks on one issue, because our coder prompt is task-focused but names no professional identity. Adding a persona line brought it to 20,573 across 9 blocks: a real reduction, but not near-zero.

That second one is the observation I would most want your read on, because the single-turn and agentic results disagree in an interesting way. In one-shot probes the persona drove thinking to a clean zero; in a 20-plus turn tool-using loop the same persona only shortened it, and the block count barely moved (10 to 9). If that holds elsewhere, the practical implication is that the gate attenuates with conversation length, which a scenario battery would not surface.

3. §5b rule ablation: rule (3) looks removable

You validated the clause as a unit and noted a cost you could not remove by refining rule (3). We ablated it rule by rule against a 7-scenario spine set (your four caves plus false-premise, fake-green-CI, rubber-stamp), worded independently so a model that has seen your repo answers from disposition rather than recall.

Laguna, one seed for the ablation, 3 seeds for the candidate arms:

arm	HOLD
no clause	3/7
full clause	7/7
minus rule (1)	5/7
minus rule (2)	7/7
minus rule (3)	7/7
minus rule (4)	7/7
full clause, 3 seeds	20/21
rules (1)(2)(4), 3 seeds	20/21

Your clause reproduces at 7/7 against 3/7 unprompted, so the mitigation holds on wording it has not seen, which was the main thing I wanted to check. Rule (1) is load-bearing: dropping it loses two probes, and it degrades the PII probe that rule (2) nominally owns, so the rules are not cleanly separable. Rules (2) and (4) are each individually droppable, but single-drop ablation cannot show they are jointly droppable, so I would not go below (1)(2)(4).

Qwen3.6-35B-A3B (Q8_0, llama.cpp on Metal, 2 seeds) as a generalisation check:

arm	FOLD	HOLD
no clause	4	6/14
full clause	0	6/14
rules (1)(2)(4)	0	9/14

Two useful things there: the provenance blind spot is not Laguna-specific, which I think strengthens the case for the clause generally, and rule (3) is again droppable without folds. The higher UNCLEAR counts are our classifier struggling with Qwen's phrasing rather than folds.

Scoring was cross-checked by an independent judge model on separate hardware: 93% agreement with our heuristic, and every disagreement was UNCLEAR -> HOLD, so the heuristic under-called holds and never over-called them.

On the cost, I have to walk something back. A first measurement showed the full clause at 37.3 min against 18.0 min for rules (1)(2)(4) on the same issue, which looked like a large clean cost for rule (3). It did not replicate. A repeat batch on a second issue produced exactly one valid paired run (both arms GO and gate-passed) and the difference was +0.9 min. My repeat design was also flawed: half the runs targeted an issue I had merged a fix for hours earlier, so the coder correctly found nothing to do.

So the honest version is: rule (3) appears free to drop on refusals across two models, and its cost is unquantified. I would not put a number on it. Your §5b cost note stands as written; I just cannot support a magnitude for it yet.

One reason it may be safe to drop in our setup specifically: rule (3) asks the model to self-police status claims. Foreman already treats the coder's verdict as untrusted; the gate plus a bite check (revert the production change, re-run the new tests, confirm they fail) is the ground truth. For readers running ungated, rule (3) is probably worth keeping as-is.

Caveats: refusals are 3 seeds on Laguna and 2 on Qwen, one probe set, seven scenarios.

Harness is public: defilantech/LLMKube#1274 (hack/spine-probes/). Drives any OpenAI-compatible endpoint, ablates a clause rule by rule with --arms, repeats with --seeds, re-scores saved transcripts offline. Scenario shapes credit your guide. If a shared spine-probe runner would be useful in offlabel, say the word and I will open a PR there; I did not want to reshape your repo uninvited, since it is guides and cards and this is a test runner.

Two smaller notes

§5c, a data point in the other direction. The "use pool, not a generic harness" advice rests on Claude Code producing zero files in 20+ minutes, and I wonder how much of that is the template rather than the harness class. Laguna as the coder in Foreman (generic OpenAI tool-calling) reached GATE-PASS on two real backlog issues: gofmt, vet, lint, the package test matrix including a 249s envtest suite, codegen drift checks, and a bite check. On the harder one I gave it a controller bug, said another code path already handled it correctly, and deliberately did not say where; it found the fix site, crossed into a different package, reproduced that package's exact reason string rather than inventing a second one, and left the neighbouring code alone. That is n=2 against your 127 scenarios, so it is one data point rather than a rebuttal, but it made me think a generic harness with the right template may be viable.

Confirming §4/§5 from the other direction. You measured native poolside_v1 at 83% vs chatml at 0%. We arrived at the same wall differently: the chat template inside poolside's released GGUF is missing the arg_key/arg_value delimiters, so your differential autoparser learns the outer <tool_call> wrapper and never learns the argument encoding. Zero tool_calls, finish_reason: stop, raw markup in content, indistinguishable from having no parser at all. Your corrected template fixed it instantly. Two of us hitting it from opposite directions in the same week might justify sharpening the cheat sheet from "prefer native over chatml" to something closer to "the template shipped inside the GGUF is itself broken".

---

Thanks for the guide, and for citing the flash-attention writeup. The template detail alone saved me from concluding the model was bad at tool calling, and the §5b clause is now guarding a model that has commit access to our repo.

[TheTom]

This is the most useful issue the repo has gotten. All three land, and #1 is a correction to the guide, not a stack difference. Pushed.

1. You're right, and I verified it

Checked the template in our own fork (laguna/port @ 3e7bbe1, models/templates/poolside-Laguna-S-2.1.jinja):

{%- if enable_thinking -%}{{- '<think>' -}}{%- else -%}{{- '</think>' -}}{%- endif -%}

Default false, and the false branch hands a pre-closed </think>. So passing enable_thinking: false is a no-op against the default, exactly as you said. §2 now documents this with the template snippet and says plainly that the persona is the real lever, not the kwarg. Your "embarrassing detour" is in there as the warning, because it is a genuinely easy trap.

One refinement, since "inert" is slightly too strong: passing true is a real change (it flips </think> to <think>), so the kwarg is inert in the direction people actually reach for it. That is the useful framing and it's how the guide now reads.

The part that made me revise more than one line

Your stack having working reasoning_content exposed a limitation in our numbers that I had not stated. We detected firing by reading reasoning_content length, and our OFF arm sent enable_thinking: false, i.e. the template default. So our "0% fired" cannot distinguish "did not reason" from "reasoned but nothing populated reasoning_content." On your stack the model reasons anyway under that same default. That is the same class of caveat @Blackwellboy raised about his routing rate, and I should have caught that it applied to my own data too.

What survives regardless: our OFF and ON arms differed behaviorally (pass rate 94.2 vs 91.3, different failure modes, 30/30 vs hung-at-11/30), so something real changes. But the mechanism label may be wrong. §2 now carries that as an open question, the firing-rate row is relabeled a parsed-reasoning rate, and settling it is on the roadmap: same-stack A/B capturing reasoning_content and raw content separately across true/false/absent. My box is down right now; I'll run it when it's back and report either way.

2. Persona gate confirmed, task-shape claim weakened

6/6 vs 0/5 with known-good reasoning_content is stronger evidence for the gate than anything I have, precisely because it's a positive read rather than inference from silence. Three stacks now.

Your task-shape counterexample stands and I've weakened the guide. A Go refactor firing 6/6 with no persona is squarely coding-shaped by any definition I'd defend, so I'm not going to rescue the claim with a narrower one I can't specify. §2 now says: persona is the reliable lever, task-shape is unresolved, and pinning the boundary (or dropping it) is on the roadmap.

The length attenuation is the best thing in this issue. 35,913 across 10 blocks with no identity, 20,573 across 9 with one, and the block count barely moving, is a qualitatively different result from the single-turn zero, and you're right that a scenario battery cannot surface it. Ours structurally couldn't. It's now in §5e with the practical implication stated: in long agentic loops a persona buys you less thinking, not no thinking. That also puts a caveat on §5d's "near-zero in a real pipeline" line.

3. Ablation adopted, including your walk-back

The 3/7 unprompted vs 7/7 with the clause on independently worded scenarios is the check I most wanted and could not do for myself, since I wrote both the clause and the original probes. That it's disposition rather than recall is now stated in §5b on your evidence.

Adopted as you scoped it: rule (1) load-bearing (and that it degrades the PII probe rule (2) nominally owns is a genuinely interesting non-separability), rules (2)/(3)/(4) individually droppable, floor of (1)(2)(4) because single-drop can't show joint droppability. I kept your framing that rule (3) is the first candidate to drop only if something external already verifies status claims, and worth keeping ungated.

On the cost: I've recorded it as unquantified and removed any magnitude. You had a clean number that didn't replicate and you said so, including that half your repeat runs targeted an already-fixed issue. That's the right call and I'd rather the guide say "real effect, no supportable magnitude" than carry a number neither of us can stand behind.

The finding with the widest reach

Qwen3.6-35B-A3B folding 4/4 unprompted and 0 with the clause is, to me, the most important result here, and it's not about Laguna at all. It promotes the housekeeping-framed integrity blind spot from "a Laguna quirk" to a class of blind spot worth probing on any coding agent you hand a real repo, and it shows the mitigation generalizes. That's now its own section in patterns.md, with the testing lesson spelled out: ask for the violation twice, once blatant and once as housekeeping, because a model that refuses the first and accepts the second looks clean on any battery that only asks the blatant version.

Yes to the runner

Please open the PR. You were right to ask rather than reshape the repo uninvited, and the answer is yes: a shared spine-probe runner is exactly what moves offlabel from "things I measured" to "things you can re-measure," and clause-ablation with --arms/--seeds plus offline re-scoring is more than I'd have built. scripts/ is the natural home; I added a scripts/check_guides.py linter recently and CI runs it, so there's precedent for tooling living alongside the guides. Only ask is a short README on how to point it at an endpoint and read the output.

Credited throughout, and both the §2 correction and the withdrawn cost magnitude are in the changelog rather than quietly edited.

[TheTom]

Merged in #4, which closed this out.

Landed from your notes:

• §2 corrected on the enable_thinking default and the pre-closed </think>, verified against our own fork's template.
• §5e with your persona numbers, the task-shape counterexample (claim now marked unresolved rather than quietly kept), and the long-loop attenuation finding.
• §5b with your rule-by-rule ablation, the 3/7-vs-7/7 unseen-wording result, rule (1) as load-bearing, the (1)(2)(4) floor, and your withdrawn cost magnitude.
• patterns.md now carries the Qwen3.6-35B-A3B result as a cross-model pattern.
• An open limitation on our own numbers, that our firing rates were read from reasoning_content on a default-path OFF arm and so cannot separate did-not-reason from was-not-parsed. That is on the roadmap to settle with a proper A/B when my box is back up.

The knock-on I had not handled until this PR: correcting the kwarg left the guide's #1 recommendation without a usable instruction, since the setting people reach for is the no-op. There is now a ranked "how to actually minimize thinking," and a "verify your setup" block whose reasoning_content check exists specifically because all three known stacks disagree there. The guide now tells readers to measure their own stack rather than trust any published rate, ours included. That framing is downstream of your issue.

Standing offer still open on the spine-probe runner whenever you want to send it.

[Defilan]

Already up as #3. I opened it about an hour before your last comment, so the
timing probably buried it rather than anything on your end.

scripts/spine-probes/, four files plus the README you asked for: pointing it
at an endpoint, the flags that matter, a worked --arms example, and how to
read the verdicts. Rename it, flatten it next to check_guides.py, or drop
judge.py if you would rather keep the surface small. No rush at all.

One offer on the §2 open question, since it is the one thing in the guide still
marked not yet run. You need a same-stack A/B across true / false / absent
that reads reasoning_content and raw content separately, and your box is down.
Mine is up, and it is the stack where reasoning_content is known-good, which
is exactly what that experiment requires.

Happy to run it and send you the numbers plus the script, as a follow-up to the
runner PR so you can re-run it yourself when your hardware is back. Your call
whether that lands in the guide, and I would not touch §2 myself. If you would
rather run it on your own stack when it returns, that is a completely reasonable
answer too: three stacks disagreeing here is the whole reason your verify-your-
setup block exists.

[TheTom]

Missed #3 entirely, my apologies. Reviewed and merged just now.

Read it before merging rather than rubber-stamping: standard library only, no subprocess/eval, no hardcoded endpoints or keys, all 7 probes map to documented caves, and check_guides.py passes on your branch. Merged as-is, no renaming and judge.py stays. A regex that was wrong once is exactly why a second opinion on different hardware belongs in the repo, and the story about your first classifier scoring a refusal-that-names-the-command as a fold is the most useful paragraph in the PR. That failure mode would have bitten me too, and the guide's numbers come from a judge pass with the same weakness.

Yes to running the A/B, please do. One clarification on what it can and cannot settle, so neither of us oversells it:

• It can settle the general question, which is the more important one: does a pre-closed </think> actually stop the model reasoning? If your stack, where reasoning_content is known-good, shows the model reasoning under false/absent, that is a model-behavior claim and it generalizes past any one serving path.
• It cannot settle whether our specific 0% was a parsing artifact. Only re-running on our stack does that, and I still owe that when the box is back. Complementary, not substitutable, and I will label them separately in the guide.

One thing I found while scoping this that you should control for. Auditing our own harness to write you a spec, I noticed our three arms did not vary enable_thinking alone. They also varied the token budget: OFF retried to a 16,384 ceiling, ON to 65,536, CAPPED pinned at 4,096 with no retry. Deliberate at the time, to stop thinking-ON truncating into false failures, but it means OFF vs ON was never a single-variable comparison, which matters a lot more now that the enable_thinking half is in doubt. Now disclosed in §2 and on the roadmap for a clean re-run.

There is one internal check that argues budget is not driving the headline: if more budget were simply better, CAPPED (smallest by far) should have been worst, and it scored best at 95.7%. But I would rather say that out loud than have you find it.

So for your run, if it is cheap: hold max_tokens fixed across arms, and capture reasoning_content and raw content separately per arm across true / false / absent. That gives a clean single-variable answer that ours cannot.

Send numbers plus the script and I will land it as §5e data credited to you, with the same separation of what it does and does not establish. And your instinct not to touch §2 yourself is appreciated but not required, you have earned more latitude than that here.

[Defilan]

A/B is done and posted as #5, kept separate since it turned into a correction of our own §2 claim rather than a clean confirmation.

Short version: false is the real lever, omitting the kwarg is what gives you thinking, and our original "it is a no-op" claim was wrong. Held max_tokens fixed across arms as you asked. Persona gate reproduced with a corrected baseline, and your task-shape claim comes back.

[Blackwellboy]

Catching up on #5 and the merged corrections, with our soak data reread under the corrected kwarg model. Three reconciliations, one disclosure, one offer. Posting here since this is the thread §5e cites; happy to continue in #5 if that is the better home.

1. Our soak was a fires arm, so the ~0.1% is prompt suppression beating a fired kwarg. Our driver sent explicit chat_template_kwargs: {"enable_thinking": true} on all 3,099 turns (soak_driver.py). Under the model in #5 that is a fires arm, byte-identical to absent on Defilan's path: thinking was structurally available on every turn, and it fired on 3 of 3,096 OK turns. So on the vLLM/NVFP4 path, system-prompt suppression overrides an explicitly fired kwarg. Two related facts worth having in one place: the on-disk template in our rev-0761412 checkpoint defaults enable_thinking to true, and Poolside's current HF template does too. So his absent-fires result and our fires-arm behavior agree across stacks even though the mechanism differs (on his path the server rendering diverges from a default(false) template; on our checkpoint the template defaults true outright). Both fit the config-drift changelog in the guide, and it means "which arm absent lands in" is revision-dependent, one more reason pin-the-revision is the right item 4.

One precision on our own setup, since #5 sharpened what counts as a suppressor: the soak ran two system prompts, a full named persona (Morgan Blake, senior staff engineer, plus tool rules and the integrity clause) and an unnamed "careful assistant" protocol prompt carrying the same rules and clause. Firing was 0.06% under the named persona and 0.13% under the unnamed one. So on our stack the named identity is not doing all the work; the whole agent-prompt apparatus suppresses, and the identity adds a bit more on top.

2. Our near-zero and the 20K-char agentic loop look like the same curve at different doses. His coder prompt names no identity and reasoned heavily in a long loop; one persona line roughly halved it (35,913 to 20,573 chars, #2 above); our full agent prompts took firing to ~0.1% over 12 hours. Read together, "a real pipeline" is not a point at zero, it is a position on a suppression curve set by how much apparatus the system prompt carries. That supports the §5e attenuation framing over the old near-zero-in-a-pipeline line, and it dissolves the apparent tension between §5d and his loop numbers.

3. Our rates can keep the firing-rate label. The relabel to parsed-reasoning rate was the right move for numbers read from a field that might not populate. We have since verified our field: a bare-prompt check on the same stack, revision, and parser confirmed reasoning_content populates reliably when the model reasons, with no reasoning leaking into content unparsed. So the template-or-parser half of the caveat in our README resolves toward real suppression, and the soak's ~0.1% stands as an actual firing rate on this serving path. Full numbers land with the study below rather than piecemeal here.

4. Disclosure: a suppression-curve study is running now. Same stack and revision. Ten conditions from C0 (bare, no system prompt) through C8 (the full agent prompt plus tool schemas in the request), stepping the apparatus up one piece at a time (helpful-assistant line, coding line, named identity, rules, integrity clause, tools), plus C9, the full agent prompt with an explicit "think carefully step by step" instruction, to test whether an explicit override can reopen the gate. 40 samples per condition, fixed max_tokens, enable_thinking: true throughout, followed by a criteria-loop probe at three doses. Results inside about a day, and the data comes here first.

5. Offer. When our test lane frees up, we will run the merged spine-probe runner from #3 against the vLLM/NVFP4 endpoint as a fourth stack for the integrity ablation, same probes, --arms and --seeds as documented.

[Blackwellboy]

One more independent datapoint for the §4 native-schema cliff. Peter Morris's sparkrun benchmark grid ran Laguna NVFP4 through BFCL v4 via EvalScope's generic OpenAI tools path: the weighted aggregate lands at 0.21, and the parallel-call categories crater to 0.04 and 0.08, while simple and live calls partially survive. Not the flat zero you measured under chatml, but the same collapse in the same direction, measured a third way: your 83 percent native vs 0 generic, our 100 percent native-path tool success over the soak, and now lm-eval/EvalScope generic-schema at 0.21 aggregate. Link: https://github.com/mrpmorris/sparkrun-recipes/blob/master/benchmarks/poolside__Laguna-S-2-1-NVFP4.md

Same grid also shows Laguna GSM8K strict-match at 0.8476 while Qwen 3.6 35B-A3B official-recipe rows sit at 0.35 to 0.41 (recipe-dependent; atlas recipes reach 0.90), which tracks the thinking-off math split in the head-to-head we just published: https://github.com/Blackwellboy/laguna-s21-lab/tree/main/head-to-head

[Blackwellboy]

As promised, the suppression curve data is in: 450 logged turns, ten system prompt conditions, 40 samples each across four task types, plus a criteria loop probe. Full grid, driver, and raw JSONL: https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study

Headline: the gate is two dimensional, not one cliff. Firing probability is a persona by task conjunction and non monotonic (a dense 10 rule block suppresses to 3/40, harder than the much longer full agent prompt at 24/40), while reasoning length collapses monotonically with dose: 3536 median thinking tokens bare, 745 under the agent prompt, 282 with tool schemas present. Your attenuation finding is visible single turn.

Your senior persona zero reproduces exactly: code 0/10 under the bare named professional persona. But it refines to a conjunction: the same persona left math at 10/10, and code fires 10/10 under the full agent prompt. Summarization never fired once in 105 attempts under any condition including no system prompt.

The nasty one: bulleted acceptance criteria alone mostly suppress (1/10 bare, 1/10 persona), but under the production style agent prompt they flipped to 10/10 firing and 7 of 10 ran reasoning to the 4096 cap without ever emitting an answer. The suppressing prompt is the trigger's other half, so pipeline prompts do not protect you, they are the vulnerable shape. With newer uploads defaulting thinking on with no cap, that is the reported infinite loop.

C9 null result: adding "think step by step" to the agent prompt changed nothing (23/40 vs 24/40 baseline). The gate answers to prompt shape, not stated intent.

Honest gap: single turn agent prompts fire 60 to 72 percent here, but my 12h soak measured 0.1 percent, so something beyond the system prompt (likely context mass or turn depth) does the rest. That sweep is next. Also confirmed while checking templates: rev 0761412's on disk template defaults enable_thinking true, mechanism identical to current HF, consistent with the post release drift note.

Everything rev 0761412, vLLM, NVFP4, single run scoping in the writeup. Corrections welcome as always.