README.md written by Claude under direction from njbrake. There may be mistakes.

laguna-otari-bridge

Run poolside/Laguna-S-2.1 — a 118B / 8B-active MoE — locally on Apple Silicon with llama.cpp + Metal, and expose it as an authenticated OpenAI-compatible endpoint.

Measured on an M4 Max Mac Studio (128GB): 37 tok/s generation at short context, 27 tok/s at 68K, with prompt processing around 430 tok/s.

Generation speed vs context depth

Generation slows as context grows — attention over the KV cache costs more per token. Sampled from 41 real requests (an agent workload, not a benchmark), the decay is smooth and predictable:

Context	gen tok/s
~9.5K	37.4
~22K	35.5
~38K	33.2
~52K	30.1
~68K	27.0

Averages: 36.6 tok/s under 20K context, 28.5 tok/s above 50K — roughly a 25% falloff across that range. Quote the short-context number only if your workload actually uses short prompts; coding agents rarely do.

Prompt processing held ~430 tok/s on average (peak 630, min 303) and, unlike generation, did not degrade appreciably with depth over the range sampled.

Findings

Most of the value here is the things that are not documented elsewhere, and that cost time to discover.

DFlash speculative decoding gets 0% draft acceptance here

The model card recommends the DFlash drafter. Enabling it made generation several times slower:

Config	gen tok/s
spec decoding off	39.5
spec decoding on, n-max=4	15.9
spec decoding on, n-max=15	6.9

The cause is visible in the server log:

draft acceptance = 0.00000 (0 accepted / 8865 generated), mean len = 1.00

Not one drafted token was ever accepted. Every drafted token is wasted compute, and the waste is proportional to n-max — which is exactly the slowdown pattern above.

This is a failure, not a tuning curve. A 0% acceptance rate means the drafter is not working, so do not read the table as "speculative decoding is bad for this model." Correctly configured, it may well help; these numbers say nothing about its potential.

Root cause is not established. What is known:

• The flags match the model card exactly, and the draft GGUF's metadata is correct (dflash.decoder_arch = laguna, block_size = 16).
• DFlash is EAGLE-style: the drafter consumes hidden states extracted from the target model's internal layers (llama_set_embeddings_layer_inp). If that extraction path is broken or unimplemented on the Metal backend, the drafter would receive garbage and accept nothing — consistent with what is observed, but unverified.
• Startup logs an initialization failure that is annotated as benign: dflash requires ctx_other to be set (this warning is normal during memory fitting). Whether it is genuinely benign here is untested.

The obvious next experiment is comparing draft acceptance on the CPU backend against Metal. If CPU accepts and Metal does not, it is a backend gap.

Runtime options

• Upstream llama.cpp does not support Laguna-S. PR #25165 covers only XS.2 and M.1, and is still open. Use poolside's fork, branch laguna, which added Laguna-S.2 support on 2026-07-20.

• MLX does not work (as of 2026-07-21). poolside publishes an official export, Laguna-S-2.1-NVFP4-mlx (4-bit, 71.9GB), but no MLX runtime can load it yet:

  $ python -c "from mlx_lm import load; load('./Laguna-S-2.1-NVFP4-mlx')"
  ValueError: Model type laguna not supported.
  

  Verified against mlx-lm 0.31.3 (latest) with mlx 0.32.0. The laguna architecture is absent from mlx-lm's model modules in both the release and git main. The nvfp4 quantization itself is supported — the architecture is the blocker.

  Tracking: mlx-lm PR #1223 (open, unmerged, and for XS.2 rather than S 2.1) and issue #1378. poolside maintains a llama.cpp fork but no mlx-lm fork. The weights were published ahead of runtime support; re-check before assuming this is still true.

A latent Metal f16 overflow can cause empty output

Metal's MUL_MAT_ID casts its activation input to f16. Laguna produces large activations in later layers, and anything above the f16 max of 65504 overflows to NaN — the model then returns empty output.

The fix is not merged anywhere: upstream PR #25442 is open, and an earlier attempt (#25389) was closed on a contributor-guideline technicality rather than on technical grounds.

It did not reproduce for S 2.1 here (verified with short and 8.2K-token prompts), and the reports were against XS 2.1 — but treat it as latent rather than fixed. metal-moe-f16-overflow.patch in this repo applies cleanly if empty output ever appears.

Reasoning tokens count against max_tokens

Laguna is a reasoning model and returns its reasoning in a separate reasoning_content field. Those tokens consume max_tokens, so a low limit yields an empty content — which looks identical to the Metal bug above. Give clients generous headroom.

The server ignores the requested model id

llama-server serves whatever single model it loaded, regardless of the model field in the request. A stale client config gets silently answered by the wrong model and looks perfectly healthy. The Caddy gate in otari-bridge/ rejects a mismatched id rather than letting that happen.

Requirements

• Apple Silicon Mac. 128GB unified memory for the Q4_K_M (75GB) quant — Q8_0 is 128GB and F16 is 235GB, both out of reach.
• ~80GB free disk.
• cmake, and a Tailscale account if you want the public endpoint.

Setup

# 1. build poolside's fork with Metal
git clone --branch laguna https://github.com/poolsideai/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 14
cd ..

# 2. fetch weights (75GB) -- note: pass filenames positionally, NOT via
#    --include, which the CLI silently ignores when filenames are given
pip install huggingface_hub
hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-Q4_K_M.gguf --local-dir ./models

# 3. serve
llama.cpp/build/bin/llama-server \
  --model models/laguna-s-2.1-Q4_K_M.gguf \
  --alias laguna-s-2.1 \
  --host 127.0.0.1 --port 8000 \
  --ctx-size 131072 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --jinja

--jinja is required — Laguna ships a Jinja chat template and tool-call parser. Tool calling works and returns proper finish_reason: tool_calls.

Context and memory

75GB of weights plus KV cache. Because 36 of the 48 layers are sliding-window (512 tokens), KV scales with only the 12 global-attention layers — roughly 7GB at 128K and 13GB at 256K, far cheaper than the layer count suggests.

macOS caps GPU-wired memory at ~75% of RAM by default, which 128K fits under. For the full 256K:

sudo sysctl iogpu.wired_limit_mb=114688   # 112GB; resets on reboot

Exposing it

otari-bridge/ puts it behind a bearer-token gate on a public HTTPS URL, for Otari or any other OpenAI-compatible client:

Internet ──TLS:443──▶ Tailscale Funnel ──▶ Caddy :9000 (auth) ──▶ 127.0.0.1:8000 (llama-server)

See otari-bridge/README.md.

License

Scripts in this repo: MIT (see LICENSE). The model itself is OpenMDW-1.1, set by poolside.