Three things that dominated coding-agent quality more than model choice did

[behrnt-slatgng]

Disclosure up front: I build Grunz, a Claude-Code-style coding agent that runs open weights instead of a frontier model. Different product, not a Zed competitor in any way that matters here, and nothing to sell in this post. Sharing three things that turned out to dominate agent quality far more than model choice did, because I spent months assuming it was the model.

1. The binding constraint on a long agent run is context, not capability

A long build run dies on context well before it dies on cost or on the model being too dumb.

Worse, the served context length is often not what you think. Across hosted endpoints, most serve around 32K regardless of what the model card advertises; a few reach 256K. It's a KV-cache budget decision on the operator's side, it's rarely documented, and it fails silently — nothing errors, the run just gets quietly worse.

The consequence for benchmarking: a benchmark number is a score for a model at one served context length, not a score for a model. The same weights at 32K and at 200K are not the same agent.

2. Compaction is its own failure surface, and it's the one nobody instruments

When the window fills, something has to decide what to throw away. If the goal statement isn't explicitly pinned through the compaction, it gets summarised into vagueness. The model then reads back its own compacted notes, no longer knows precisely what it was doing, and quietly restarts the plan — usually re-deriving work it already finished.

From the outside this looks like "the model forgot my instructions." The instructions weren't buried in the middle of a long context; they were compressed out.

Two things that helped more than anything else I tried:

• pin the goal and the acceptance criteria verbatim through every compaction, never summarised
• log where compaction fired in a run, so you can correlate a bad run with the point it lost the plot

Multi-agent orchestration made this worse, not better, in my experience — every handoff is another chance to drop the goal, and the compaction boundary and the handoff boundary rarely line up.

3. On open weights specifically: tool-call format drift, not refusal

The failure mode people expect from open models is refusal or lower capability. In an agent loop the thing that actually breaks is output-format adherence.

This is sharpest on abliterated / decensored variants, where instruction-following and format compliance degrade noticeably before knowledge does. The model still knows what to do. It emits a tool call that's slightly off-spec, the parse fails, and the loop either stalls or retries into the same wall.

The practical implication for anyone building an agent on open weights: your tool-call parser needs to be substantially more forgiving than it does against a frontier model, and you want to measure format-compliance rate as a first-class metric rather than inferring it from task success. A strict parser turns a recoverable formatting wobble into a failed run.

---

None of this is Zed-specific, but all three surfaced only once I had agents running long enough to hit them, and I'd have saved months knowing them at the start. Happy to go deeper on any of it. Equally interested in how the agent panel here handles the compaction boundary, since that's the part I've found hardest to get right.