Two things worth knowing if you run abliterated models or fall back to hosted endpoints

[behrnt-slatgng]

Disclosure first so it isn't buried: I build Grunz, a hosted chat + coding agent that runs abliterated open weights. It's cloud, not local. If running models on your own machine is the point for you — and in this repo it usually is — Jan is the right tool and I'm not trying to talk you out of it. Posting two findings that cost me real time, because they're relevant to anyone who ever falls back to a hosted endpoint for models too big for their hardware.

1. The served context window has almost nothing to do with the model card

The advertised window is a property of the weights. The window you actually receive from a hosted provider is a property of whoever is serving them.

Across the endpoints I've tested, most serve around 32K regardless of what the card claims. A few reach 256K. I have not found one actually serving the 1M figures that show up in release posts.

This is reasonable from the operator's side — it's a KV-cache budget traded against concurrency — but it's almost never documented, and it fails silently. Nothing errors. The conversation just quietly starts losing its own beginning.

If you want your real number rather than the claimed one: send a deliberately oversized prompt and read the error text, because providers usually leak the true maximum there. Or watch where the context actually starts dropping in a long chat and count backwards. Provider docs have not been reliable for this.

Worth knowing for a local-first app specifically: this is one of the few places where running locally gives you a hard guarantee that a hosted endpoint doesn't. You set the context length yourself and you know what it is.

2. Abliteration damages instruction-following before it damages knowledge

If you run any of the abliterated / decensored variants, this is the failure mode to expect.

The first thing to degrade is not what the model knows — it's how well it obeys. It still knows the material. It gets measurably worse at respecting the instruct template, the prefill, the stop sequences, and any structured-output contract.

In practice that looks like formatting conventions collapsing partway through a session, system-prompt instructions decaying over a long context, and JSON or tool-call output drifting if you run anything that parses it. For an app with a tool-calling or MCP layer, that second one matters a lot more than it sounds — the model doesn't refuse, it just emits something slightly off-spec and the parse fails.

Practical consequence: when you evaluate an abliterated variant against its base, test format compliance separately from output quality. A variant can be better at prose and worse at doing what it was told, and a single vibes test won't separate those.

My working hunch is the damage compounds with quantization damage, so staying at higher quants on smaller models is worth it — but that's a hunch from limited testing, not a measured result, and I'd rather be corrected than keep repeating it.

Happy to answer anything about running these in production. Genuinely interested in which abliterated variants people here rate for holding instruct-following, since that seems to vary between variants a lot more than raw capability does.