K3 Loop

Kimi K3 writes a web page, looks at a screenshot of what it built, and fixes it. One model closes the loop. While it does that, we measure whether the two things that make K3 unusual actually matter.

Running on DigitalOcean Serverless Inference — kimi-k3 at inference.do-ai.run, OpenAI-compatible.

Why this and not another benchmark

K3 shipped on 16 July 2026 with open weights on 27 July: 2.8T parameters, 16 of 896 experts active, a 1M-token context, and native vision. Moonshot's own showcase leans on one capability above all others — vision in the loop, where the model iterates between code and live screenshots rather than reasoning about a UI from a text description.

Their documentation also carries a warning almost nobody has tested. K3 was trained with its thinking history preserved in context. If a harness drops reasoning_content between turns, Moonshot say generation quality "may become highly unstable." No benchmark, no test case, no way to detect it. Rohit's writeup calls it an undocumented contract and says as much explicitly. Meanwhile mem0's K3 tutorial hit 60% design regressions and reached for an external memory layer — without ever preserving reasoning_content. They patched the symptom.

So this repo does two things at once:

• E1 — is the vision load-bearing? Same loop, same model, but feedback is either three screenshots or a measured description of the same DOM. The description gives measurements and never conclusions, so neither arm is handed the answer.
• E2 — does dropping thinking history hurt? Same loop, both arms see screenshots, but one replays reasoning_content verbatim and the other strips it.

Results, with confidence intervals and p-values, are on the /lab page and in docs/results.md. Raw traces are committed under runs/.

What we found probing the model first

Before designing anything we established what kimi-k3 actually does on DO's endpoint, because the docs don't say. Full write-up in docs/probe-findings.md; reproduce with npm run probe.

The load-bearing answers: reasoning_content is returned and can be replayed verbatim, a stripped history is also accepted (so E2 has a control), the real limit is 1,048,576 total tokens — confirmed by the API's own error text, not the catalog, which omits it — multiple images per request work, tools and images coexist in one turn, and streaming exposes reasoning_content deltas separately from content.

Two things to plan around: latency is 6–30s for small calls and minutes with images, and reasoning_effort varies non-monotonically on single samples (high produced more thinking than max), so don't build a cost model on it.

Quick start

npm install
npx playwright install chromium
cp .env.example .env   # add your model access key

Get a key from the DigitalOcean console under Gradient → Serverless Inference. A dop_v1_ API token also works as a bearer token, which is why the CLI scripts can fall back to your local doctl config.

npm run probe                     # verify the model does what we think it does
npm run test                      # 82 tests, no API calls needed
npm run dev                       # the studio at localhost:3000

Run the experiments:

# one run, to see the loop work
npx tsx scripts/bench.ts --task article-hero --arms vision --trials 1 --iterations 3

# the full grid
npx tsx scripts/bench.ts --arms all --trials 3 --iterations 4 --concurrency 2

npx tsx scripts/report.ts                      # analysis to stdout
npx tsx scripts/report.ts --markdown           # the results doc
npx tsx scripts/rescore.ts runs/e1e2/*.jsonl   # re-score old traces after a rubric change

Keep --concurrency low. kimi-k3 returns 429 Platform overloaded under load — six parallel runs killed our first grid attempt outright. The client now backs off up to two minutes and honours Retry-After.

How it works

brief ──► K3 ──► one self-contained HTML file
                      │
                      ▼
          Playwright renders at 1440 / 768 / 390
                      │
        ┌─────────────┴─────────────┐
        ▼                           ▼
  3 screenshots              DOM description
  (vision arm)                 (blind arm)
        └─────────────┬─────────────┘
                      ▼
              back to K3 with the current source
                      │
                      ▼
              critique + revision, repeat

Two rules keep it honest:

The feedback contains only the render. Never our rubric, never our pass/fail verdict, and never a named defect — the blind arm gets measurements, not conclusions. Leaking any of those would make an external oracle do the work and both arms would converge on our judgement instead of their own. We got this wrong first: an early version of the DOM description printed lines like PROBLEM: 3 text elements fail WCAG AA contrast, which is a rubric check restated in prose. That biased E1 toward the blind arm. It's fixed, and a test now asserts the description contains no verdict language.

Every request asserts the arms differ by exactly one field. assertOnlyReasoningDiffers runs on every single call, comparing the preserved and stripped message arrays. It is a pure function, so it costs nothing, and it means a refactor cannot quietly turn E2 into a comparison of nothing. It is also unit-tested against tampering in five different ways.

Scoring

Each task carries a machine-checkable rubric evaluated against the renderer's geometry probe — overflow, clipping, WCAG AA contrast against effective background, overlapping text, consistent left alignment, tap-target size, sub-12px text. Nothing is graded by a model, so there's no LLM-judging-an-LLM circularity and no rubric drift between runs.

Our first rubric saturated at 100% on iteration one, which measures nothing, so it was tightened. One of the added checks came from the model: in the very first smoke run, K3 spontaneously reported that a masthead at 70rem and a headline column at 62rem had misaligned left edges. Our rubric had missed it entirely. That defect is now consistent-left-alignment.

Regressions are computed, not judged. A rubric check that passed at iteration i and fails at some later j is a regression by construction. That's the same failure mem0 described, but measured mechanically.

Statistics

Small N. Every comparison reports a percentile bootstrap CI and a two-sided permutation p-value, and the report says plainly when a difference isn't supported. A difference in means across a handful of trials is not evidence, and presenting one as though it were would be the easiest way to publish something false.

Safety

The renderer executes model-authored JavaScript, and the deployed app shows the result to strangers. So:

• Content is injected with setContent, never written to disk.
• Every outbound request from the page is aborted. A test asserts a fetch() from page content fails rather than assuming it.
• Fresh browser context per render; Chromium's sandbox stays on; container runs as non-root.
• Hard timeouts — an infinite loop is a plausible model output, and there's a test for that too.
• Browser previews use <iframe sandbox="allow-scripts"> without allow-same-origin, so scripts get a null origin.
• The model access key is server-side only and never appears in a response body.
• Spend ceilings are enforced in src/lib/k3/budget.ts and throw rather than warn. Live runs are rate limited per IP.
• Free-form briefs are off by default in production.

Layout

path	what it is
src/lib/k3/harness.ts	the instrument — history management, preserved vs stripped
src/lib/k3/client.ts	hand-rolled HTTP client, deliberately not the OpenAI SDK
src/lib/k3/render.ts	sandboxed Playwright renderer, screenshots and DOM description
src/lib/k3/loop.ts	the iteration, protocol parsing, feedback construction
src/lib/k3/tasks.ts	seed tasks and their rubrics
src/lib/k3/metrics.ts	regressions, bootstrap CIs, permutation tests
scripts/probe.mjs	step-0 capability probe
scripts/bench.ts	headless experiment runner
runs/	committed traces — the numbers in the writeup come from these files
docs/probe-findings.md	what kimi-k3 actually does on DO's endpoint
docs/design-notes.md	methodology choices and threats to validity
docs/capacity-and-limits.md	the 429 investigation: capacity, not quota

client.ts is hand-rolled on purpose. The openai SDK has no reasoning_content in its message model, and round-tripping assistant messages through it risks dropping the one field E2 exists to measure. A silent strip there would make the two arms identical and the result a confident zero.

Deploying

doctl apps create --spec .do/app.yaml
./scripts/set-prod-key.sh              # interactive terminal: prompts with echo off
./scripts/set-prod-key.sh ./keyfile    # or read it from a file's first line
./scripts/set-prod-key.sh --help       # or set it in the console, no terminal needed

Use a model access key from Gradient > Serverless Inference, not a dop_v1_ API token. API tokens carry full account scope, and this app has a public URL — a leak would expose the account rather than just inference spend.

Docker rather than a buildpack, because Playwright needs Chromium and its system libraries. Set DIGITAL_OCEAN_MODEL_ACCESS_KEY as an encrypted secret before the first deploy. runs/ is copied into the image explicitly — the Lab page reads it at request time, so Next's build tracing doesn't pick it up.

Note that a single iteration takes minutes. Check any proxy's idle timeout before assuming a dropped SSE connection is a bug in the app.

Known limits

• Latency dominates. Thinking is always on at maximum effort and cannot be disabled. Live runs are a poor demo format; recorded replay is the better one.
• 429 Platform overloaded is common on this model right now, and it is not an account quota — our published rate limits sat at 4,499 of 4,500 requests remaining while calls were failing, and kimi-k2.6 answered fine on the same key at the same moment. It is shared capacity for this one model, flapping over minutes. Full evidence in docs/capacity-and-limits.md; scripts/wait-for-capacity.sh waits for a window to open.
• DO publishes no per-token price for kimi-k3, so every dollar figure here is an upper bound at Moonshot's list price and labelled as such.
• The tasks are small, self-contained pages. This is deliberate: K3 scores poorly on large interconnected repositories (roughly 6% F1 on Semgrep's largest enterprise test repo against about 20% for peers), so a monorepo demo would be testing its worst dimension.
• reasoning_tokens is not broken out in usage, so thinking cost cannot be separated from answer cost. E2 measures reasoning length in characters instead.