Repository navigation
Eval results
heft21 edited this page Nov 16, 2026
·
1 revision
Drydock's eval/ harness gives a model real modding tasks in plain words and checks the files on disk afterwards, never what the model said. Each task is run three times.
Two local models served by llama.cpp on one RTX 3090, core profile:
| Task | What it asks | Qwen3.5 9B (Q8_0) | Qwen2.5-Coder 14B (Q8_0) |
|---|---|---|---|
| copy-variant | copy a vanilla definition into the mod | 3/3 | 0/3 |
| new-mod | start a mod that depends on Vanilla and audits clean | 3/3 | 0/3 |
| water-question | explain why planetary water does not show in the editor | 1/3 | 0/3 |
| edit-field | rename one block without touching the rest | 0/3 | 0/3 |
| name-not-guid | point a field at existing content by name | 0/3 | 0/3 |
| delta-add | add an element to a delta-encoded list | 0/3 | 0/3 |
| Total | 7/18 | 0/18 |
Qwen2.5-Coder 14B chose the right tool with the right arguments every time, but wrote the call as a JSON block in its answer instead of making it. Nothing ran.
Qwen3.5 9B makes real tool calls. Its failures show where Drydock could forgive more than it does today:
-
edit-field: it sent the new name wrapped in a list (
["Big Thruster"]), and the list was written into a text field. -
name-not-guid: it wrote a plain name where a reference goes, without the
@that turns a name into a GUID. -
delta-add: it wrapped the new element as
{"Key": ..., "Value": ...}with a key of its own. -
water-question: the answer lives in the
planet-waterguide; it found it once in three runs.
Two models and three runs each are a first look, not a ranking.
From a source checkout, with a model server running:
python eval/run_eval.py --base-url http://localhost:1234/v1 --model <name> --profile core --repeat 3
Drydock, by Heft · github.com/heft21/drydock-se2 · MIT license · not made or endorsed by Keen Software House