Skip to content

Eval results

heft21 edited this page Nov 16, 2026 · 1 revision

Drydock's eval/ harness gives a model real modding tasks in plain words and checks the files on disk afterwards, never what the model said. Each task is run three times.

First measurements (2026-11-16)

Two local models served by llama.cpp on one RTX 3090, core profile:

Task What it asks Qwen3.5 9B (Q8_0) Qwen2.5-Coder 14B (Q8_0)
copy-variant copy a vanilla definition into the mod 3/3 0/3
new-mod start a mod that depends on Vanilla and audits clean 3/3 0/3
water-question explain why planetary water does not show in the editor 1/3 0/3
edit-field rename one block without touching the rest 0/3 0/3
name-not-guid point a field at existing content by name 0/3 0/3
delta-add add an element to a delta-encoded list 0/3 0/3
Total 7/18 0/18

What went wrong

Qwen2.5-Coder 14B chose the right tool with the right arguments every time, but wrote the call as a JSON block in its answer instead of making it. Nothing ran.

Qwen3.5 9B makes real tool calls. Its failures show where Drydock could forgive more than it does today:

  • edit-field: it sent the new name wrapped in a list (["Big Thruster"]), and the list was written into a text field.
  • name-not-guid: it wrote a plain name where a reference goes, without the @ that turns a name into a GUID.
  • delta-add: it wrapped the new element as {"Key": ..., "Value": ...} with a key of its own.
  • water-question: the answer lives in the planet-water guide; it found it once in three runs.

Two models and three runs each are a first look, not a ranking.

Run it yourself

From a source checkout, with a model server running:

python eval/run_eval.py --base-url http://localhost:1234/v1 --model <name> --profile core --repeat 3

Clone this wiki locally