Surfaced while running evals for the #364 PR. Pre-existing — not introduced by that PR (verified below), so filing separately rather than widening its scope.
Symptom
evaluations/icp-cli.json case 15 ("Full-stack Motoko config artifacts") fails one behavior with the skill loaded:
❌ Uses @icp-sdk/bindgen (>= 0.4.0) Vite plugin with a didFile path pointing to the committed .did file
→ The bindgen block uses a 'candid' key instead of the required 'didFile' key,
and no version is specified for @icp-sdk/bindgen.
The model writes the Vite plugin as icpBindgen({ candid: "...", outDir: "..." }) instead of didFile:. That config is silently wrong — bindgen ignores the unknown key and has no .did path.
Confirmed pre-existing
Case 15 was re-run against the unmodified skill content (whole change set stashed):
BEFORE: ❌ ... configures the path via a 'candid' key instead of the expected 'didFile' field
AFTER: ❌ ... uses a 'candid' key instead of the required 'didFile' key
Identical failure, identical reason, 5/6 both ways. The only line the #364 PR touches on that case is the version number inside the expectation string.
Root cause: two different keys for the same concept
The skill uses two key names for "path to the .did file", and the wrong one dominates by weight:
| Key |
Where it belongs |
Occurrences in skills/icp-cli/ |
candid: |
icp.yaml, under recipe.configuration |
SKILL.md:124, 134, 149, 369, dfx-migration.md:102 |
didFile: |
vite.config.js, inside icpBindgen() |
binding-generation.md:16, 20, dev-server.md:62 — and not once in SKILL.md except one prose aside at :163 |
An agent reading SKILL.md — the entry point — sees candid: presented four times as the key for the .did path and didFile essentially never. binding-generation.md:91 then compounds it by discussing candid: in the same paragraph as bindgen:
…commit it to the repo, and specify candid: in the recipe config. If candid is omitted, the recipe auto-generates the .did…
Nothing anywhere states that these are two distinct keys in two distinct files.
Suggested fix
Make the distinction explicit rather than implicit — one sentence in binding-generation.md next to the Vite example, and a clause in SKILL.md pitfall 12:
The bindgen Vite plugin key is didFile, not candid. candid: is an icp.yaml recipe.configuration key that tells the recipe where the .did is; didFile: tells the binding generator. They point at the same file from two different config files, and bindgen silently ignores an unknown candid key.
Then re-run node scripts/evaluate-skills.js icp-cli --eval 15 — it should go 6/6.
Surfaced while running evals for the #364 PR. Pre-existing — not introduced by that PR (verified below), so filing separately rather than widening its scope.
Symptom
evaluations/icp-cli.jsoncase 15 ("Full-stack Motoko config artifacts") fails one behavior with the skill loaded:The model writes the Vite plugin as
icpBindgen({ candid: "...", outDir: "..." })instead ofdidFile:. That config is silently wrong — bindgen ignores the unknown key and has no.didpath.Confirmed pre-existing
Case 15 was re-run against the unmodified skill content (whole change set stashed):
Identical failure, identical reason, 5/6 both ways. The only line the #364 PR touches on that case is the version number inside the expectation string.
Root cause: two different keys for the same concept
The skill uses two key names for "path to the
.didfile", and the wrong one dominates by weight:skills/icp-cli/candid:icp.yaml, underrecipe.configurationSKILL.md:124, 134, 149, 369,dfx-migration.md:102didFile:vite.config.js, insideicpBindgen()binding-generation.md:16, 20,dev-server.md:62— and not once inSKILL.mdexcept one prose aside at:163An agent reading
SKILL.md— the entry point — seescandid:presented four times as the key for the.didpath anddidFileessentially never.binding-generation.md:91then compounds it by discussingcandid:in the same paragraph as bindgen:Nothing anywhere states that these are two distinct keys in two distinct files.
Suggested fix
Make the distinction explicit rather than implicit — one sentence in
binding-generation.mdnext to the Vite example, and a clause inSKILL.mdpitfall 12:Then re-run
node scripts/evaluate-skills.js icp-cli --eval 15— it should go 6/6.