Skip to content

icp-cli: agents use 'candid:' instead of 'didFile:' in the bindgen Vite plugin config #367

Description

@marc0olo

Surfaced while running evals for the #364 PR. Pre-existing — not introduced by that PR (verified below), so filing separately rather than widening its scope.

Symptom

evaluations/icp-cli.json case 15 ("Full-stack Motoko config artifacts") fails one behavior with the skill loaded:

❌ Uses @icp-sdk/bindgen (>= 0.4.0) Vite plugin with a didFile path pointing to the committed .did file
   → The bindgen block uses a 'candid' key instead of the required 'didFile' key,
     and no version is specified for @icp-sdk/bindgen.

The model writes the Vite plugin as icpBindgen({ candid: "...", outDir: "..." }) instead of didFile:. That config is silently wrong — bindgen ignores the unknown key and has no .did path.

Confirmed pre-existing

Case 15 was re-run against the unmodified skill content (whole change set stashed):

BEFORE:  ❌ ... configures the path via a 'candid' key instead of the expected 'didFile' field
AFTER:   ❌ ... uses a 'candid' key instead of the required 'didFile' key

Identical failure, identical reason, 5/6 both ways. The only line the #364 PR touches on that case is the version number inside the expectation string.

Root cause: two different keys for the same concept

The skill uses two key names for "path to the .did file", and the wrong one dominates by weight:

Key Where it belongs Occurrences in skills/icp-cli/
candid: icp.yaml, under recipe.configuration SKILL.md:124, 134, 149, 369, dfx-migration.md:102
didFile: vite.config.js, inside icpBindgen() binding-generation.md:16, 20, dev-server.md:62 — and not once in SKILL.md except one prose aside at :163

An agent reading SKILL.md — the entry point — sees candid: presented four times as the key for the .did path and didFile essentially never. binding-generation.md:91 then compounds it by discussing candid: in the same paragraph as bindgen:

…commit it to the repo, and specify candid: in the recipe config. If candid is omitted, the recipe auto-generates the .did

Nothing anywhere states that these are two distinct keys in two distinct files.

Suggested fix

Make the distinction explicit rather than implicit — one sentence in binding-generation.md next to the Vite example, and a clause in SKILL.md pitfall 12:

The bindgen Vite plugin key is didFile, not candid. candid: is an icp.yaml recipe.configuration key that tells the recipe where the .did is; didFile: tells the binding generator. They point at the same file from two different config files, and bindgen silently ignores an unknown candid key.

Then re-run node scripts/evaluate-skills.js icp-cli --eval 15 — it should go 6/6.

Metadata

Metadata

Assignees

No one assigned

    Labels

    generatedCreated by an AI agenthallucinationSkill produces incorrect or fabricated outputskill-improvementImprovement to an existing skill

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions