Skip to content

fix(icp-cli): prevent recipe and frontend config hallucinations#121

Merged
marc0olo merged 1 commit intomainfrom
fix/icp-cli-recipe-hallucinations
Mar 25, 2026
Merged

fix(icp-cli): prevent recipe and frontend config hallucinations#121
marc0olo merged 1 commit intomainfrom
fix/icp-cli-recipe-hallucinations

Conversation

@marc0olo
Copy link
Member

@marc0olo marc0olo commented Mar 25, 2026

Closes #120

Summary

  • Enriched the "Available recipes" table with exact versioned type strings and required/optional config fields — agents no longer need to guess recipe formats
  • Added releases page link so agents can verify latest recipe versions when the skill goes stale
  • Surfaced critical values from reference files (@icp-sdk/core >= 5.0.0, outDir: "./src/bindings", command === "serve" mode guard) directly in the main skill body

Why this is sufficient

The issue reported four hallucination cases:

  1. Wrong recipe version (@dfinity/motoko@v3.0.0) — the skill already had correct type strings in config examples, but the "Available recipes" table listed them without versions. An agent scanning that table could still guess. The enriched table now redundantly provides exact strings.
  2. Missing candid field — already covered by pitfall #12 and the config examples. No change needed.
  3. Trusted stale template repo — this was an upstream bug in icp-cli-templates, fixed by PR SPA? #23. Not a skill issue.
  4. Wrong @icp-sdk/core version, outDir, and mode guard — the reference files had this information, but the eval agent (Sonnet) didn't read them. Surfacing the critical values as one-liners in the main skill fixes this without duplicating the full reference content.

Eval results

The "Full-stack Motoko hello-world with React frontend" eval (the exact scenario from #120) went from 6/9 → 9/9 after these changes. The three previously failing checks (@icp-sdk/core version, outDir path, mode guard) now all pass.

Closes #120

- Enrich the "Available recipes" table with exact versioned type strings
  and config fields so agents don't guess bare names or wrong versions
- Add releases page link for agents to verify latest recipe versions
- Surface critical values from reference files (package versions, outDir,
  mode guard) directly in the main skill so agents get them without
  needing to read the reference files
@marc0olo marc0olo requested review from a team and JoshDFN as code owners March 25, 2026 09:14
@github-actions
Copy link

Skill Validation Report

Validating skill: /home/runner/work/icskills/icskills/skills/icp-cli

Structure

  • Pass: SKILL.md found
  • Pass: all files in references/ are referenced

Frontmatter

  • Pass: name: "icp-cli" (valid)
  • Pass: description: (507 chars)
  • Pass: license: "Apache-2.0"
  • Pass: metadata: (2 entries)

Markdown

  • Pass: no unclosed code fences found

Tokens

File Tokens
SKILL.md body 3,616
references/binding-generation.md 764
references/dev-server.md 690
references/dfx-migration.md 1,921
Total 6,991

Content Analysis

Metric Value
Word count 1,841
Code block ratio 0.19
Imperative ratio 0.13
Information density 0.16
Instruction specificity 0.80
Sections 17
List items 45
Code blocks 20

References Content Analysis

Metric Value
Word count 1,627
Code block ratio 0.21
Imperative ratio 0.15
Information density 0.18
Instruction specificity 0.75
Sections 16
List items 29
Code blocks 8

Contamination Analysis

Metric Value
Contamination level low
Contamination score 0.12
Primary language category config
Scope breadth 3
  • Warning: Language mismatch: javascript, shell (2 categories differ from primary)

References Contamination Analysis

Metric Value
Contamination level low
Contamination score 0.03
Primary language category javascript
Scope breadth 2
  • Warning: Language mismatch: shell (1 category differ from primary)

Result: passed

Project Checks


✓ Project checks passed for 1 skills (0 warnings)

@marc0olo marc0olo merged commit f6be531 into main Mar 25, 2026
6 checks passed
@marc0olo marc0olo deleted the fix/icp-cli-recipe-hallucinations branch March 25, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

icp-cli skill: list exact recipe type strings to prevent LLM hallucination

2 participants