Skip to content

feat: enhance algolia-cli skill with evals and improved docs 🔬#4

Merged
leonardocuco merged 1 commit intomainfrom
enhance/algolia-cli-skill-improvements
Mar 18, 2026
Merged

feat: enhance algolia-cli skill with evals and improved docs 🔬#4
leonardocuco merged 1 commit intomainfrom
enhance/algolia-cli-skill-improvements

Conversation

@leonardocuco
Copy link
Contributor

Summary

  • Evaluated the algolia-cli skill with 3 realistic test scenarios (migrate records, synonyms/rules, backup/API key), comparing with-skill vs without-skill baseline
  • Improved skill from 82% → 100% pass rate across all evals (baseline without skill: 40%)
  • Optimized description for better triggering accuracy (48% → 57% test accuracy)
  • Added skill differentiation between algolia-cli and algolia-mcp

Key improvements to SKILL.md

  • Added -y flag to all write commands in quick reference tables
  • New Synonym Type Guide with decision rules for synonym vs oneWaySynonym
  • Expanded non-interactive mode convention with exhaustive command list
  • New Common Workflows section (migrate, backup, restore patterns)
  • Skill vs algolia-mcp differentiation table
  • Direct invocation guidance (/algolia-cli fallback)
  • Optimized description for triggering

Eval results

Eval Without Skill With Skill (before) With Skill (after)
Migrate records 20% 80% 100%
Synonyms & rules 50% 67% 100%
Backup & API key 50% 100% 100%
Average 40% 82% 100%

Test plan

  • Ran 3 eval scenarios with-skill and without-skill (iteration 1 baseline)
  • Verified improvements with iteration 2 runs (100% pass rate)
  • Ran trigger description optimization (5 iterations, 20 queries)
  • Verified no false triggers on 9 should-not-trigger queries (100% precision)
  • Manual review of SKILL.md readability and completeness

🤖 Generated with Claude Code

…d triggering

Improve the algolia-cli skill based on systematic evaluation (3 test scenarios,
with-skill vs without-skill baseline). The skill now scores 100% on all evals
compared to 40% without the skill.

Changes:
- Add -y flag to all write commands in quick reference tables
- Add Synonym Type Guide (synonym vs oneWaySynonym decision rules)
- Expand non-interactive mode convention with exhaustive command list
- Add Common Workflows section (migrate, backup, restore)
- Add skill differentiation table (algolia-cli vs algolia-mcp)
- Add direct invocation guidance (/algolia-cli fallback)
- Optimize description for better triggering accuracy
- Update marketplace.json description
- Add cross-reference in algolia-mcp for write operations
- Add evals framework (evals.json + EVAL_RESULTS.md)
- Add *-workspace/ to .gitignore for eval workspaces

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@leonardocuco leonardocuco changed the title feat: enhance algolia-cli skill with evals and improved docs feat: enhance algolia-cli skill with evals and improved docs 🔬 Mar 18, 2026
@leonardocuco leonardocuco merged commit e7d04a6 into main Mar 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant