Skip to content

feat: enhance algolia-mcp skill with evals and improved docs 🔬#5

Merged
leonardocuco merged 1 commit intomainfrom
enhance/algolia-mcp-skill-improvements
Mar 18, 2026
Merged

feat: enhance algolia-mcp skill with evals and improved docs 🔬#5
leonardocuco merged 1 commit intomainfrom
enhance/algolia-mcp-skill-improvements

Conversation

@leonardocuco
Copy link
Contributor

Summary

  • Added evaluation framework for the algolia-mcp skill with 5 test scenarios comparing with-skill vs without-skill baseline
  • Surfaced critical reference content into main SKILL.md for more robust behavior (filter syntax, clickAnalytics, thresholds, interpretation benchmarks)
  • Added common workflows (Search Quality Audit, Recommendation Setup Check)

Key improvements to SKILL.md

  • Search Filter Syntax section — facetFilters OR/AND arrays, numericFilters strings, Unix timestamp date filtering, attribute selection
  • Analytics Key Details section — clickAnalytics: true guidance with which tools support it, data delay warning
  • Interpreting Results section — no-results rate benchmarks, click position patterns, low CTR diagnosis
  • Recommendation Thresholds section — 50/60/75 guidance, model parameter requirements table
  • Common Workflows — Search Quality Audit (6-step diagnosis), Recommendation Setup Check
  • algolia-cli cross-reference for write operations

Eval results

Eval Without Skill With Skill
Search with filters 17% 100%
Analytics report 33% 100%
Recommendations 33% 100%
Multi-step investigation 17% 100%
Date filtering + pagination 33% 100%
Average 27% 100%

The skill's value is overwhelmingly about MCP tool interface knowledge — every baseline fabricated wrong tool names, skipped the discovery step, and missed clickAnalytics: true.

Test plan

  • Ran 5 eval scenarios with-skill and without-skill
  • Verified 100% pass rate across all with-skill runs
  • Confirmed baselines fail on tool names, discovery workflow, and parameter conventions
  • Manual review of SKILL.md readability and completeness

🤖 Generated with Claude Code

…tent

Add evaluation framework and improve SKILL.md by surfacing critical details
from reference files into the main skill body for more robust behavior.

Changes to SKILL.md:
- Surface filter syntax (facetFilters OR/AND, numericFilters strings)
- Surface clickAnalytics: true guidance with supported tools list
- Surface recommendation thresholds (50/60/75) and model parameter table
- Add analytics interpretation benchmarks (no-results rate, click positions)
- Add Common Workflows (Search Quality Audit, Recommendation Setup Check)
- Add algolia-cli cross-reference for write operations

Evals (5 scenarios, 100% with-skill vs 27% baseline):
- Search with filters (facetFilters + numericFilters + facet values)
- Analytics report (tool selection + clickAnalytics + date params)
- Recommendations (model params + threshold + trending-items)
- Multi-step investigation (diagnose no-results + CTR comparison)
- Date filtering (Unix timestamps + pagination + combined filters)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@leonardocuco leonardocuco changed the title feat: enhance algolia-mcp skill with evals and improved docs feat: enhance algolia-mcp skill with evals and improved docs 🔬 Mar 18, 2026
@leonardocuco leonardocuco merged commit 9265e81 into main Mar 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant