Evaluation harness for the dotnet-skills Claude Code plugin. Tests whether AI coding assistants correctly activate and benefit from .NET development skills.
- Skill Activation Accuracy - Given a user's .NET development task, does the model correctly identify which skill(s) to use from the catalog of 31 skills?
- Skill Effectiveness - Once activated, does a skill actually improve the quality of generated code compared to a baseline (no skill)?
- Size Impact - Several skills exceed Claude's recommended 500-line limit. Does this hurt effectiveness, or are larger skills fine?
The dotnet-skills plugin is consumed by multiple AI coding assistants:
- Claude Code (primary) - reads SKILL.md on activation, can read additional files via progressive disclosure
- GitHub Copilot - supports agent skills with multi-file progressive disclosure (since Dec 2025)
- OpenCode - reads SKILL.md from
~/.config/opencode/skills/
All three platforms support the same skill directory structure with progressive disclosure: SKILL.md serves as the entry point (<500 lines recommended), with sibling reference files loaded on demand. See Claude skill best practices.
The initial evaluation targets the 5 Akka.NET skills, which are the most critical and also the most likely to have overlap/confusion issues:
| Skill | Lines | Over 500-line limit? |
|---|---|---|
| akka-net-best-practices | 1123 | 2.2x |
| akka-net-testing-patterns | 1280 | 2.6x |
| akka-hosting-actor-patterns | 614 | 1.2x |
| akka-net-management | 685 | 1.4x |
| akka-net-aspire-configuration | 719 | 1.4x |
This harness connects to Claude models via OpenRouter + LiteLLM for raw API calls. Claude Code cannot invoke or test itself programmatically. We simulate skill discovery by testing three mechanisms:
- Tool-based (
tool) — model gets aSkilltool listing all skill names + descriptions (mirrors Claude Code's plugin system) - Compressed index (
compressed) — the terse Vercel-style routing snippet (~15 lines) injected into the system prompt - Fat index (
fat) — all 31 skill names + full descriptions injected into the system prompt
All mechanisms share the same neutral system prompt with no mention of skills or evaluation. Test prompts are goal-oriented developer conversations with C# code snippets — not quiz-style questions about skills.
For effectiveness testing, full SKILL.md content is injected as context.
Full run: 23 Akka.NET positive cases + 8 non-Akka negative cases. The tool mechanism was dropped from comparison — it has 100% activation rate (including 100% FPR on negatives), which measures tool-calling bias rather than skill discovery quality.
| Metric | Haiku | Sonnet | Opus |
|---|---|---|---|
| Activation Rate | 45.2% | 41.9% | 16.1% |
| True Positive Rate | 56.5% | 56.5% | 17.4% |
| False Positive Rate | 12.5% | 0.0% | 12.5% |
| Accuracy (when activated) | 46.2% | 84.6% | 0.0% |
| Mean Prompt Tokens | 688 | 688 | 689 |
| Mean Completion Tokens | 2,315 | 2,337 | 2,770 |
| Mean Total Tokens | 3,002 | 3,025 | 3,459 |
| Metric | Haiku | Sonnet | Opus |
|---|---|---|---|
| Activation Rate | 16.1% | 19.4% | 12.9% |
| True Positive Rate | 13.0% | 21.7% | 13.0% |
| False Positive Rate | 25.0% | 12.5% | 12.5% |
| Accuracy (when activated) | 33.3% | 60.0% | 0.0% |
| Mean Prompt Tokens | 1,956 | 1,956 | 1,957 |
| Mean Completion Tokens | 2,262 | 2,123 | 2,643 |
| Mean Total Tokens | 4,217 | 4,078 | 4,600 |
-
Sonnet + compressed index is the clear winner. 56.5% TPR, 0% FPR, 84.6% accuracy when activated — best activation rate, best precision, zero false positives, all at the lowest token cost (~3K tokens/case).
-
Compressed index outperforms fat index across all models. Despite having ~3x fewer tokens (688 vs 1,956 prompt tokens), compressed achieves higher TPR and better accuracy on every model. The fat index's wall of text paradoxically makes models less likely to reference skills — information overload suppresses activation.
-
Opus essentially doesn't activate from index-based discovery. Every single "activation" across both mechanisms was a "serialization" substring false positive — 0% real accuracy. Opus treats the skill index as informational context but never cites specific skill names in responses. This is a detection methodology issue, not necessarily a model quality issue — Opus may be incorporating the knowledge without naming the skill.
-
Haiku and Sonnet have identical TPR (56.5%) on compressed, but Sonnet is far more accurate when it activates (84.6% vs 46.2%). Haiku often activates related-but-wrong Akka skills.
-
"serialization" false positive in detection. The skill name
serializationis short enough that it gets substring-matched when models discuss serialization concepts in their responses. This is the only "activation" Opus shows. Need to filter short/generic skill names or require citation format. -
Sonnet compressed has perfect negative discrimination. 0% FPR — never activated on any of the 8 non-Akka negative cases. Haiku and Opus both have 12.5% FPR on compressed (1 false positive each, both "serialization").
Does having a skill loaded produce better code? For each of 15 tasks across 5 Akka.NET skills, we generate code with and without the skill injected as context. An LLM judge (Sonnet) scores both outputs on a per-skill rubric with randomized A/B ordering to avoid position bias.
| Metric | Value |
|---|---|
| Total Cases | 15 |
| Skill Wins | 13 |
| Baseline Wins | 2 |
| Ties | 0 |
| Win Rate | 86.7% |
| Mean Baseline Score | 2.80 / 5 |
| Mean Enhanced Score | 4.40 / 5 |
| Mean Improvement | +1.60 |
| Skill | Cases | Win Rate | Mean Improvement |
|---|---|---|---|
| akka-hosting-actor-patterns | 3 | 100.0% | +2.00 |
| akka-net-aspire-configuration | 3 | 100.0% | +2.33 |
| akka-net-testing-patterns | 3 | 100.0% | +2.33 |
| akka-net-management | 3 | 66.7% | +1.00 |
| akka-net-best-practices | 3 | 66.7% | +0.33 |
-
Skills overwhelmingly improve output quality. 86.7% win rate with a +1.60 mean improvement on a 1-5 scale. The judge provides detailed, criterion-by-criterion reasoning referencing specific code patterns.
-
Specialized skills show the largest gains. Testing patterns (+2.33), Aspire configuration (+2.33), and hosting patterns (+2.00) teach the model things it genuinely doesn't know — modern Akka.Hosting.TestKit, Aspire orchestration patterns, GenericChildPerEntityParent. The baseline model defaults to legacy patterns (Akka.TestKit.Xunit2, static seed nodes, manual Props).
-
General best-practices skill shows weakest improvement (+0.33). Sonnet already knows general Akka.NET patterns like DistributedPubSub vs EventStream and supervision strategies. The skill helps at the margins but doesn't transform output quality.
-
The two baseline wins are defensible. eff-002 (RSS polling error handling): the baseline produced cleaner CancellationToken lifecycle management. eff-012 (health endpoints): the baseline used built-in Akka.Hosting health check methods while the enhanced version over-engineered with custom implementations. In both cases, the skill's extra context may have pushed the model toward more complex solutions when simpler ones were better.
Do oversized skills (>500 lines) hurt or help? All 5 Akka skills exceed Claude's recommended 500-line SKILL.md limit. We ran effectiveness evals twice per skill: once with full content, once truncated to 500 lines.
| Skill | Lines | Full Improvement | Truncated (500) Improvement | Winner |
|---|---|---|---|---|
| akka-net-best-practices | 1123 | +0.33 | +0.67 | Truncated |
| akka-net-testing-patterns | 1280 | +2.67 | +2.67 | Same |
| akka-hosting-actor-patterns | 614 | +2.67 | +2.33 | Full (+0.33) |
| akka-net-management | 685 | +0.67 | +1.67 | Truncated |
| akka-net-aspire-configuration | 719 | +2.33 | +1.67 | Full (+0.67) |
-
Truncation doesn't consistently help or hurt — it depends on the skill. Two skills perform better truncated, one is identical, and two perform better with full content.
-
For management, truncation dramatically improved results. Win rate jumped from 66.7% to 100%, and the health endpoint task (eff-012) that was a baseline win at full size became an enhanced win when truncated. The extra content beyond 500 lines was actively confusing the model.
-
Specialized skills benefit from full content. Aspire configuration (+2.33 vs +1.67) and hosting patterns (+2.67 vs +2.33) — the model genuinely lacks training data for these topics, so the extra examples and reference material add real value.
-
General-knowledge skills perform better truncated. Best-practices and management cover topics where Sonnet has decent baseline knowledge. The extra content beyond 500 lines becomes noise that dilutes the signal.
-
Progressive disclosure is the right solution. Rather than choosing between "full" and "truncated", skills should keep their core patterns in SKILL.md (<500 lines) and move detailed examples/reference material into sibling files that are loaded on demand. This gives the model focused guidance by default while making specialized content available when the task calls for it. All three platforms (Claude Code, Copilot, OpenCode) support this pattern.
# Clone
git clone https://github.com/Aaronontheweb/dotnet-skills-evals.git
cd dotnet-skills-evals
# Install (requires Python 3.11+)
uv sync
# Configure API key
cp .env.example .env
# Edit .env and add your OPENROUTER_API_KEY
# The dotnet-skills repo is cloned automatically on first run.
# Default path: ~/repositories/dotnet-skills
# Override with: export DOTNET_SKILLS_REPO=/path/to/dotnet-skills# Realistic skill activation eval (all 3 discovery mechanisms)
dotnet-evals eval-activation --model haiku --dataset datasets/activation/akka_test_cases.jsonl
# Run specific mechanism(s)
dotnet-evals eval-activation --model sonnet --mechanism compressed --mechanism fat \
--dataset datasets/activation/akka_test_cases.jsonl
# Skill effectiveness (does the skill improve output quality?)
dotnet-evals eval-effectiveness --model sonnet --skill akka-net-best-practices
# Size impact (full skill vs truncated to 500 lines)
dotnet-evals eval-size --model sonnet --skill akka-net-best-practices --max-lines 500
# Compare authoring strategies (original vs condensed vs progressive disclosure)
dotnet-evals scaffold-variants # Create template dirs for variant authoring
dotnet-evals eval-variants --model sonnet --skill akka-net-best-practices --dataset datasets/effectiveness/akka_test_cases.jsonl
# List all skills with sizes
dotnet-evals list-skillsTest cases use goal-oriented developer prompts with C# code snippets (not documentation-style questions):
{
"id": "act-001",
"user_prompt": "I have a notification system where actors publish events... [includes code snippet]",
"expected_skills": ["akka-net-best-practices"],
"acceptable_skills": [],
"should_activate": true,
"category": "best-practices-pubsub"
}{
"id": "eff-001",
"skill_name": "akka-net-best-practices",
"task": "Build an actor that publishes user notifications across cluster nodes",
"rubric_file": "rubrics/akka-best-practices.yaml"
}Per-skill evaluation criteria with weighted scoring dimensions.
- Reformat oversized skills with progressive disclosure — Split the 5 Akka skills (614-1280 lines) into <500 line SKILL.md files with sibling reference files. Eval data shows this should preserve the gains for specialized content while eliminating noise for general-knowledge topics.
- Run effectiveness evals on Haiku and Opus — Current results are Sonnet-only. Haiku may need more skill guidance (per Claude docs: "Does the Skill provide enough guidance?"); Opus may need less.
- Expand beyond Akka skills — Test the 26 non-Akka skills (e.g., csharp-coding-standards at 1510 lines, testcontainers, aspire-integration-testing) to see if findings generalize.
- Variant comparison — Test original vs condensed vs progressive disclosure authoring strategies to quantify the improvement from restructuring.
- Improve activation eval methodology — The current text-detection approach has fundamental issues (see activation results above). Consider alternative approaches: semantic similarity matching, structured output with skill selection, or skip activation entirely and focus on effectiveness.
| Eval Type | Cases | API Calls | Cost (Sonnet) |
|---|---|---|---|
| Activation | ~31 | 31 | ~$0.50 |
| Effectiveness | ~20 | 60 | ~$3-5 |
| Size impact | ~10 | 60 | ~$3-5 |
| Total MVP | ~$10-15 |
Use --model haiku for cheaper iteration during dataset development.
Apache 2.0