Telemachus

Evaluation and deployment for evolved agent skills.

Telemachus left home unformed and came back capable. That is the arc this project is named for — and the reason the name is not "another optimizer."

Skill files are now a trainable parameter. Two production efforts already optimize them and publish numbers. What nobody ships is the part that decides whether an evolved skill is actually better, and the path that gets it safely into production.

Status: open research problem. See How we work — async only, no meetings.

---

Start with what already exists, because it is more than you think

Anyone proposing this project must first know it is not an empty field.

Microsoft SkillOpt — MIT, ~15,700 stars, actively developed

"SkillOpt treats the skill file as a trainable parameter living outside a frozen target model." A forward/backward/update cycle: the frozen model runs a batch with the current skill; a separate optimizer model reads the resulting trajectories in reflection minibatches; bounded add/delete/replace edits are applied with validation gating that rejects edits below current performance, plus a rejected-edit buffer and textual learning rates. Artifact: best_skill.md.

Benchmark (GPT-5.5, direct chat)	Baseline → SkillOpt
Six-benchmark average	58.8 → 82.3
SpreadsheetBench	41.8 → 80.7
OfficeQA	33.1 → 72.1

Reported as winning all 52 cells (6 benchmarks × 7 models × 3 execution modes) against human-written skills, one-shot LLM skills, Trace2Skill, TextGrad, GEPA and EvoSkill. All numbers are Microsoft's own; we found no independent replication.

gskill — GEPA's own pipeline

"Given any GitHub repository, gskill creates important agent skill files for coding agents." Uses GEPA's optimize_anything plus SWE-smith task generation, which "converts a static repository into an active training environment." The artifact is literally .claude/skills/{repo_name}/SKILL.md.

Mini-SWE-Agent with gpt-5-mini at 300 rollouts: Jinja 55% → 82%, Bleve 24% → 93%. Transfer to other harnesses was uneven, and one cell regressed (sonnet-4-5 on Jinja, 100% → 98.5%).

The Hermes self-evolution plan — right design, broken implementation

NousResearch/hermes-agent-self-evolution specifies exactly the right thing: four tiers (SKILL.md files, tool descriptions, system-prompt sections, tool code), a gated deploy path, and the principle "Benchmarks are GATES, not fitness functions." Phase 1 shipped; phases 2–5 are planned. Three defects make shipped Phase 1 non-functional, each verifiable by reading both sides of the interface:

1. GEPA never runs. The code calls dspy.GEPA(metric=…, max_steps=…) inside a bare except Exception that falls back to MIPROv2. DSPy's GEPA has no max_steps parameter, asserts exactly one of max_metric_calls/max_full_evals/auto, requires a reflection_lm, and requires a five-argument metric. The call satisfies none of the four, so every run prints "Running GEPA optimization" and silently uses MIPROv2.
2. The skill text is not an optimizable parameter. It is passed as a runtime dspy.InputField and stored as a plain attribute, while DSPy optimizers mutate predictor signature instructions. An open issue confirms it empirically: MIPROv2 moved holdout behaviour +7.1% while the saved evolved_skill.md was byte-identical to the baseline.
3. Every run fails the constraint gate. Validation runs on the frontmatter-stripped body, while the structure check requires the text to start with --- and contain name:/description: — so a bare body never passes, and the code takes the "FAILED constraints — not deploying" branch every time.

Its headline +39.5% is the better of two held-out examples, produced by BootstrapFewShot, not GEPA; the other example moved 0.0%, and the report's own body states the average was +20.7%. n = 2.

None of this is a criticism of the design. The design is good, which is why it is worth finishing properly.

And the demand is documented

"From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills" (arXiv:2607.01456) analysed 238 real-world skills and found over 99% contain at least one skill smell, and "once introduced, skill smells rarely disappear as skills evolve."

---

So what is actually left?

Four things, precisely.

1. Sets of skills, not one skill

Both existing tools optimize a single skill file. Nobody optimizes a library — including the routing description: fields that decide which skill fires. That is where skill-selection failures actually live, and it is a different optimization problem.

2. Harness-in-the-loop, not chat mode

SkillOpt's headline table is direct chat mode. gskill's transfer to a real coding harness was uneven and produced a regression. Optimizing against the harness the skill will actually run in is unclaimed — and here is why it matters:

Harness choice changes Pass@1 by 27.4 percentage points. Model choice changes it by 29.4. — Claw-SWE-Bench, arXiv:2606.12344

A skill tuned in chat mode and deployed into a harness has crossed a boundary nearly as large as changing the model.

3. The deployment half — which everyone specifies and nobody ships

Hermes's plan specifies pytest gate → benchmark subset → full benchmark → top-3 only, PR-only deployment. It built none of it: no PR automation, run_test_suite defined and never called, zero benchmark code. Neither SkillOpt nor gskill claims this either.

4. An independent replication

SkillOpt's 52-cell sweep is self-reported with no third-party replication we could find. A credible independent replication is itself a shippable, citable contribution — and a much better first project than yet another optimizer.

---

v0.1 scope

1. A correct GEPA wiring, as the reference implementation of what Hermes got wrong: exactly one of max_metric_calls/max_full_evals/auto; a reflection_lm; a five-argument metric (gold, pred, trace, pred_name, pred_trace) returning score and feedback text — because GEPA's textual feedback is the entire mechanism, and a metric that returns a bare float removes it; and the skill body exposed as an optimizable predictor instruction, never an InputField.
2. A skill-smell linter implementing the arXiv:2607.01456 taxonomy (13 higher-level and 44 lower-level components) as a cheap pre-gate — run on the reassembled file (frontmatter + body), which is precisely the bug that kills Hermes's Phase 1.
3. A real holdout evaluator with the LLM judge actually invoked, plus a significance check that refuses to report a delta at n=2.
4. A gated deploy path: pytest → smell gate → holdout → benchmark gate → open a PR carrying the diff, before/after numbers, gate results, and the optimizer/model/seed used.
5. Pluggable backends — SkillOpt, GEPA, MIPROv2 — and a head-to-head report on the same skills. Publish the losses.

---

The strongest objection, and the honest answer

The objection: Microsoft SkillOpt already does the core thing, is MIT, has ~15,700 stars, is actively pushed, and reports beating GEPA across all 52 evaluation cells. Building another skill optimizer risks being a worse SkillOpt.

The answer is to not build another optimizer. Telemachus is the evaluation and deployment layer around whichever optimizer wins. Support SkillOpt as a first-class backend. Compete only on the four things above: multi-skill routing, harness-in-the-loop evaluation, smell remediation, and gated PR deployment.

And state plainly, in the README and in every report: if SkillOpt beats us on its own benchmarks, we will say so.

---

A naming correction, since it circulates

The mechanism behind this work is GEPA — Genetic-Pareto, from "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" (arXiv:2507.19457, ICLR 2026 Oral). It maintains a candidate pool, selects by Pareto illumination (keep candidates that win on at least one training instance; prune strictly dominated ones), mutates by having an LLM read execution traces and propose new instructions, and merges complementary lineages. Reported to outperform GRPO by ~6% on average using up to 35× fewer rollouts.

JEPA is a different thing entirely — Joint-Embedding Predictive Architecture (arXiv:2301.08243), a computer-vision self-supervised method with no connection to prompt optimization. A case-sensitive grep of the Hermes repo returns 80 occurrences of GEPA and zero of JEPA. Any text saying "JEPA" in this context is wrong.

---

Getting started

You do not need permission and you do not need to talk to anyone first.

1. Read this README and HOW-WE-WORK.md. Five minutes.
2. Pick an issue labelled good first issue. Each one is written to be self-contained — enough detail to start without asking questions.
3. Comment "I'm taking this." That is the whole claiming process. If someone claimed it and has gone quiet for two weeks, take it anyway.
4. Fork, branch, open a small pull request. One idea per PR. A 40-line PR gets read the same day; a 1,000-line one waits for a weekend.
5. Add yourself to CONTRIBUTORS.md in that first PR — your name and what you are working on.

Have a bigger idea instead? Open a pull request adding rfcs/NNNN-your-idea.md using the template, and argue for it in the PR. That is how anything structural gets decided here.

Not sure where you fit? Open an issue saying what you know how to do and what you would enjoy. Someone will point you at the part of this that needs you.

A good first week

• Take any skill file you already use, run it through SkillOpt, and report what changed and whether it actually got better. Publish it even if the answer is "no difference".

How we work

• Everything happens in issues and pull requests. No meetings, ever.
• Anything structural starts as an RFC — a PR adding rfcs/NNNN-title.md.
• Claim an issue by commenting on it. No permission needed.
• CONTRIBUTORS.md names people and their workstream. Add yourself in your first PR.
• Start with anything labelled good first issue.

Licence

Apache 2.0.