Skip to content

v0.16.0 — 15 domain skills, 500-case + holdout suites, six matcher fixes

Choose a tag to compare

View all tags
@anhnguyen0905 anhnguyen0905 released this 29 Jul 17:20
· 21 commits to main since this release
v0.16.0
c64a4c1

Highlights

  • 15 new domain skills ship with the plugin: accounting-bookkeeping, financial-audit, personal-finance, sme-operations, hr-recruiting, sales-pipeline-crm, customer-support-ops, ecommerce-operations, legal-contract-basics, training-curriculum-design, real-estate-analysis, ux-research-wireframing, manufacturing-ops-planning, warehouse-operations, json-data-wrangling. Ablation on the new 500-case suite: removing them drops the pass rate from 500/500 to 274/500 — these domains previously had zero retrievable coverage.
  • Six real matcher defects fixed in scripts/skill-match.mjs (each unit-tested): e-final plural stemming (cycles→cycl never met cycle), stacked name-word credit across query terms, an over-strict relevance floor for ultra-diagnostic terms (Incoterms/AOV/NPS — now rare by both IDF and document frequency so index growth can't relax it), duplicate index names selected twice, a semantically wrong precision-guard rule, and an over-categorical audit heuristic.
  • Honest evaluation: reports now carry MRR, per-scope recall, expectNone false-positive rate, and the sha256 of the exact index measured; the index builder warns about duplicate skill names.
  • Frozen independent holdout (tests/fixtures/scenarios-holdout.json, authored by a separate agent that never saw the tuned suite): first measurement 100/100, Hit@1 99.0%, MRR 0.995.
  • Step-2 facet table gains 9 non-engineering facets so the runtime classifier has vocabulary for accounting/audit, personal finance, SME ops, HR, legal, sales/support, education, real estate, and logistics requests.

Measured results

Suite Pass Hit@1 MRR
500-case scope suite 500/500 91.6% 0.946
Original 100-case suite 100/100 85.9% 0.903
Frozen independent holdout 100/100 99.0% 0.995

Full dual-review reconciliation (Claude Fable 5 + independent Codex round, 12 findings verified) lives in the run-local .codex-flow/skill-selection-test/EVALUATION-FINAL.md.

Also in this release

  • .github/workflows/publish.yml: pushing a v* tag publishes to npm automatically (OIDC trusted publishing, --provenance).
  • npm: @anhnguyen0905/codex-mcp@0.16.0 is published; the plugin's .mcp.json pins it.

Full changelog: https://github.com/anhnguyen0905/codex-mcp/blob/main/CHANGELOG.md

Assets 2
Loading