Skip to content

v3.10.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 05 Oct 13:16
· 57 commits to main since this release
v3.10.0
4fda294
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

brew update && brew upgrade agentops · update skills · checksums · verify provenance

Highlights

AgentOps 3.10 is a release about the skills themselves. All 28 were audited, tested with Claude Code's plugin evaluator on Claude Opus 5.5 and edited where the test showed a gap. No command or skill name changes.

The test found two problems in 3.9.0. Most skills never loaded on a plain request: across 24 skills the matching one loaded in 22 of 72 runs. And when a skill did load, its most useful rule was often buried. In 3.10 the descriptions use the words a user would type, and each skill leads with the rules a model misses unaided. On the same 28 requests the agent met 349 of 372 practice criteria with 3.10, 292 with 3.9.0 and 269 with no plugin.

Loading is better and still incomplete. On requests written without sight of the descriptions, the matching skill loaded in 29 of 48 runs with 3.10 against 11 of 48 with 3.9.0. Nine skills did not load on their blind request. The evaluation report has every number and its limits.

Claude Exec is new: one prompt through headless claude -p with scoped permissions, one time bound and a reported exit status. The eval cases now ship in the repository, and the README says plainly that the skills are a kit to change.

Upgrade Notes

  • Update the plugin, the npx install or the source checkout as usual. Every 3.9 command and skill name still works, and nothing needs migrating.
  • In a source checkout, run git pull --ff-only, then rerun ao skills link with the selectors you used before, adding --skill claude-exec if you want the new skill. Without selectors it links every skill in the library. The ao binary changes only its version number.
  • Expect skills to load more often on plain requests. Nothing became mandatory, and invoking a skill by name works as before.
  • Validate no longer needs ao. With ao installed it binds the change to a content manifest; without it, it names the commit and the changed paths.
  • A plugin update replaces the installed copy of every skill. If you changed skills in place, move those changes to a fork or a source checkout first. The README's "Make it yours" section shows how.
  • Upgrading from 3.8 or earlier: read the 3.9 release notes first. That release removed skills, retired workflows and changed the Codex plugin layout.

At a Glance

Area What changes for the user
Skill loading Descriptions are rewritten in user phrasing. On blind requests the matching skill loaded in 29 of 48 runs, up from 11 of 48.
Skill content Each skill leads with the rules a model misses unaided, and most return a fixed output shape.
New skill claude-exec runs one prompt through headless Claude with scoped permissions and a time bound. The menu is 29 skills.
Validate Runs without ao and no longer stalls on a file outside its own directory.
Evidence evals/plugin-eval/ ships 29 behavior cases, 25 routing cases and a grader. The README has an Evidence section.
Customizing The README's "Make it yours" section covers cutting, rewriting, adding and measuring skills.

Product Areas

Skills and Workflows

  • Changed: All 28 skill descriptions are rewritten in the words a user would type, each within 26 words and 180 characters. In the behavior suite the matching skill loaded in 59 of 72 runs on 3.10 against 22 of 72 on 3.9.0. On blind requests it loaded in 29 of 48 against 11 of 48. council, implement and review did not load on their behavior request.
  • Changed: Each skill opens with the few rules an unaided model tends to miss, as a short checklist. Most now return a fixed output shape: Plan's one-slice block, Implement's handoff, Validate's verdict skeleton, Doc's handoff template with an unknown slot for each field and Memory's entry template are examples. Maintainer detail moved into references/.
  • Added: Claude Exec runs one caller-supplied prompt through headless Claude Code (claude -p). Tools and permission mode are scoped to the task, one time bound is spent by a retry instead of renewed, output is captured, and the exit status is reported as a fact. Its commands were run against Claude Code 2.1.282. It is caller-selected, like Codex Exec and AGY Native.
  • Changed: Validate judges from its own text. It no longer requires a file in another skill's directory, and it accepts the commit and changed paths as the subject's identity when ao is not installed. Its verdict order is written out and matches the CLI: an identity, freshness or coverage gap is NOT_PROVEN before a proven failure is FAIL.
  • Changed: Navigate counts a row as proven by cited evidence: the passing check that exercises the criterion, or a Validate PASS where a fresh review was required. In 3.9.0 it still demanded a Validate PASS for every row, which contradicted that release's rule that ordinary changes finish on checks and CI.
  • Changed: Premortem may run in the context that wrote the plan when no fresh context can be started. It must say that the independence check is missing, return its findings inline, and cannot save the result as a durable review.
  • Changed: Using GC sends a ready bead to the Mayor by mail. Direct gc sling is only for a city with no live Mayor, and whoever uses it takes over tending that run.
  • Changed: Security runs a scripted scan once per request; the repeat-until-quiet loop applies only to the manual hunt. A review reports a result for every vulnerability class (finding, clean or not assessed) and names a remediation class, with no plan, owner or ship decision.
  • Changed: AGY Native stops and reports when agy is missing, with no silent fallback to Codex, Claude or anything else. The retired ban on claude -p is removed from it and from the dispatch reference.
  • Changed: Reverse Engineer gives each row exactly one verdict and has a docs-only mode: a capability known only from documentation is unverified and cannot support steal.
  • Changed: Doc returns a handoff in the response when no location is named and creates no file. New CDLC handoffs still go only to protected non-Git storage.
  • Fixed: AGY Native claimed a five-minute default for --print-timeout. The CLI default is no limit, so the skill now requires an explicit timeout.
  • Fixed: Security's OWASP checklist said the redteam script covered secrets, input validation, SQL injection and XSS automatically. It scans only repository prompt and control surfaces. The references now name prompt_redteam.py scan instead of a subcommand that does not exist.
  • Fixed: Craft Goal stated its stop condition five times with two different pass counts. Interview said it creates no file while appending notes. Refactor's reference told readers to tidy messages and to self-grade PASS or FAIL. Each now says one thing.
  • Fixed: Idea Genie's validator paths resolve in an installed copy, and its portfolio shape is shown inline.
  • Fixed: Codex Exec says that an installed copy has no scripts/lib/codex-exec.sh and gives the direct codex exec form.
  • Fixed: Implement lists the evidence-orphan scan as not run when ao is absent, instead of leaving it out.

Eval, Validation, and Release Gates

  • Added: evals/plugin-eval/behavior/ has one case per skill for claude plugin eval: a request, four or five criteria and a check that the skill loaded. evals/plugin-eval/routing/ has 25 blind requests that check loading alone.
  • Added: evals/plugin-eval/grade.py grades a response against all of its criteria in one judge call. The evaluator's default Haiku judge failed criteria that responses plainly met, and its Opus judge cost several times the generation. The script agreed with the three-vote Opus judge on 439 of 458 criteria.
  • Added: The evaluation report and its scorecard record the 3.10 results per skill and per criterion, beside 3.9.0 and a no-plugin baseline.
  • Changed: Skill Eval explains how claude plugin eval relates to the repository probe runner. It says to confirm a skill loaded before reading a zero delta, to calibrate the judge first and to pass --no-publish, because the evaluator uploads its report by default.

Docs and Onboarding

  • Added: The README's "Make it yours" section says the skills are a kit: start with a few, cut what you override, rewrite what almost fits, write your own, mix libraries, change the workflows and measure what you change.
  • Added: The README's "Evidence" section reports the evaluation and its limits. PRODUCT.md records the same result beside the earlier coding pilot.
  • Changed: The README and the install guide show ao as optional for Validate.
  • Changed: The README is rebuilt around DevOps paved paths, with a concrete behavior-driven example, a "Beyond AgentOps" section for Gas City and the Agentic Coding Flywheel, and refreshed workflow diagrams.

Known Issues

  • Nine skills did not load on a blind request: doc, implement, navigate, reality-check, refactor, research, review, security and validate. An agent asked a direct question about pasted code often answers without loading a skill. Invoke the skill by name when you want its rules applied.
  • implement does not load on a quick fix request. Its description no longer claims every edit.
  • The evaluation covers one request per skill on one model. It shows that the guidance changes behavior on those requests, and nothing about outcomes on real tasks.
  • Two evaluation criteria fail by design with 3.10. AGY Native's fallback criterion was written against the old ban on print mode, which 3.10 removes from the skill. RPI's sign-off criterion contradicts the validation rule that 3.9.0 introduced. Both are kept unchanged so results stay comparable.
  • The Gas City command facts in Using GC were checked against a local gc that reports version edge. The 1.4.0 facts are unverified.
  • Four files in evals/agentops-core/ and tests/skills/test-finding-registry-flow.sh expect skill text that 3.9.0 already lacked. No gate runs them.

Full changelog

Full changelog

AgentOps 3.10 is a release about the skills themselves. All 28 were audited, tested with Claude Code's plugin evaluator on Claude Opus 5.5 and edited where the test showed a gap: descriptions now use the words a user would type, and each skill leads with the rules a model misses unaided. On the same 28 requests the agent met 349 of 372 practice criteria with 3.10, 292 with 3.9.0 and 269 with no plugin. On requests written blind, the matching skill loaded in 29 of 48 runs against 11 of 48 on 3.9.0; nine skills still did not load there. Claude Exec is new, the eval cases ship in the repository, and no command or skill name changes.

See the curated release notes for upgrade notes and known limits, and the evaluation report for every number.

Added

  • Claude Exec runs one caller-supplied prompt through headless Claude Code (claude -p) with tools and permission mode scoped to the task, one time bound that a retry spends instead of renewing, captured output and the exit status reported as a fact. It is caller-selected, like Codex Exec and AGY Native. The menu is now 29 skills.
  • evals/plugin-eval/ holds two suites for claude plugin eval: 29 behavior cases that compare an agent with and without the plugin, and 25 routing cases that check whether the matching skill loads on a request that never names it. evals/plugin-eval/grade.py grades a response against all of its criteria in one judge call.
  • The README gains "Make it yours", on cutting, rewriting, adding and measuring skills, and "Evidence", with the evaluation results and their limits.
  • Most skills gain a fixed output shape, such as Plan's one-slice block, Implement's handoff, Validate's verdict skeleton, Doc's handoff template and Memory's entry template.

Changed

  • Every skill description is rewritten in user phrasing, within 26 words and 180 characters. In the behavior suite the matching skill loaded in 59 of 72 runs on 3.10 against 22 of 72 on 3.9.0. Routing phrases that tests pin moved to frontmatter triggers on Implement and Premortem.
  • Each skill opens with the few rules an unaided model tends to miss. Maintainer detail moved into references: Codex Exec's guarded runner, Council's modes, Using GC's trust pre-seeding, the Research and Reverse Engineer pack and invocation contracts, Skill Builder's build mechanics, Skill Eval's readouts, Memory's toil evidence and Plan's resume and handoff rules.
  • Validate judges from its own text and runs without ao. It no longer requires a file in another skill's directory, accepts the commit and changed paths as the subject's identity, and writes out its verdict order.
  • Navigate counts a row as proven by cited evidence: the passing check that exercises the criterion, or a Validate PASS where a fresh review was required.
  • Premortem may run in the context that wrote the plan when no fresh context can be started. It must say the independence check is missing and cannot save the result as a durable review.
  • Using GC sends a ready bead to the Mayor by mail; direct gc sling is only for a city with no live Mayor.
  • Security runs a scripted scan once per request and keeps the repeat-until-quiet loop for the manual hunt. A review reports a result for every vulnerability class and names a remediation class, with no plan, owner or ship decision.
  • AGY Native stops and reports when agy is missing, with no silent fallback to another runtime. The retired ban on claude -p is removed from it and from the dispatch reference.
  • Reverse Engineer gives each row exactly one verdict and treats a capability known only from documentation as unverified.
  • Doc returns a handoff in the response when no location is named and creates no file. New CDLC handoffs still go only to protected non-Git storage.
  • Skill Eval explains how claude plugin eval relates to the repository probe runner, and says to confirm a skill loaded before reading a zero delta, to calibrate the judge first and to pass --no-publish.
  • The README and the install guide show ao as optional for Validate.

Fixed

  • AGY Native claimed a five-minute default for --print-timeout. The CLI default is no limit; the skill now requires an explicit timeout.
  • Security's OWASP checklist said the redteam script covered secrets, input validation, SQL injection and XSS automatically. It scans only repository prompt and control surfaces, and the references now name prompt_redteam.py scan instead of a subcommand that does not exist.
  • Craft Goal stated its stop condition five times with two different pass counts. Interview said it creates no file while appending notes. Refactor's reference told readers to tidy messages and to self-grade PASS or FAIL. Each now says one thing.
  • Idea Genie's validator paths resolve in an installed copy, and its portfolio shape is shown inline.
  • Codex Exec says that an installed copy has no scripts/lib/codex-exec.sh and gives the direct codex exec form.
  • Implement lists the evidence-orphan scan as not run when ao is absent.

Full Changelog: v3.9.0...v3.10.0

Assets 12
Loading