Repository navigation
Inspector V2 Working Group Meeting - Sept 30, 2026 #3368
cliffhall
started this conversation in
Meeting Notes - Inspector V2 WG
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Inspector V2 Working Group Meeting - Sept 30, 2026
Agenda
Attendees
Discussion
A long two-person session. It opened on the 2.9.0 smoke test and the standing problem of testing on Windows, spent its middle on how agents behave in this repo — skills that may not be loading, review cycles that grow instead of converge — and @BobDickinson's eval harness for
mcpdo, then closed on the shape of extensions, progressive disclosure, and gettingmcpdoa proper page in the docs.2.9.0 smoke test: minor finds, all handled. @cliffhall ran a quality-gate review and posted it to the PR. The smoke test across the release turned up a few things; the ones that needed it were fixed on
v2/mainand merged on into the release branch, and everything still outstanding has an open issue. A couple of items couldn't be tested directly. A large share of this milestone's issues came from a QA engineer friend's static analysis report, with a pile of similar size waiting for the next release. Some findings are the "picky reviewer" kind that don't need handling, but some were good catches — the job is to smoke-test and validate each one rather than fix every theoretical edge case.We still can't test on Windows. @cliffhall wants Ola to try the release on Windows — and is uneasy that Windows testing always falls to her. @BobDickinson has had plenty of Windows-specific PR feedback (the secure token storage work drew a lot of "this doesn't work this way on Windows") without ever running it there. Options weighed: BrowserStack covers browsers on other platforms, but not "boot this Windows, install this package, try it in every browser"; Parallels works but costs a few hundred dollars plus a Windows license; VMware still doesn't run on Apple Silicon; multi-boot partitions are something nobody wants to give up disk for. @BobDickinson's most promising lead is GitHub Actions, which he has used to build and sign on all three platforms — a scripted test is comparable to a build, and Playwright can drive the UI. @cliffhall wants this "amped off to CI," not local VMs, and if a product exists that does it, he'll ask the Linux Foundation for a license: "they seem pretty cool with giving us what we need."
Copilot and the rules: unsigned commits and broken CI. @BobDickinson has seen unsigned commits from Copilot, and twice in recent days it broke CI — once by skipping
prettierafter a final tweak, once by committing without running the coverage gate. @cliffhall suspects the unsigned ones predate the skills, so the fix is to check whether recent commits are signed first, then squash and sign the branch. Formatting and coverage before commit are rules inAGENTS.md, not just skills, so a miss there is worth interrogating: ask the agent why it dropped the ball and let it do the analysis. If something is spelled out in a way Claude follows and Copilot doesn't, that's a chink to shore up.A likely culprit surfaced on the spot: @BobDickinson's agent doesn't request a Copilot review on its own — he does it manually every time — which @cliffhall called "definitely a sign that something is off the rails," since the Copilot CLI evals say
pr-flowshould load. The agent had been running for a week and a half; he had rebased but never restarted it. "It's like a Windows machine. You got to give it a boot every week." @cliffhall floated a self-health-check skill that suggests a fresh session after a while.Review cycles that grow instead of converge. @BobDickinson saw the same thing in both of his big PRs: the reviewer kept picking at the seams of a contract — token storage split between a JSON file and the keychain ought to be atomic — and his agent kept patching each symptom. After many rounds he stopped it and pointed at the root cause; the next round deleted about 500 lines. Agents "can zoom way, way out, but can't zoom out just a little bit." Some findings simply weren't worth having — a ping that answers during shutdown, two processes racing on a file lock — where returning an error is fine. @cliffhall had hit the reviewer half of this: a 40-round run that turned out to be twenty-odd rounds of bugs in a guard nobody asked for. That's why
pr-flownow tells the agent to fix defects in what the PR added and decline pre-existing behavior, new capabilities, and hardening beyond the issue, filing anything worth doing as its own issue. @BobDickinson noted his case is partly the other side — the author agent not recognizing a pattern — and partly that he could have stated the goal better.Evals for
mcpdothat test behavior, not just triggering. @BobDickinson adopted the skills-eval approach but keeps his evals outside the skill directory, sincemcpdo's skill is public and a skills install would copy them in. Two findings. First, because agents already know about MCP, "what MCP servers do I have?" gets answered without the skill — and one line inAGENTS.mdfixes it, somcpdo's agent-help command gained a mode that prints that line for an agent to install. Second, "did the skill fire" is binary, so he instrumentedmcpdowith an event log and added runtime cases: did it call the CLI, with the right parameters? About eight of each, three runs apiece in both Copilot and Claude Code, about 25 minutes — run it when the skill or any command help text changes.Those runtime tests found real bugs in the two trickiest flows. Without a TTY, a command can never wait for more input — Claude Code would hang on a login unless it was backgrounded. So OAuth now hands back the link and exits; the agent can poll, and any later command completes the flow. Elicitation returns the whole form — fields, types, defaults — plus a separate completion command to supply the values one at a time or all at once. Part of the guidance lives in the skill and part in the response text, since only the agent ever sees that response. What's under test is the combination: skill, help text, and responses together.
Will better models make skills unnecessary? At work, @BobDickinson runs with no skills at all, and from Opus 4.8 up the model works out local conventions like a new human developer would; older models need the hand-holding. @cliffhall's view is that skills are now part of the codebase, with their own tests, and refining them will be continual — not a one-way trim. Prompting styles change between model versions (all-caps "never" rules gave way to explaining the goal), models shift after post-training, and teams sometimes go back to an older model that communicates better. There's no mechanism for mapping skills to models. @BobDickinson's broader point: a real product would run its evals across several models and re-run them whenever a new one lands. On cost, he noted per-token API billing has no flat-rate cushion — about $1,500 of Linux Foundation credits went into his two PRs alone.
Events is coming, and is a candidate for the automated pipeline. @BobDickinson flagged the events working group's announcement that OpenAI is supporting it — "slightly behind skills," and likely an official extension soon. It doesn't look bad: subscribe with an ordinary MCP call, then poll, long-poll, or take a webhook. Webhooks are the tricky part, needing an externally reachable URL forwarded to something the Inspector listens on. It's "bigger than a breadbox," so he'll file an issue and see how the automated issue → PR process handles something that size.
What counts as an official extension is still unsettled. @cliffhall pointed at the maintainers'-day minutes: extensions have no rules yet, tasks was rushed into the protocol and is now deprecated in favor of an extension, and the signal for "officially supported" is currently an org repo — with abandoned ones needing removal and a canonical list on the docs site, like the SDK list. @BobDickinson's confusion from preparing an internal talk: the docs still imply tasks is official. Both agreed that some extensions — skills, events — are effectively standards-track and every good client should support them, while niche ones won't go in the Inspector. Luca, meanwhile, wants to move the Inspector's modern-tasks code into the tasks extension's SDK, and the nightly SDK watch will pick that up once there's a package to watch.
Keep the core simple. @cliffhall's lesson from the groups-and-filters SEPs: the protocol is hard to back things out of, so protect it — and then sampling-with-tools went in, turned a simple feature into a feedback loop, and sampling is now deprecated. Extensions are what let the ecosystem try things without changing the core. @BobDickinson proposed an OSI-style stack: a plumbing layer (connect, negotiate, send and receive) with SDKs that almost never change; a conventions layer above it (tool lists, calls, schemas); and extensions on top.
Progressive disclosure is the answer to gateway bloat. In a gateway, @BobDickinson finds the bloat isn't the tool list but the input schemas — 10k is common, and he saw one at 64k. A model only needs to know what tools exist to decide it needs one; it can ask for the schema then. His gateway demo shows it: a find-tool search alone was clumsy, but giving the agent a catalog of tool sets — just server names, descriptions, and truncated instructions — first made it reason far better. That's how skills work too. @cliffhall: it's exactly what groups were for — filtering was declined, so the scope narrowed to grouping alone, and that working group became the progressive disclosure WG. Getting 11 tools out of a thousand is the big win.
This is why the Inspector moves early now. Extensions like skills and events start from people with real problems — events got more urgent once the protocol's async callbacks went away — who build a solution and prove it before the fine points get debated. @cliffhall: getting skills over MCP into the Inspector fast is exactly what the old codebase couldn't afford, and now "if it changes we will update it and if it goes away we will remove it," with no regressions. Goose was the first client to adopt the MCP Apps extension after OpenAI's early version, and clients will lag; the Inspector can be the reference implementation a server author tests against the moment they add events. The idea from last time stands: let extension authors tell us when theirs is ready.
Agents on the contributor Discord. @cliffhall has access to Den's Guild Bridge MCP server, which reaches the contributor Discord, and wants an agent to produce a brief on what each working group is doing. He also floated a per-channel bot that answers questions from the Inspector repo, docs, and spec — as when a user asked whether the redirect URL is surfaced anywhere, and an agent gave the answer and filed an issue to add it — with prompt injection as the obvious risk.
mcpdogets a demo and a docs page. @BobDickinson has a demo deck: what it is, when to use it, a human demo, then the agentic use case and a demo of that, in about ten minutes. He's rehearsed it twice and will record it today for @cliffhall to review. The 2.10 milestone has an issue to documentmcpdoand some Docker work; @cliffhall thinksmcpdodeserves a first-class page, plus a security section that gathers the hints currently scattered across pages. @BobDickinson is best placed to write it. Part of the docs tree needs core-maintainer approval as well as docs-maintainer approval, so there's about a week to campaign for reviews. And @cliffhall has a contact at the AIF / Linux Foundation who can put videos on their YouTube channel, so the demo could be embedded in the docs — possibly the start of a series covering the TUI and CLI too.Next Steps
prettier, the coverage gate, and requesting a review; check whether recent commits are signed, then squash and sign the branch; look back over the larger automated PRs for the symptom-patching pattern; file an issue for events support as a test of the automated pipeline; record themcpdodemo; write the first-classmcpdodocs page with a consolidated security section for 2.10.Transcript
Operational Details
v2/main- v2 Inspector, merged tomainfor milestone releasesv1/main- deprecated v1 Inspector, urgent security patches onlyv2v2Repo contributions
WG Leads:
All reactions