feat: add CLI expertise bundle overlay with on-demand expert agent - #271
Merged
Conversation
Adds comprehensive user-facing documentation for the experimental /provider model-pinning feature, covering: - Mental model: each provider entry combines one account + one model - Worked example with two real Anthropic accounts and four models - Configuration, naming guidance, and default precedence via priority - Interactive usage: /provider, /provider use <name>, /provider auto - Limits: same-vendor only, session-only, top-level-only, orchestrator-dependent - Comparison table for three related but distinct 'provider' surfaces All instructions verified against live container with two real Anthropic accounts. Closes #267 and #268. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Add a composable bundle overlay that provides every CLI session with an expert agent that answers questions about the CLI itself. The expert reads from six existing docs (PROVIDER_PINNING, CONTEXT_LOADING, INTERACTIVE_MODE, GOAL_COMMAND, OUTPUT_FORMATS, SPAWN_PRECEDENCE), loaded on-demand when spawned (16K tokens), paired with a thin awareness routing pointer (317 tokens) loaded every session to prevent guessing. Bundle structure (repo root, matching layout of amplifier/core/foundation): - behaviors/cli-expertise.yaml: composed behavior including the agent - agents/cli-expert.md: context-sink expert with @-mentions of six docs - context/cli-awareness.md: routing pointer, deliberately bounded token budget - bundle.md: marker file for nested-bundle detection Modified: - pyproject.toml: force-include bundle dirs into installed wheel at amplifier_app_cli/_bundle/ (not package root, to avoid shadow effects) - runtime/config.py: _build_app_cli_behaviors() resolves bundle from installed package location, composes behavior only (never bundle.md) Fix: commands/run.py error messages now point to valid commands (/provider list, /provider add, /provider --help instead of /provider use) Known costs documented in PR body: docs/ duplication in wheel, root bundle.md affects repo as nested-bundle, sdist untested. Refs: #267, #268, #269 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Brian Krabach (bkrabach)
marked this pull request as ready for review
August 14, 2026 19:30
Brian Krabach (bkrabach)
added a commit
that referenced
this pull request
Aug 14, 2026
Add two new read-only diagnostic subcommands to the in-session /provider slash
command, closing a diagnostic dead end when provider pins fail.
BEFORE: /provider supported exactly three forms—status, use <name>, auto. When a
pin failed with ✗ cannot pin conversation provider 'fable': it is not mounted in
this session, the user had no in-session way to find out WHY. Bad API key? Wrong
model name? They had to leave the session and run amplifier provider test /
amplifier provider models in a shell.
AFTER: Two new read-only diagnostic forms, callable mid-conversation:
• /provider test [name] — connectivity check against the live mounted
providers. No name tests all, concurrently, bounded at 15s per provider.
• /provider models [name] — lists models a provider actually offers. No name
uses the active provider.
FILES
• amplifier_app_cli/provider_diagnostics.py (new, 114 lines)
Shared primitives: invoke_list_models(), ProviderTestResult,
test_provider_connectivity(), format_model_line(). Async-aware;
15s timeout per provider; single source of truth for connectivity semantics.
• amplifier_app_cli/provider_loader.py (+13/-4)
get_provider_models() now delegates to invoke_list_models() rather than
keeping its own inline implementation. One less independent implementation
of the same mechanic.
• amplifier_app_cli/main.py (+171/-4)
Slash command dispatch wiring, two new handlers (_handle_provider_test,
_handle_provider_models), usage string generation, error messages.
Updated /help description; unknown-subcommand error lists all five forms.
• tests/test_provider_test_and_models_command.py (new, 402 lines, 29 tests)
Unit and integration coverage for both commands, all error paths.
THREE DESIGN DECISIONS
1. MOUNTED PROVIDER REALITY, NOT DISK CONFIG
These operate on THIS SESSION'S MOUNTED PROVIDERS, not settings.yaml.
The existing amplifier provider test/models CLI commands read config from
disk and instantiate a throwaway provider. The slash versions read
coordinator.get("providers") — the live mounted instances. That is the
entire reason to run them mid-conversation instead of in a shell: if the
mounted set and the on-disk config disagree, the session's reality is what
the user needs. The session's mounted instances are never closed after
testing (asserted in tests); only the CLI's throwaway instances are.
2. NOT GATED ON CONVERSATION.PROVIDER_PIN CAPABILITY
/provider use and /provider auto require pin capability and refuse loudly
without it. These two are pure read-only diagnostics and are useful
precisely WHEN pinning is unavailable or failing — gating them would remove
the diagnostic exactly when it is needed most. Dispatched before the
capability gate; tested with pin=None.
3. SHARED LOGIC AT THE RIGHT SEAM
The existing click commands in commands/provider.py were left untouched —
they are disk-config-driven and their tests mock at that boundary. What is
genuinely shared is the lower mechanic ("call list_models() on a provider
object, async-aware") and the definition of "connectivity is OK", both of
which now live once in provider_diagnostics.py. So the two surfaces cannot
drift on what "OK" means.
CONCURRENCY & UX
/provider test with no name runs all targets via asyncio.gather, each
wrapped in asyncio.wait_for(..., 15s) so one hung provider cannot freeze
the REPL. Progress line prints before await. Wall clock tracks the slowest
provider (~1.1s cold, ~0.5s warm), not the sum.
Markup safety: [name] in usage strings was parsed as Rich markup and
silently vanished. Switched to <name>, matching /provider use <name> style.
EVIDENCE
• 29 new tests; 1413 passed / 1 skipped / 13 deselected / 1 xfailed
Same pass count as baseline (verified re-run against original stash).
ruff clean; pyright error count identical to baseline (none in new code).
• Verified end-to-end in live TUI (build 2026.08.14-25e9944), 8 checks:
✓ /help lists all five forms
✓ /provider models anthropic-sonnet — real network, 3 models, aligned
✓ /provider models (no name) — defaults to priority winner, byte-identical
✓ /provider test anthropic-sonnet — ✓ 0.1s, singular grammar
✓ /provider test (all four) — 4/4 ✓, 3+3+37+31 models across vendors
✓ Concurrency confirmed — wall clock ≠ sum(individual)
✓ /provider test nope — clean refusal, no traceback
✓ Pin regression intact: pin → pinned footer → unpin → priority winner
OUT OF SCOPE
add, edit, remove, install, manage were NOT added. They mutate persistent
config, a different concern from "who is answering me right now", and
mixing them in would blur the line: amplifier provider changes what is
available tomorrow; /provider changes who is answering now.
Closes: #267, #268, #269, #271 (extends /provider feature)
Generated with [Amplifier](https://github.com/microsoft/amplifier)
Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a composable bundle overlay that provides every CLI session with an expert agent answering questions about the CLI itself. The expert reads from six existing docs (PROVIDER_PINNING, CONTEXT_LOADING, INTERACTIVE_MODE, GOAL_COMMAND, OUTPUT_FORMATS, SPAWN_PRECEDENCE), loaded on-demand when spawned (~16K tokens), paired with a thin awareness routing pointer (~317 tokens) loaded every session to prevent guessing and steer toward delegation.
Test Plan
uv tool install git+...pathsite-packages/amplifier_app_cli/_bundle/contains agents, behaviors, bundle.md, context, docsapp-cli-cli-expert, answer cited docs not in awareness pointer, cost differential ($0.18 vs $0.01) corroborates doc loadKnown Costs
docs/(~80KB) is duplicated into the wheel; site-packages copy is not source of truthbundle.mdmakes this repo loadable as a bundle — foundation's nested-bundle detection will find it when walking up from paths inside the repoforce-includedeclared undertargets.wheelonlyReferences
Extends /provider feature work from #267, #268, #269
Generated with Amplifier