feat: out-of-band kind=run_skill dispatcher (plan §007-008)

[ericodom]

Summary

Launch-blocker #1 from today's session. Replaces the post-§U6 placeholder rejector with a real dispatcher: scheduled jobs, admin "Run now", chat-intent-dispatched skill runs, and webhook-triggered runs (with non-null agentId) now actually execute the target skill via the same Strands agent loop the chat turn uses.

Plan: docs/plans/2026-04-24-008-feat-skill-run-dispatcher-plan.md

What ships

Five focused units, dependency-ordered, each a separate commit:

U1 — Shared runtime-config helper + new service-auth REST endpoint

• New packages/api/src/lib/resolve-agent-runtime-config.ts factors the ~300-line agent/template/skills/MCP/KB/guardrail resolution out of chat-agent-invoke.ts. Helper is the single source of truth; both chat-invoke and the new endpoint call it. 9 unit tests.
• New packages/api/src/handlers/agents-runtime-config.ts exposes GET /api/agents/runtime-config?tenantId=X&agentId=Y (Bearer API_AUTH_SECRET). 13 handler tests.
• chat-agent-invoke.ts refactored to call the helper; zero behavior change. Per-turn fields (thread context, sandbox preflight, user resolution) stay inline.
• Terraform route + handler registration in lambda-api/handlers.tf.

U2 — Extract _execute_agent_turn helper in server.py

• ~170-line chat-loop prologue factored into a reusable helper. Same env setup, skill install, workspace ready, eval span, messages build, system prompt, _call_strands_agent, tool-cost drain. Pure lift-and-shift refactor.
• Chat do_POST now calls the helper; dispatcher will too.
• All 234 existing container tests pass unchanged.

U3 — Python api_runtime_config client + real dispatcher

• New container-sources/api_runtime_config.py: urllib GET with 3-attempt retry (5xx + transient retry, 4xx terminal, 404 → AgentConfigNotFoundError). 9 unit tests.
• run_skill_dispatch.py rewritten: null-agentId fails fast; otherwise fetch config → build synthetic chat turn ("Run the sales-prep skill with these inputs: ...") → call _execute_agent_turn → POST /api/skills/complete with {type: inline, payload} on complete, reason on failure. 16 new tests cover happy path + all failure modes.
• Dockerfile uses wildcard COPY container-sources/ /app/ so new module picked up automatically; _boot_assert.py::EXPECTED_CONTAINER_SOURCES updated per the institutional learning.

U4 — Flip TS emitters to InvocationType: "Event"

• packages/api/src/graphql/utils.ts::invokeSkillRun, packages/api/src/handlers/skills.ts::invokeAgentcoreRunSkill, packages/lambda/job-trigger.ts all flip RequestResponse → Event. Webhook path inherits via invokeSkillRun.
• Why: agent loop routinely takes 10-60s; RequestResponse with 28s socket timeout falsely timed out. Event returns 202 on enqueue; the HMAC-signed /api/skills/complete callback is the authoritative execution-result signal.
• Does NOT cross feedback_avoid_fire_and_forget_lambda_invokes — that rule governs paths without callbacks; we have a durable HMAC-signed one.
• Chat-agent-invoke stays RequestResponse (needs the synchronous response for the thread message stream).

U5 — Smoke scripts + CHECKS.md refresh

• scripts/smoke/{chat,catalog}-smoke.sh + CHECKS.md + README.md flip passing language off "every envelope terminates with U6 unsupported-runtime" onto "row reaches complete or failed with a specific reason; only stuck-running / transport errors FAIL."

ce-code-review fixes (autofix + P0s, post-U5)

Four additional commits address findings from ce-code-review's multi-agent pass against U1–U5:

• Autofix (ac59a32) — removed stale comment in chat-agent-invoke.ts describing nonexistent currentUserEmail propagation; updated scheduled.test.ts + harness README to reflect post-§U4 Event + HMAC callback semantics.
• P0-A (c61ff58) — agentId threading. The SkillRunInvokePayload type was missing agentId; all four TS emitters stripped the field; the Python dispatcher rejected every non-webhook envelope with _MISSING_AGENT_REASON. Added agentId: string | null to the type, threaded through startSkillRun.mutation.ts, skills.ts::startSkillRunService, webhooks/_shared.ts, and job-trigger.ts. 3 regression tests (including an InvocationType=Event pin).
• P0-B (7351cbe) — cross-tenant user-email leak. resolveAgentRuntimeConfig's users lookup scoped only on users.id; any holder of API_AUTH_SECRET could enumerate cross-tenant emails via the currentUserId query param. All three users lookups (currentUserId, human_pair email, human_pair name) now AND on users.tenant_id. Added a regression test that captures the actual where() predicates.
• P0-C (5f03f00) — async-invoke retry non-idempotency. AWS Lambda default MaximumRetryAttempts=2 on the §U4 Event invoke would re-run the agent turn (re-burning Bedrock tokens) on any transient failure. Set MaximumRetryAttempts=0 on agentcore-invoke via aws_lambda_function_event_invoke_config; added a thinkwork-<stage>-agentcore-async-dlq SQS queue (14-day retention, SSE-managed) for failure visibility.

Key technical decisions

1. Reuse _call_strands_agent via _execute_agent_turn — one code path for the agent loop; the chat-invoke and run_skill paths cannot drift.
2. Runtime config fetched inside the container via GET /api/agents/runtime-config — keeps the four TS emitters lean (no envelope bloat); single resolver for both chat + dispatcher; tenant-scoped 404 returns don't leak cross-tenant existence.
3. Fail fast on null agentId — scheduled/catalog/chat-intent always carry one; only webhook resolvers emit null, and tenant-admin fallback routing is a product decision that belongs in its own PR.
4. Synthetic user message, not SKILL.md injection — the model reads the skill's SKILL.md via the AgentSkills plugin's progressive disclosure; we just tell it to run it with the given args. No harness magic.
5. One-shot dispatch, not retried — agent turn is non-idempotent (Bedrock tokens); /api/skills/complete atomic CAS + 15-min reconciler provide the safety net; DLQ preserves observability of dropped invokes.

What's explicitly NOT in this PR

• Refactor of _call_strands_agent (1,400-line function; too risky for this change).
• Streaming the skill-run output.
• Cancellation cooperation inside the agent loop (cancelSkillRun mutation flips the row; mid-run polling is separate work).
• Webhook null-agentId fallback routing — fails with named reason for now.

Test plan

• pnpm -r typecheck (minus @thinkwork/agent-tools pre-existing no-tsc) — green
• pnpm -r test — 1378 passed, 7 skipped across api (1385 total); 68 passed lambda
• uv run pytest packages/agentcore-strands/agent-container/ — 249 passed
• pnpm exec prettier --check on touched packages — clean
• terraform validate on agentcore-runtime module — clean
• Mandatory grep for old rejector strings returns zero hits
• Post-merge: run catalog-smoke on dev for sales-prep, confirm row reaches complete or fails at a specific connector
• Post-merge: schedule a run via the admin "Run now" button, confirm same
• Post-merge: log-grep dispatch_run_skill in CloudWatch to confirm the new code is serving (per docs/solutions/workflow-issues/deploy-silent-arch-mismatch-took-a-week-to-surface-2026-04-24.md)

Deploy ordering notes

• Migration 0029 already applied on dev (from feat(U6): delete composition runner + skill_catalog.execution/mode #542). This PR adds no new migrations.
• The AgentCore runtime update (deploy.yml's container rebuild + UpdateAgentRuntime) is required for the new Python code to serve — per docs/solutions/workflow-issues/agentcore-runtime-no-auto-repull-requires-explicit-update-2026-04-24.md the ECR push alone is not enough. The merge pipeline handles this.
• Warm containers may boot pre-env-injection during terraform-apply per project_agentcore_deploy_race_env; the 15-min reconciler is the backstop. The new api_runtime_config.fetch reads env at call time, not import, so post-race recovery is automatic.
• The new aws_sqs_queue.agentcore_async_dlq + aws_lambda_function_event_invoke_config.agentcore resources are additive; first apply creates them and flips retry semantics. No data migration needed.

Residual Review Findings

ce-code-review (autofix mode, 10 reviewers, 8 returned findings) surfaced the P0s above plus the P1/P2/P3 work listed here. Durable sink: this PR body (tracker filing was not invoked for in-scope launch work). Full artifact on disk at .context/compound-engineering/ce-code-review/20260424-151323-e0154520/findings.md in the worktree.

Actionable (P1) — should ship in a follow-up before/alongside launch

• P1 | reliability-02 — aws_lambda_function_event_invoke_config now exists, but consider EventInvoke destinations for OnSuccess too if operator-side observability wants completion streaming. Not blocking.
• P1 | reliability-05 — post_skill_run_complete in run_skill_dispatch.py returns silently when API_AUTH_SECRET / THINKWORK_API_URL are missing. Row stays running until the 15-min reconciler. Should log ERROR + raise so the AWS retry (now 0, but future-proof) or the operator sees it immediately.
• P1 | testing-1 — _execute_agent_turn (~170 LOC, heavy hidden state) has zero direct unit tests. Add at minimum: happy path, empty response, tool-call path, exception propagation.
• P1 | testing-2 — PostCompletionTests.test_missing_env_logs_and_returns asserts nothing; false confidence. Assert return value + log emission.
• P1 | testing-4 — resolve-agent-runtime-config branches (guardrail-assigned path, human-pair email fallback path, humanN lookup) have no direct unit tests; current coverage is indirect via chat + dispatcher integration.
• P1 | kieran-python async-def-without-awaits — dispatch_run_skill is async def with zero awaits. asyncio.run(...) makes it work, but either drop async or convert the blocking urllib calls to awaitable HTTP so the label reflects reality.
• P1 | kieran-python late-imports — from server import _execute_agent_turn inside the function hides a module-load cycle. Either declare explicitly or extract _execute_agent_turn into its own module.
• P1 | kt-002 — three inlined copies of the invokeAgentcoreRunSkill envelope constructor (utils.ts, skills.ts, job-trigger.ts). Extract to a shared helper so future envelope-shape changes (like today's agentId thread) happen once, not thrice.

Polish (P2) — defer to ops hardening pass

• _tool_costs global accumulates across invocations if _execute_agent_turn raises (reliability-03, reliability-07) — wrap in try/finally _tool_costs.clear().
• No soft deadline inside dispatch_run_skill; runaway agent loop burns full 900s budget (reliability-04).
• Missing completion_hmac_secret in envelope strands the row on 401 terminal (reliability-06).
• Guardrail-blocked response is posted as successful skill completion instead of failed with guardrail_blocked reason (correctness-2, adv-05).
• Tenant built-in-tool kill-switch not propagated to skill-run dispatch (correctness-3).
• Dead fallback reads: scope.get('invokerUserId' | 'agentId' | 'threadId') never match TS producer shape — remove to prevent silent future breakage (correctness-4, api-contract-002).
• _execute_agent_turn leaves workspace/composer/hindsight env vars set on warm containers (correctness-5).
• _FETCH_RETRY_DELAYS and _COMPLETE_RETRY_DELAYS identical tuples in two modules (kieran-python retry-delays-duplicated).
• _cfg(camel, snake) closure repeated 3 places (kieran-python envelope-scope-fallback-duplicated-three-times).
• (k: any) cast in chat-agent-invoke.ts defeats KnowledgeBaseConfig type (kt-003).
• JSONB unchecked casts in resolveAgentRuntimeConfig rely on schema stability (kt-004).
• run_skill_dispatch reaches past the underscore prefix into server._execute_agent_turn (M2).
• RuntimeConfigFetchError(code=None) conflates missing-env with retry-exhaustion (M4).
• Empty-skills envelope silently posts "no skills available" deliverable as complete (adv-04).
• Unvalidated response_text posted verbatim as deliveredArtifactRef with no size/shape check (adv-06).
• Warm-container env leak via exception in do_POST snake_case/camelCase ordering (adv-07).
• Admin cancel race — dispatcher blind to skill_runs.status flip mid-run; 4xx on callback is only signal (adv-08).
• Several coverage gaps: _build_synthetic_payload asserts only 4 of 16+ keys (testing-5); scope casing mismatch untested (testing-6); agents-runtime-config handler test mocks the helper wholesale (testing-7); _execute_agent_turn edge cases — response_text=None, empty skills, None mcp_configs — untested (testing-8).

Advisory (P3)

• (k: any) parameter naming drift (kt-005).
• Inlined UUID_RE duplication — 6th+ copy (kt-006).
• agents-runtime-config handler hand-rolls OPTIONS instead of using handleCors/cors() (kt-007).
• _urlopen_with_retry's assert last_exc is not None (reliability-08).
• Runtime-config fetch budget sums to ~14s but each call allows timeout=15s × 4 attempts = 60s worst case (reliability-09).
• No circuit breaker on runtime-config fetch — API outage cascades across N dispatches (adv-09).
• HMAC header parse edge case — bare-hex accepted alongside sha256=... (adv-10).

🤖 Generated with Claude Code