Skip to content

feat(cert): pin certification modules to training-agent tenants#3930

Merged
bokelley merged 2 commits intomainfrom
bokelley/cert-module-tenant-pinning
May 3, 2026
Merged

feat(cert): pin certification modules to training-agent tenants#3930
bokelley merged 2 commits intomainfrom
bokelley/cert-module-tenant-pinning

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

@bokelley bokelley commented May 3, 2026

Summary

Each certification module now declares the training-agent tenants its lessons exercise. Sage hands learners deterministic per-tenant URLs (/signals/mcp for B3, /brand/mcp + /governance/mcp for C2, etc.) instead of the legacy single-URL alias that pre-dated the multi-tenant migration in #3713.

Closes the wrong-tenant footgun for cert work: a learner on a signals module no longer gets pointed at /sales/mcp, finds get_signals missing, and hits Unknown tool.

Why

Before this PR, three injection sites in certification-tools.ts hardcoded ${trainingAgentUrl}/mcp (the legacy alias) for every module's demos:

  • buildCertificationContext line 498 — fires at the top of every Sage turn for any in-progress module
  • get_certification_module line 1367 — module preview
  • start_certification_module line 1490-1494 — module entry point

PUBLIC_TEST_AGENT_URLS (the per-tenant map shipped in #3713) was referenced in exactly two lines — both for URL recognition during auth resolution, never for steering. Cert content had zero references to per-tenant URLs.

How

Schema (migration 464): certification_modules.tenant_ids TEXT[]. Order is significant — index 0 is primary. NULL means "no pinning — fall back to discovery extension" (today's behavior; safe default for future modules).

Backfill: all 20 seeded modules. Single-tenant modules (B3, S3, S4) declare one id; multi-tenant ones (A3 tour, C1/C2/C3/C4 buyer flows, D2, S5) declare an ordered list.

Plumbing:

  • New tenantUrlsForModule() in server/src/training-agent/config.ts — resolves ids → URLs at the prompt boundary. Decoupled from canonical hostname (https://test-agent.adcontextprotocol.org lives in one place).
  • New exported formatTenantBlock() in certification-tools.ts — collapses single-tenant modules to agent_url: "..." and emits primary + sibling list for multi-tenant modules with explicit "use primary by default; switch when needed; the discovery extension is the documented escape hatch" guidance.
  • All three injection sites updated to use the helper. buildCertificationContext unions tenant_ids across active modules so a learner with two specialisms in flight gets a single deterministic source of truth.

Mapping (full table)

Module tenant_ids
A1 Why AdCP [sales]
A2 Your first media buy [sales]
A3 The AdCP landscape [sales,signals,governance,creative,brand]
B1 Designing your product catalog [sales]
B2 Product discovery + creative specifications [sales,creative]
B3 Measurement, signals, optimization [signals,sales]
B4 Build project — your first sales agent [sales]
C1 Multi-agent buying and media planning [sales,signals,governance]
C2 Brand identity and compliance protocols [brand,governance]
C3 Creative workflows and sponsored intelligence [creative,brand,creative-builder]
C4 Build project — your first buyer agent [sales,signals,brand]
D1 MCP server architecture [sales]
D2 Supply path and agent trust [sales,governance]
D3 RTB migration patterns [sales]
D4 Build project — AdCP infrastructure [sales,signals]
S1 Media buy mastery [sales]
S2 Creative mastery [creative,creative-builder]
S3 Signals and audiences [signals]
S4 Capstone: Governance [governance]
S5 Sponsored Intelligence [brand,creative,governance]

Lays groundwork for #3712

The persona harness was going to need to assert "the LLM correctly picked the right tenant from the discovery extension." With pinning in place, the assertion is "for module M, did Sage steer the persona to a tenant in M.tenant_ids[], primary first?" — a deterministic check, not an LLM-quality measurement.

Tests

  • 9 unit tests for tenantUrlsForModule + formatTenantBlock (single, multi, empty, hyphenated ids, trailing slash)
  • 5 integration tests against a real Postgres (migration applied) — verifies field is populated, multi-tenant order is preserved, A3 tour declares all five user-facing tenants, S-track specialist deep dives map to canonical tenants
  • Verified locally end-to-end: docker compose postgres + npm run typecheck + new tests pass

Test plan

  • CI green
  • No regression in existing storyboard matrix
  • Once merged, deploy + verify start_certification_module S4 over MCP emits /governance/mcp (not /mcp)

🤖 Generated with Claude Code

Each certification module now declares the training-agent tenants its
lessons exercise. Sage emits deterministic per-tenant URLs at start /
get / context-build time instead of the legacy single-URL alias that
pre-dated the multi-tenant migration in #3713. Closes the wrong-tenant
footgun for cert work — a learner on a signals module no longer gets
pointed at /sales/mcp, finds get_signals missing, and hits Unknown tool.

Schema: `certification_modules.tenant_ids TEXT[]` (ordered — index 0 is
primary). NULL means "no pinning — fall back to discovery extension"
(safe default for future modules). Migration 464 backfills all 20
seeded modules; multi-tenant ones (C1, C2, C3, C4, D2, S5, A3) declare
the full set with primary first.

Plumbing: `tenantUrlsForModule()` in training-agent/config.ts resolves
ids → URLs at the prompt boundary; `formatTenantBlock()` in
certification-tools.ts collapses single-tenant to a one-liner and emits
primary + sibling list for multi-tenant modules. Three injection sites
updated: buildCertificationContext (unions tenant_ids across active
modules), start_certification_module, get_certification_module.

Lays groundwork for the persona harness in #3712 — assertions become
"for module M, did Sage steer the persona to a tenant in M.tenant_ids?"
rather than "did the LLM correctly infer from the discovery extension?".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address feedback from code-reviewer, adtech-product-expert,
education-expert, and prompt-engineer on #3930.

Substantive changes:

1. NULL out A3, C3, S5 in migration 464. The training agent has no
   tenant that serves si_* tools — verified empirically against the
   local stack (every tenant + the legacy /mcp returned zero si_*
   in tools/list). Pinning these to a sibling would ship a
   confidently-wrong URL into Sage's prompt. Tracked as #3940
   (add an si tenant + repin).

2. B3 reduced to [sales]. Publisher learners shouldn't be pointed
   at /signals/mcp — signals is a buy-side discovery surface they
   consume on their own sales agent, not a tenant they operate.

3. C1 drops governance. Per migration 288, governance lessons live
   in C2; pinning here was speculative and added URL noise with no
   matching curriculum content.

4. Multi-tenant prompt block rewritten. Tagged "Internal — do not
   narrate to the learner" with an explicit error-driven trigger
   ("on unknown tool → GET /.well-known/adagents.json → switch
   sibling → retry"). Without the tag Sage paraphrases the URL list
   into the conversation; without the trigger she treats the
   discovery extension as docs prose, not a procedure.

5. buildCertificationContext caches the active-modules fetch in a
   moduleCache Map and reuses it in the per-module loop (was
   double-fetching on every Sage prompt build). Module ids
   normalized once at the boundary — no more toUpperCase mismatch
   between the union loop and the per-module loop.

6. Migration 464 backfill is now `UPDATE ... WHERE tenant_ids IS NULL`
   so a stale DB with hand-edited rows survives a re-run intact.

Tests updated: unit tests assert the new "Internal — do not narrate"
framing + explicit error trigger; integration tests assert SI
modules are NULL, B3 doesn't touch signals, C1 doesn't touch
governance.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley
Copy link
Copy Markdown
Contributor Author

bokelley commented May 3, 2026

Expert review pass complete (code-reviewer, adtech-product-expert, education-expert, prompt-engineer). Pushed 1a680a9a4c addressing all blockers + real findings.

Substantive changes:

  1. NULL out A3, C3, S5. All three teach SI tools that no training-agent tenant serves (verified empirically — every tenant + legacy /mcp returned 0 si_* in tools/list). Pinning them to a sibling would ship a confidently-wrong URL into Sage's prompt. Filed Sponsored Intelligence: training-agent has no tenant serving si_* tools #3940 to add an si tenant + repin.
  2. B3 → [sales] (was [signals, sales]). Publisher learners shouldn't be pointed at the buy-side /signals/mcp — signals is a surface they consume on their own sales agent, not one they operate.
  3. C1 drops governance (was [sales, signals, governance]). Per migration 288, governance lessons live in C2; pinning here was speculative.
  4. Multi-tenant prompt block rewritten. Now tagged **Internal — do not narrate to the learner** with an explicit error-driven trigger (on "unknown tool" → GET /.well-known/adagents.json → switch sibling → retry). Prevents Sage from paraphrasing URL noise into the conversation and gives a concrete recovery procedure instead of background docs prose.
  5. buildCertificationContext memoized + module-id normalization at the boundary. Was double-fetching active modules on every Sage prompt build with a .toUpperCase() mismatch between two loops.
  6. Migration backfill guarded with WHERE tenant_ids IS NULL so stale-DB hand-edits survive a re-run.

Tests (16 passing): unit tests assert new "Internal — do not narrate" framing + explicit error trigger; integration tests assert SI modules are NULL, B3 doesn't touch signals, C1 doesn't touch governance.

Verified end-to-end in docker — local stack with the revised mapping emits /governance/mcp for S4 (single-tenant), the multi-tenant block with internal framing for C2, and falls back to legacy /mcp for the SI modules.

@bokelley bokelley merged commit fc591c1 into main May 3, 2026
21 checks passed
@bokelley bokelley deleted the bokelley/cert-module-tenant-pinning branch May 3, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant