Agentic Persona Framework

[don-petry]

The need

We're accumulating agentic personas — Dev-Lead, PR-Reviewer, Bob (Scrum Master),
Mary (Analyst), and now Murat (Test Architect) — but each has been defined,
triggered, trusted, and rolled out bespoke. Concrete problems:

• No shared definition of what a persona is. Some are Copilot profiles,
  some are vendored BMAD framework agents, some are workflow prompt libraries.
  There's no single index-of-record for a persona's identity, where it's
  implemented, how it fires, and how it ships.
• Inconsistent trust/permission posture. "Which surfaces can this agent act
  on, and can it write or only advise?" is answered per-persona, by hand.
  A new persona's blast radius isn't bounded by default.
• No promotion gate. Personas reach broad use without a held-out eval
  proving they help — even though we already run the evals/ held-out
  discipline for skills.
• Two divergent canary registries. canary-rings.json (human-driven
  rollout) and .github-private release/registry.yml (the autonomous
  Release-Manager loop) drifted apart — divergent run_workflow names and gate
  encodings — so onboarding meant editing two places that disagree.

The concept — a Persona Definition of Done

One manifest, one trigger checklist, one file layout, one eval gate, one
registration step.

• Manifest as index-of-record (persona.yml, schema-validated). It points
  at the layers that implement a persona; it never duplicates them.
• Layer over the frameworks, don't replace them. Framework-backed personas
  pin the vendored framework version and flag any local override as an explicit
  upstream-contribution candidate (we build on BMAD / BMAD Test Architecture and
  intend to contribute back).
• Advisory by default; write is opt-in. A new persona is advisory on every
  surface; write access is opted into per surface, each with a gate label and a
  trust floor. Blast radius starts at "leaves a comment."
• Held-out eval gate before stable. Reuses the existing evals/ validator
  • holdout-guard.yml.
• Register once. canary-rings.json becomes the single source of truth;
  release/registry.yml is derived from it, and stable's * member resolves
  at runtime (all org repos are public — no fleet topology to hide).

Decisions taken (baked into the first cut)

1. Registry consolidation — single source of truth in canary-rings.json,
   eliminate the parallel hand-maintained registry.yml, resolve * at runtime.
2. Manifest is a layer over the frameworks — first-class "backed by vendored
   framework X @ version Y" fields plus a local-overrides section flagged as
   upstream-contribution candidates.

Scope / non-goals

• In scope: the standard, the manifest schema, the copy-me template, one worked
  example (Murat).
• Not in this cut: wiring Murat to a dedicated advisory workflow (stays draft,
  consumed by-path today); generating registry.yml from canary-rings.json;
  expanding Murat's held-out eval set to the min_cases gate.

Work items

• Add agentic persona standard, manifest schema, and template scaffold .github#670 — the standard, manifest JSON schema, and
  copy-me TEMPLATE/ scaffold.
• Add Murat persona instance — worked example for the persona standard #1174 — Murat as the worked
  wrap-a-vendored-agent example.

Follow-ups

• Promote Murat past draft (dedicated advisory workflow + runtime: block +
  canary-rings.json entry + ≥ min_cases held-out cases).
• Derive release/registry.yml from canary-rings.json (kill the last drift).
• Onboard the next persona through the template to pressure-test the DoD.

[github-actions[bot]]

Enhancement: Sharper Framing, Prior Art, and Validation Path

Problem reframe

The underlying problem is not just "bespoke persona definitions" — it is uncontrolled blast-radius expansion without commensurate quality evidence. Each new persona today can silently gain write access on high-consequence surfaces (PR approvals, issue triage, merge-queue comments) without proving it helps more than it harms. The "Persona Definition of Done" should be understood as a promotion protocol — the manifest is its index, but the eval gate is its teeth.

---

Prior art & competitive signal

System	Year	Key pattern	Relevance
Oracle Open Agent Spec	2025–26	Framework-agnostic YAML (agent.yaml) declaring tools, tasks, dependencies, and conformance tests. Portable across Python/Node SDKs.	Validates the "manifest as index-of-record" approach. Your persona.yml is narrower (org-internal, CI-native) but structurally aligned — interop possible if you later want external personas.
Microsoft Agent Governance Toolkit	Apr 2026	Privilege-ring model (Ring 0–3), per-ring resource caps, trust-score-based promotion, covers all 10 OWASP Agentic risks. Sub-ms policy enforcement via Cedar/OPA/YAML rules.	The "advisory-by-default, write is opt-in" posture maps directly to Rings 2–3 (standard/sandbox). Promotion gates match the held-out eval requirement. The ring model gives vocabulary for trust floors.
Four-Stage Agent Rollout Gate (FutureAGI, 2026)	Jul 2026	Shadow → Canary → Percentage → Full, each with distinct eval questions and statistical gates (Welch's t-test, rubric convergence, auto-rollback triggers).	Directly applicable to persona promotion: draft = shadow (consumed only by-path), canary = canary-rings entry, stable = full. The eval gate before stable is the Shadow→Canary transition.
Shadow Mode CI for AI Agents (DevOps.com, Jul 2026)	Jul 2026	Agents run in isolated Docker alongside live traffic, scored on structural validity + policy adherence + semantic quality. Statistical thresholds gate promotion, not binary pass/fail.	Supports the eval design: held-out cases as a "shadow traffic" proxy where the persona's output is scored without affecting production.
GitHub Copilot agent safety guardrail	2026	Built-in prohibition on agents modifying their own .github/agents configuration.	Validates your "agents cannot self-modify" invariant. Extend this to: a persona's workflow cannot edit its own manifest or eval set (already covered by holdout-guard.yml).
BMAD Method v6.x	2026	12+ persona definitions as "Agent-as-Code" Markdown with handoff protocols, vendored as git subtrees, Party Mode for multi-persona sessions. ~49K stars.	You already vendor BMAD — the persona framework formalizes the relationship: definition.layers[].type: vendored-framework-agent with pinned version + override flagging.

---

Options table

Option	Trade-off	When to choose
A. Manifest-only (current plan)	Lowest cost, fastest to land; eval gate exists but is soft (placeholder cases). Risk: personas get promoted with thin eval coverage because the gate is advisory.	Right if you'll immediately follow with eval-set expansion and hard min_cases enforcement.
B. Manifest + hard eval gate at schema level	Add min_holdout_cases as a required field in persona.schema.json (default 5, recommended 10+). The holdout-guard and validate-cases CI jobs enforce this — a persona cannot move past draft if its holdout set is below threshold.	Right if you want mechanical prevention of premature promotion — the schema becomes the gate, not just documentation.
C. Manifest + shadow-mode stage	Add status: shadow between draft and canary. Shadow means the persona's workflow runs on every trigger but its output is logged, not posted (à la DRY_RUN=1). Scored offline against held-out cases using production inputs.	Right if you want statistical confidence before even canary-level blast radius. Higher infra cost (dual-run).
D. Manifest + Microsoft-style privilege rings	Map trust floors to numbered rings (0=owner, 1=maintainer, 2=member, 3=collaborator). Per-ring resource caps (max comment length, max issues created/day, max tool invocations).	Right if you're onboarding untrusted/third-party personas. Overkill for a 5-persona internal fleet today.

Recommendation: Start with B (cheapest mechanical safety), add C as a follow-up when the fleet grows past ~8 personas and the dual-run cost is justified by the risk.

---

Risks & unknowns

1. Schema drift with upstream BMAD. BMAD v6.x ships near-daily. Your manifest pins a version, but the customize.toml / skill files under the vendored agent may change semantics between the pin point and the next bump. Mitigation: Add a CI check (tests/test_vendored_persona_compat.bats) that validates the pinned version's interface contract (expected skill files exist, agent markdown parses cleanly).

2. Eval set starvation. The Murat seed set is 5 cases. Five held-out cases do not give statistical power to detect regressions. With 5 cases, a single flipped eval produces a 20% score swing — indistinguishable from noise. Mitigation: Enforce min_holdout_cases >= 10 for canary, >= 20 for stable, as schema-level minimums by status.

3. Registry derivation gap. The plan says release/registry.yml will be "derived from canary-rings.json" — but there's no script today that does this derivation. Until it ships, the two registries will continue to drift. Mitigation: Ship the derivation script (scripts/sync-registry-from-rings.sh) as a pre-commit hook or lint step before or alongside the persona standard landing.

4. Surface-permission combinatorial explosion. The trigger matrix (7 surfaces × {advisory, write} × trust floors × gate labels) becomes unwieldy past 10 personas. Visualizing "who can write where" across the fleet is manual grep. Mitigation: Add a scripts/persona-permission-matrix.sh that reads all persona.yml files and emits a fleet-wide markdown table, wired into the weekly README refresh.

---

Phased validation / pilot path

Phase	Gate	Deliverable	Success metric
0 — Land the standard	PR review (human)	.github#670 + .github-private#1174 merged	Schema validates; Murat manifest passes persona.schema.json
1 — Harden eval minimums	CI (automated)	min_holdout_cases enforced per status in schema; expand Murat to ≥10 holdout cases	validate-cases.py fails if a persona at canary+ has < threshold cases
2 — Derive registry	CI (automated)	scripts/sync-registry-from-rings.sh + lint check; remove hand-maintained registry.yml fields	registry.yml byte-matches derivation output; drift produces CI failure
3 — Second persona through the template	Manual (pressure-test)	Onboard Mary (Feature Ideator) or Bob (Scrum Master) through the TEMPLATE scaffold	Onboarding takes < 1 hour; no out-of-band steps required
4 — Fleet permission matrix	Scheduled (weekly)	persona-permission-matrix.sh in README refresh	Matrix renders correctly; covers all personas in personas/
5 — Shadow-mode stage (future)	Design spike	Add status: shadow + DRY_RUN=1 wiring to persona's reusable workflow	Shadow persona logs output without posting; offline scoring pipeline exists

---

Success metrics

• Onboarding velocity: Time from "decide to add persona" to "draft manifest merged" drops below 1 hour (vs. current ~1 day of bespoke setup).
• Blast-radius containment: Zero write-mode regressions on surfaces where a persona is advisory-only (tracked via agent-shield.yml violation count = 0).
• Eval coverage: Every persona at canary+ has ≥10 held-out cases; held-out pass rate ≥ 80% on the scorer's rubric.
• Registry convergence: registry.yml is 100% derived from canary-rings.json — manual edits to registry.yml produce lint failure within 1 week of derivation script shipping.
• Framework version currency: Vendored framework pins are ≤ 2 minor versions behind upstream at any point (tracked by Dependabot-equivalent for subtrees).