Persona framework — pre-rollout review findings

[don-petry]

Review of the persona framework (#752, #754, .github-private#1279) before onboarding more personas. Findings are ordered by what gets worse as N grows.

The systemic one: errors read as negative answers

Not a bug — a habit, and the reason to fix this before N personas. Four instances in ~200 lines of routing:

Where	Error	Silently becomes
pm_manifest_query	YAML won't parse	"manifest says no"
manifest curl	5xx / network blip	"not a persona"
labels fetch (|| echo "")	API error	"no labels" → opt-out bypassed
alias handle	404	"not a persona"

A raw.githubusercontent hiccup silently disables every persona fleet-wide, with no error anywhere. A transient labels API failure lets an agent act on an item explicitly marked hands-off.

Owner: #754 (unmerged code) — being fixed there, not by dev-lead.

Blocking rollout

1. address.aliases[] is architecturally broken — DECISION NEEDED

#752 promises aliases as the rename migration path. #754's index-free routing cannot deliver it: alias petry-projects/qa-architect → personas/qa-architect/persona.yml → 404 → mention silently dropped. Contract and implementation contradict.

We just did the exact rename aliases exist for (murat → qa-lead); it only worked because nothing referenced it yet.

• Option A (recommended): drop aliases from the schema. A broken feature is worse than an absent one, and removing a field is cheap before N personas adopt it.
• Option B: derived personas/aliases.json, generated by validate-personas.py, living beside the manifests in .github-private (does not invert the canary-rings.json → release/registry.yml derivation direction), fetched only on 404.

2. gate_label declared but never enforced — SECURITY

Schema requires write surfaces to declare a gate label. Nothing checks it is applied. The first write-mode persona writes ungated. dev-lead is a write agent — this blocks its onboarding. Found independently by CodeRabbit. Owner: #754.

Promised but does not exist

These matter most because the standard reads as though they are real.

3. The team-property CI check — asserted in 4 places, exists nowhere

persona.schema.json, §4.1, TEMPLATE/persona.yml, and personas/qa-lead/persona.yml all claim CI verifies the addressing team. Nothing does. A persona can ship with a secret team (unmentionable → silently dead) or notifications enabled (pages humans on every mention). → .github-private story.

4. <id>:hands-off labels are inert

Schema requires opt_out_label; nothing provisions it. qa-lead:hands-off exists in zero repos; the canonical set is 7 hardcoded labels in a bash array in scripts/apply-repo-settings.sh. The mandated escape hatch does not work, and it is O(N personas × M repos). → .github story.

5. The eval gate is unenforced

Principle 5: "No persona reaches stable without an eval gate." canary-rollout never reads evals/. It is a checklist item, not a gate. (Predates this work — #670.)

6. evals.path is a single path, but the eval tree is skill-keyed

evals/ holds deep-review, triage, spec-drift — those are pr-review's evals. A persona with several skills cannot express them. Blocks pr-review onboarding. Schema gap.

7. The trigger matrix is mostly aspirational

qa-lead declares six surfaces; exactly one (mention) has an implementation. bridge is decorative — the router ignores it. N personas ⇒ N manifests claiming capabilities that do not exist.

8. Opt-out cannot work on discussions

Discussions have no labels API, so the router skips the check entirely. <id>:hands-off can never opt out a discussion.

Precision and UX

9. The router fires where GitHub renders no mention

Verified: fenced code blocks, inline code, and blockquotes all trigger. GitHub's one-click Quote reply prefixes >, so quote-replying to a comment that mentioned a persona re-fires it. Neither recursion axis catches human-quoting-human. Owner: #754.

10. review_requested has no routing path

Requesting a review from the team is the most natural way to address a persona on a PR, and §4 says "Reviewer-assignment counts here too". Not implemented.

11. No acknowledgement

Mention a persona → silence, indistinguishable from a silent skip. Fix that avoids #860: a 👀 reaction, not a comment. Reactions do not trigger workflows, so there is no loop risk.

12. No quota cap on fan-out

One comment can address N personas ⇒ N agent runs. The PR-cap gate covers PR creation, not mentions.

Verified NOT problems — do not re-litigate

• author_association works. don-petry returns MEMBER despite all three org members being private (public_members is empty). The trust floor is sound.
• .github-private is PUBLIC (private: false); unauthenticated raw fetch returns 200. CodeRabbit's 🔴 Critical on Add the persona @-mention router #754 ("private repo → 404 → every persona skipped") is a false positive driven by the repo name. The index-free fetch is sound.

Ownership

• Add the persona @-mention router #754 (open, mine): systemic fail-open trio, gate_label, contents: read, PyYAML, mention precision.
• dev-lead stories: team-property CI check (.github-private), hands-off label provisioning (.github).
• Decision needed: aliases (A or B), eval-gate enforcement, evals.path multi-skill shape.

Blocks the persona onboarding of dev-lead, pr-review, and business-analyst.

[don-petry]

Work breakdown

Fixing in #754 (unmerged code — dev-lead branches from main, so it cannot touch these):

• the systemic fail-open trio, gate_label enforcement, contents: read, PyYAML, mention precision

dev-lead stories (scoped to one repo each, real files, PR-sized):

• Verify persona addressing teams in CI .github-private#1280 — verify persona addressing teams in CI (the check asserted in 4 places that does not exist)
• Provision <id>:hands-off labels so persona opt-out actually works #756 — provision <id>:hands-off labels so opt-out actually works

Persona onboarding (blocked on the above):

• Onboard dev-lead as a compliant persona .github-private#1281 — dev-lead (blocked: gate_label unenforced + no evals; highest blast radius, least governed)
• Onboard pr-review as a compliant persona .github-private#1282 — pr-review (blocked: evals.path single-path gap + pr-review has channel tags + fleet consumers + a soaking candidate but is NOT in canary-rings.json #710 registration; two-mention-path decision)
• Onboard business-analyst as a compliant persona .github-private#1283 — business-analyst (blocked: no runtime exists; converges with incubation's unbuilt Ask 3)

Decisions needed before onboarding (not delegated — these are design calls):

1. aliases — drop (recommended) vs derived personas/aliases.json
2. eval-gate enforcement — make principle 5 real, or stop claiming it
3. evals.path shape — single path vs multi-skill list (pr-review proves this is not theoretical)

[don-petry]

Filed petry-projects/.github-private#1290 — PR comments from maintainers are mechanically inert.

It's the same shape as this tracker's headline finding, one layer up: an action that looks like it worked but silently did nothing. Established mechanically, not inferred:

• .github's ruleset does enforce required_review_thread_resolution: true — so an inline review comment creates a thread that blocks merge, and prompts/dev-lead/fix-reviews.md walks reviewThreads and actions it.
• An issue comment (what gh pr comment and the UI's main box produce) creates no thread. So it doesn't block, and fix-reviews.md never reads it — verified: zero references to issues/<n>/comments in that prompt.

The same words, posted two ways, have opposite force — and the ergonomic default is the silent one.

Evidence: my fail-open finding on #757 at 03:55:10Z → merged 04:03:48Z, defect intact, now being fixed in #765.