[Schema Consistency] Schema Consistency Check — 2026-06-09

[github-actions[bot]]

Summary

• Inconsistencies Found: 3 (1 behavioral/docs, 1 decorative enum, 1 doc gap)
• Categories Analyzed: Schema ↔ Parser/Runtime, Schema ↔ Documentation, Parser ↔ Documentation
• Strategy Used: S23 (enum runtime reachability) + S20 (enum behavioral completeness) + S17 (IgnoredFrontmatterFields doc audit) — proven-strategy mode (day_mod=0)
• New Strategy: No — re-verification + deeper dig on standing findings
• Notable resolution this run: S23 primary finding (runtime ignored role_name) is now fixed.

This run prioritized re-verifying the high-signal standing findings from cache rather than re-enumerating fields. One previously-reported runtime bug has been resolved; one related behavioral/documentation gotcha remains; two long-standing low-priority inconsistencies persist.

Critical Issues

None that cause silent security bypass. The most impactful item is a behavioral semantics gap in on.roles matching that can surprise users (see below). It fails closed (over-restrictive), so it is a usability/footgun issue rather than a security hole.

✅ Resolved Since Last Run

on.roles runtime now reads role_name — Previously check_permissions_utils.cjs read only repoPermission.data.permission (which the GitHub API returns as admin/write/read/none), making the granular enum values maintain/triage/maintainer effectively dead. The current code at actions/setup/js/check_permissions_utils.cjs:242-251 now reads role_name and prefers it:

const rawRoleName = repoPermission.data.role_name;
const normalizedRoleName = roleName === "maintainer" ? "maintain" : roleName;
const normalizedPermission = permission === "maintainer" ? "maintain" : permission;
const effectiveRole = normalizedRoleName || normalizedPermission;

The maintainer → maintain alias is also handled on both the required and effective side. Granular roles now function. 🎉

Schema ↔ Documentation Mismatches

1. on.roles matching is exact-equality, not a privilege threshold (undocumented gotcha)

• Schema (main_workflow_schema.json:1840-1854): roles enum [admin, maintainer, maintain, write, triage, read, all], default [admin, maintainer, write].
• Runtime (check_permissions_utils.cjs:252-256): matching is strict equality — normalizedRequired === effectiveRole inside .some(...). There is no privilege hierarchy and no expansion in the compiler (confirmed — only skip-roles/roles extraction exists, no level expansion).
• Docs imply a threshold/level reading:
  • triggers.md:385 — "based on repository permission level"
  • compilation-process.md:217 — "actor has admin/maintainer/write permission"
• Impact: A user who narrows to roles: [write] intending "write or above" will have their admin and maintainer actors rejected, because admin !== write. The default [admin, maintainer, write] masks this because it enumerates all three explicitly. This is a footgun: the natural mental model (higher roles satisfy a lower requirement) does not hold.
• Fix options: (a) Document explicitly that roles is an exact-match allowlist, not a minimum threshold; or (b) implement hierarchy expansion (write ⇒ also accept admin, maintain).

Evidence — exact-match logic

// actions/setup/js/check_permissions_utils.cjs:252
const hasPermission = requiredPermissions.some(requiredPerm => {
  const normalizedRequired = requiredPerm === "maintainer" ? "maintain" : requiredPerm;
  return normalizedRequired === effectiveRole;   // strict equality, no hierarchy
});

Schema ↔ Parser/Implementation Mismatches

2. experiments.*.analysis_type is a decorative enum

• Schema (main_workflow_schema.json:2875): enum [t_test, mann_whitney, proportion_test, bayesian_ab].
• Implementation (pkg/cli/experiments_analyze_statistics.go): the value is stored (line 120, a.AnalysisType = cfg.AnalysisType) and printed to stderr (line 298, Test type : %s) only. The analyzer never branches on it — the only switch (line 345) is on a.Recommendation, not AnalysisType.
• Impact: All four declared statistical tests run the same computation. Users selecting mann_whitney or bayesian_ab get t-test-equivalent behavior with no warning. The enum advertises capability the code doesn't deliver.
• Fix options: (a) implement per-test-type branching; or (b) if intentional, collapse the enum / mark the unimplemented values as not-yet-supported in the schema description.

Parser ↔ Documentation Mismatch

3. user-invokable (IgnoredFrontmatterFields) is undocumented

• Code (pkg/constants/constants.go:315): IgnoredFrontmatterFields = []string{"user-invokable"} — silently accepted/ignored during frontmatter validation.
• Docs: 0 mentions across docs/src/content/docs/**.
• Impact: Low. Intended as a Copilot custom-agent passthrough field; harmless but undocumented, so users can't discover why it's silently accepted.
• Fix: One-line note in the frontmatter reference, or a comment clarifying it's an external-agent passthrough.

Recommendations

1. Document on.roles as exact-match (highest value) — add an explicit note in triggers.md that roles is an allowlist, not a minimum threshold; roles: [write] does not include admins. Consider implementing hierarchy expansion if threshold semantics are intended.
2. Resolve analysis_type — either implement the test-type branching or document which values are currently no-ops.
3. Document user-invokable — brief mention in the frontmatter reference.

Strategy Performance

• Strategies Used: S23 (runtime enum reachability) · S20 (enum behavioral completeness) · S17 (ignored-fields doc audit)
• Findings: 3 standing (1 of the prior S23 sub-findings resolved)
• Effectiveness: HIGH — re-verification caught a real fix (role_name) and confirmed the residual gotcha
• Should Reuse: YES — S23/S20 remain the most productive lenses; S17 is low-effort/low-yield but cheap

Next Steps

• Clarify on.roles matching semantics in docs (or add hierarchy expansion)
• Implement or document-out the analysis_type enum values
• Document the user-invokable ignored field

References: §27188458090

Generated by ✅ Schema Consistency Checker · 120.7 AIC · ⌖ 32.4 AIC · ⊞ 9.7K · ◷

• expires on Jun 9, 2026, 10:42 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Schema Consistency Checker.

A newer discussion is available at Discussion #38294.