[Schema Consistency] Schema Consistency Check - 2026-06-10

[github-actions[bot]]

Summary

• Inconsistencies Found: 3 (all standing / previously-reported, re-confirmed still open)
• Categories Analyzed: Schema ↔ Parser, Schema ↔ Docs, Parser ↔ Docs, Schema ↔ Workflows
• Strategy Mode: Proven (day-of-year 161 → day_mod=1)
• New Critical Issues: None — fresh leads checked this run were all clean

This run re-verified the three open findings from prior runs against current main (HEAD 073404c) and ran two fresh proven-strategy passes (documentation coverage for newer top-level fields, and schema-default vs compiler-default consistency for max-tool-denials). All fresh leads were consistent. The three standing findings remain valid; one source file moved location but its behavior is unchanged.

Critical Issues

None. No type mismatches, missing required-field enforcement, or schema/parser field gaps that would let invalid input through were found this run.

Findings (all standing, re-confirmed)

1. Decorative enum: experiments.*.analysis_type is never branched on (Strategy 20)

The schema defines an enum analysis_type: [t_test, mann_whitney, proportion_test, bayesian_ab], but the analyzer only stores and prints the value — it never branches its computation on which test was selected.

• Schema: pkg/parser/schemas/main_workflow_schema.json (experiments.*.analysis_type enum)
• Stored: pkg/cli/experiments_analyze_statistics.go:120 (note: file moved from pkg/workflow/js/ to pkg/cli/ since last run)
• Printed only: pkg/cli/experiments_analyze_statistics.go:298 (Test type : %s to stderr)
• The only switch (line 345) branches on Recommendation, not AnalysisType

Impact: All four enum values produce identical statistics. Users selecting mann_whitney vs t_test get the same computation — the choice is silently inert. Either implement per-test logic or document the field as advisory/label-only.

2. Permission matching uses exact equality with no privilege hierarchy (Strategy 23, secondary)

The role/permission check compares the required role against the actor's effective role with exact string equality and no hierarchy expansion.

• Code: actions/setup/js/check_permissions_utils.cjs:254 → return normalizedRequired === effectiveRole;
• The primary half of this finding (the API's role_name not being read) is now RESOLVED (lines 242–246 prefer role_name, normalizing maintainer→maintain).

Impact: A workflow configured with a custom roles: [write] will reject an admin or maintainer actor, because matching is exact rather than "this level or higher." Docs (compilation-process.md, triggers.md — "based on repository permission level") imply threshold/level semantics. The default [admin, maintainer, write] masks the gotcha for most users, so it surfaces only with custom single-value role lists. Undocumented gotcha — either expand a privilege hierarchy or document that role matching is exact-set membership.

3. user-invokable frontmatter field is undocumented (Strategy 17)

• IgnoredFrontmatterFields = []string{"user-invokable"} at pkg/constants/constants.go:318
• Appears in 0 files under docs/src/content/docs/

Impact: Low. The field is intentionally silently ignored (a Copilot custom-agent field), but it has zero user-facing documentation explaining that it is accepted-but-ignored.

Fresh leads checked this run (all clean)

Verification details

• Doc coverage (Strategy 7) for newer top-level fields: secret-masking (4 files), observability (21), max-tool-denials (curated at engines.md:272,285), resources (19), user-rate-limit (6), runs-on-slim (7), mcp-scripts (17). No new zero-coverage top-level field. disable-model-invocation / run-install-scripts are no longer current top-level schema properties.
• Default consistency (Strategy 21) for max-tool-denials: schema default: 5 == constants.DefaultMaxToolDenials (pkg/constants/engine_constants.go:386) == fallback in pkg/workflow/copilot_engine_execution.go:620. Consistent.
• Schema↔parser field gaps: field_gaps.in_parser_not_schema is empty; remaining gaps are top-level fields accessed via map[string]any (no yaml tags) — known false positives.
• redirect field: schema string, handled across compiler, documented in frontmatter.md. Consistent.

Recommendations

1. analysis_type: implement per-test-type branching in the experiments analyzer, OR mark the enum as label-only in the schema description so users don't expect different statistics per value.
2. Permission hierarchy: document that custom roles: lists are matched as exact-set membership (admin does not implicitly satisfy write), OR add hierarchy expansion in check_permissions_utils.cjs.
3. user-invokable: add a one-line note to the frontmatter reference that the field is accepted-but-ignored (Copilot custom-agent compatibility).

Strategy Performance

• Strategies used: 17, 20, 23 (re-verify) + 7, 21 (fresh) — all proven
• Findings: 3 standing re-confirmed, 0 new
• Effectiveness: HIGH for re-verification; fresh passes confirmed prior fixes hold
• Should Reuse: YES — S20/S23 remain the highest-signal proven strategies

Next Steps

• Decide on analysis_type: implement branching or relabel as advisory
• Document or fix permission exact-match semantics for custom roles: lists
• Add a doc note for the ignored user-invokable field

References: §27258453758

Generated by ✅ Schema Consistency Checker · 113.7 AIC · ⌖ 32 AIC · ⊞ 9.5K · ◷

• expires on Jun 10, 2026, 10:52 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Schema Consistency Checker.

A newer discussion is available at Discussion #38531.