[Schema Consistency] Schema consistency audit: parser diff false positives and doc drift

[github-actions[bot]]

🔍 Schema Consistency Check - 2026-08-04

Summary

• Inconsistencies Found: 4
• Categories Analyzed: Schema, Parser/Compiler, Documentation, Workflows
• Strategy Used: Field gap + targeted grep verification
• New Strategy: No

Critical Issues

1. Pre-computed parser field diff is currently unreliable

   • /tmp/gh-aw/agent/schema-diff.json reports parser_yaml_fields: [], causing every schema field to appear in field_gaps.in_schema_not_parser.
   • But the implementation clearly handles many frontmatter fields in pkg/workflow/frontmatter_types.go, pkg/workflow/frontmatter_serialization.go, and related compiler code.
   • Impact: any automation that treats in_schema_not_parser as authoritative will over-report inconsistencies and can mis-prioritize real schema drift.

2. Top-level github-app is implemented and documented, but absent from the user-facing frontmatter guide sectioning

   • Schema defines top-level github-app as a first-class field.
   • Implementation supports it as a fallback for nested token minting paths (for example safe outputs, checkout, tools, activation), with evidence in tests and workflow fixtures such as pkg/cli/workflows/test-top-level-github-app-*.md.
   • The main guide docs/src/content/docs/reference/frontmatter.md references on.github-app and skills[].github-app, but does not provide a dedicated top-level ### GitHub App (github-app:) section near other top-level fields.
   • Impact: users can miss that github-app is available as a workflow-wide fallback rather than only as a nested field.

3. Deprecated safe-outputs.mentions.allow-team-members is still shown in docs as if current

   • Schema marks allow-team-members as deprecated in favor of allowed-collaborators.
   • Docs acknowledge deprecation in glossary text, but docs/src/content/docs/reference/safe-outputs.md:1830 still shows:
     • allow-team-members: true # Allow repo collaborators (default: true)
   • That example is especially misleading because the comment describes the semantics of allowed-collaborators, not the deprecated alias name.
   • Impact: users may keep authoring deprecated configuration and misunderstand which field is canonical.

4. max-runs deprecation is split across docs in a way that can confuse field meaning

   • Schema defines top-level max-runs as a deprecated alias for max-turns.
   • Implementation includes codemods such as pkg/cli/codemod_max_runs_to_max_turns.go to migrate it.
   • Documentation also uses max-runs-per-window under user-rate-limit, and glossary.md contains both concepts.
   • Impact: because max-runs and max-runs-per-window are both present in docs, users can easily conflate the deprecated AI invocation cap with rate limiting. This is more of a docs-information-architecture inconsistency than a code bug, but it is actionable.

Documentation Gaps

• Missing dedicated top-level github-app guide section in docs/src/content/docs/reference/frontmatter.md.
• Deprecated alias still shown in examples: safe-outputs.md uses allow-team-members in example YAML.
• Cross-linking around max-runs is weak: deprecation exists, but the distinction from user-rate-limit.max-runs-per-window is easy to miss.

Schema Improvements Needed

• No hard schema bug was confirmed in this pass.
• The biggest schema-adjacent problem is actually tooling/data quality: the generated schema-diff.json parser field extraction is incomplete or broken for parser-side YAML fields.
• Consider recording both:
  • parser-native YAML tags, and
  • workflow/frontmatter map-based field extraction paths
    so the pre-computed diff better reflects reality.

Parser Updates Required

• No parser/compiler behavior defect was proven for the checked fields.
• However, the repository should treat /tmp/gh-aw/agent/schema-diff.json as heuristic input, not source of truth, until parser field extraction is fixed.

Workflow Violations

• No repository workflow violation against the checked schema fields was confirmed.
• Existing workflow fixtures actively use and test several newer fields (github-app, timeout-minutes, etc.), which supports the implementation side.

Recommendations

1. Fix the schema-diff generator so parser_yaml_fields is not empty and field gap reports stop generating false positives.
2. Add a dedicated top-level github-app section to docs/src/content/docs/reference/frontmatter.md, with a short example and explicit fallback semantics.
3. Replace allow-team-members with allowed-collaborators in docs/src/content/docs/reference/safe-outputs.md examples, keeping the old name only in deprecation notes.
4. Clarify max-runs vs max-runs-per-window with a short “do not confuse” note in frontmatter.md or glossary.md.
5. Consider adding a small CI assertion that compares deprecated fields used in docs examples against schema deprecated: true metadata.

Strategy Performance

• Strategy Used: Field gap + targeted grep verification
• Findings: 4
• Effectiveness: High
• Should Reuse: Yes

Next Steps

• Fix schema diff extraction for parser/compiler field discovery
• Update frontmatter guide for top-level github-app
• Update safe outputs docs to stop advertising deprecated allow-team-members
• Improve deprecation guidance for max-runs

Key Evidence

Selected evidence

• /tmp/gh-aw/agent/schema-diff.json
  • parser_yaml_fields: []
  • field_gaps.in_schema_not_parser contains effectively all top-level schema fields
• docs/src/content/docs/reference/frontmatter.md:485
  • Has a redirect: section, confirming schema/docs alignment for that field
• docs/src/content/docs/reference/frontmatter.md:265
  • Has runs-on-slim guidance, confirming the pre-computed “not in workflow compiler” gap is a false lead
• docs/src/content/docs/reference/safe-outputs.md:1830
  • Shows deprecated allow-team-members example
• pkg/cli/codemod_max_runs_to_max_turns.go
  • Confirms max-runs is deprecated and migration-supported
• pkg/cli/workflows/test-top-level-github-app-safe-outputs.md
  • Confirms repository tests/examples rely on top-level github-app fallback behavior

Next Actions

• Prioritize the schema-diff generator bug first; it affects the reliability of future audits.
• Then clean up the top-level github-app and deprecated mention-filter docs.

References:

• §30883244759

Generated by ✅ Schema Consistency Checker · gpt54 · 14.1 AIC · ⌖ 7.88 AIC · ⊞ 8.3K · ◷

• expires on Aug 4, 2026, 10:20 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-05T06:20:49.107Z.

Closed by Workflow