feat: Add schema-feature-coverage agentic workflow for 100% schema field coverage#21856
feat: Add schema-feature-coverage agentic workflow for 100% schema field coverage#21856
Conversation
|
Hey A few things to address before this is ready for review:
If you'd like a hand picking this back up, here's a ready-to-use agentic prompt:
|
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new Codex-powered scheduled agentic workflow intended to audit top-level fields in pkg/parser/schemas/main_workflow_schema.json against .github/workflows/**/*.md and open PRs adding minimal “schema demo” workflows for any uncovered fields.
Changes:
- Introduces
.github/workflows/schema-feature-coverage.mddescribing the auditing/remediation procedure and PR creation rules (max 10 PRs per run). - Adds the compiled GitHub Actions workflow
.github/workflows/schema-feature-coverage.lock.ymlgenerated from the markdown workflow.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/workflows/schema-feature-coverage.md | Defines the agent instructions for schema field extraction, coverage detection, and PR-based remediation. |
| .github/workflows/schema-feature-coverage.lock.yml | Generated compiled workflow to run the new schema coverage checker on a weekly schedule and via workflow_dispatch. |
Comments suppressed due to low confidence (2)
.github/workflows/schema-feature-coverage.md:143
- The guidance for
resourcesuses an array of objects withurl, but in the schemaresourcesis an array of strings (relative paths to workflow/action files to fetch). As written, a generated demo workflow would be schema-invalid (and likely semantically wrong forgh aw add).
| `services` | `{redis: {image: "redis:7"}}` | Docker services |
| `resources` | `[{url: "https://example.com/resource.md"}]` | Additional resource URLs |
| `mcp-servers` | `{my-server: {command: "npx", args: ["-y", "@modelcontextprotocol/server-memory"]}}` | MCP server config object |
.github/workflows/schema-feature-coverage.md:145
- The suggested demo shape for
mcp-scripts({my-script: {steps: [...]}}) doesn’t match the schema.mcp-scriptsentries require at leastdescriptionplus exactly one ofscript/run/py/go(and optionalinputs,env, etc.). If this guidance is used, generated demo workflows will not validate/compile.
| `mcp-servers` | `{my-server: {command: "npx", args: ["-y", "@modelcontextprotocol/server-memory"]}}` | MCP server config object |
| `mcp-scripts` | `{my-script: {steps: [{name: "Run", run: "echo hello"}]}}` | MCP script definitions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| ```bash | ||
| for field in $(jq -r '.properties | keys[]' pkg/parser/schemas/main_workflow_schema.json | sort); do | ||
| count=$(grep -rl "^${field}:" .github/workflows/ --include="*.md" 2>/dev/null | wc -l | tr -d ' ') |
There was a problem hiding this comment.
Step 2’s coverage check uses grep -rl "^${field}:" across the entire workflow markdown file. This can produce false positives when a schema key appears at the start of a line inside fenced code blocks / examples in the body, even if it’s not present in the YAML frontmatter (i.e., not actually “used” by the workflow). Consider extracting only the frontmatter section (between the first two --- lines) for each file and running the key check against that extracted YAML, or otherwise parsing frontmatter keys deterministically before computing coverage.
| count=$(grep -rl "^${field}:" .github/workflows/ --include="*.md" 2>/dev/null | wc -l | tr -d ' ') | |
| count=0 | |
| while IFS= read -r file; do | |
| # Extract YAML frontmatter (between the first two '---' lines) and check for the field as a top-level key | |
| if awk 'BEGIN{in_frontmatter=0} /^---[ \t]*$/{if(in_frontmatter==0){in_frontmatter=1; next} else{exit}} in_frontmatter==1{print}' "$file" | grep -q "^${field}:"; then | |
| count=$((count + 1)) | |
| fi | |
| done < <(find .github/workflows -type f -name '*.md') |
| | `private` | `true` | Boolean — marks workflow as non-shareable | | ||
| | `run-name` | `"Schema Demo #${{ github.run_number }}"` | Custom run name string | | ||
| | `secret-masking` | `{steps: [{name: "Mask", run: "echo masked"}]}` | Secret masking config | | ||
| | `secrets` | `{MY_SECRET: {required: false}}` | Secrets configuration | |
There was a problem hiding this comment.
The suggested demo value for the secrets field ({MY_SECRET: {required: false}}) does not match the schema: each entry must be either a string expression or an object with required value (and optional description). If the workflow uses this guidance, generated demo workflows will be invalid and fail compilation.
This issue also appears in the following locations of the same file:
- line 141
- line 143
| | `secrets` | `{MY_SECRET: {required: false}}` | Secrets configuration | | |
| | `secrets` | `{MY_SECRET: {value: "${{ secrets.MY_SECRET }}", description: "Demo secret value"}}` | Secrets configuration | |
Adds a Codex-powered weekly workflow that audits schema field coverage across all agentic workflows and opens PRs for any gaps.
What it does
jq -r '.properties | keys[]' pkg/parser/schemas/main_workflow_schema.json | sortproduces a canonical sorted list of all top-level schema propertiesgrep -rl "^${field}:"scans every.github/workflows/**/*.md(includingshared/) to count usage per field.github/workflows/schema-demo-<field>.mdand opens a focused PR — up to 10 per run (alphabetical order; subsequent weekly runs handle the rest)noopimmediatelyConfiguration
strict: trueworkflow_dispatchcreate-pull-request(max: 10)bash(jq/grep),edit(write demo files),github(remote, default toolsets)Demo workflow template
Each generated PR adds a file like:
Field descriptions are pulled directly from the schema via
jq '.properties["<FIELD>"].description'so PR bodies stay accurate without hardcoding.Original prompt
🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.