feat: structured output mode — constrain agent responses to a declared JSON schema

[Copilot]

Agentic workflows currently produce freeform text, forcing downstream jobs to parse unstructured output with brittle regex or prompt-engineering hacks. This adds a first-class structured-output frontmatter field that declares a JSON Schema the agent's response must conform to, with compile-time validation and runtime enforcement.

Changes

New structured-output frontmatter field

• Accepts either schema (inline JSON Schema object) or schema-file (repo-relative path to a .json file)
• Exactly one must be specified; both or neither is a compile-time error
• Schema is validated at compile time using santhosh-tekuri/jsonschema/v6 (already a project dependency)

Runtime behavior (generated YAML)

• Pre-agent step writes the resolved schema to /tmp/gh-aw/structured-output-schema.json
• GH_AW_STRUCTURED_OUTPUT_SCHEMA and GH_AW_STRUCTURED_OUTPUT_FILE env vars injected into all engine execution steps (copilot, claude, codex, crush, gemini, opencode) — parallel to the existing applySafeOutputEnvToMap pattern
• Post-agent actions/github-script step (runs always()) reads /tmp/gh-aw/structured-output.json, validates it is well-formed JSON, and sets the structured_output step output

Codex native enforcement

• When structured-output is configured, --output-schema <schema-path> is passed to the Codex exec command, constraining the model at the token-sampling level to produce schema-conformant JSON (belt-and-suspenders on top of the file-write convention)
• Not applied for detection runs

Agent job output

• structured_output added to the agent job's outputs map, exposing the compact validated JSON string to downstream jobs via ${{ needs.agent.outputs.structured_output }}
• Structured output file included in the agent artifact for traceability

Data model

• StructuredOutputConfig added to FrontmatterConfig and WorkflowData
• Extraction happens in extractAdditionalConfigurations alongside cache-memory, repo-memory, and mcp-scripts

Example

---
on:
  issues:
    types: [labeled]
engine:
  id: copilot

structured-output:
  schema:
    type: object
    properties:
      decision:
        type: string
        enum: [APPROVE, REQUEST_CHANGES, ESCALATE]
      reasoning:
        type: string
        minLength: 10
      confidence:
        type: number
        minimum: 0
        maximum: 1
    required: [decision, reasoning, confidence]
    additionalProperties: false
---

Downstream job:

jobs:
  route:
    needs: agent
    steps:
      - run: |
          DECISION=$(echo '${{ needs.agent.outputs.structured_output }}' | jq -r .decision)
          if [ "$DECISION" = "ESCALATE" ]; then
            gh issue edit ${{ github.event.issue.number }} --add-label "needs-human"
          fi

Alternatively, reference a schema file to keep large schemas out of the frontmatter:

structured-output:
  schema-file: .github/schemas/triage-output.schema.json

[lpcox]

Review: Engine-Level Structured Output Support

This PR implements structured output as a file-based convention — the agent is given env vars pointing to a schema file and an output file path, then a post-agent step validates the JSON. This is engine-agnostic but leaves significant native capabilities unused.

How Each Engine Supports Structured Outputs Natively

Copilot CLI — ❌ No native structured output support

• Copilot CLI does not expose a --response-format or JSON-mode flag
• The env-var + file-write approach is the only viable path
• The agent must be instructed via system prompt to write JSON to the file path

Claude CLI — ⚠️ Partial native support available but unused

• Claude CLI (claude) supports --output-format json which constrains the final response to valid JSON
• The Anthropic API natively supports tool_use with structured schemas and prefilled responses for JSON mode
• Gap: This PR could pass --output-format json to the Claude CLI command when structured output is configured, providing a belt-and-suspenders guarantee (native JSON mode + post-validation)

Codex CLI — ⚠️ Partial native support available but unused

• OpenAI's API supports response_format: { type: "json_schema", json_schema: {...} } which provides guaranteed schema-conformant output at the model level
• The Codex CLI does not currently expose this as a flag, but OpenAI's structured outputs are the gold standard for this feature — the model is constrained at the token sampling level
• Gap: If/when Codex CLI adds a --response-format flag, this PR's architecture should pass the schema through

Gemini — ⚠️ Native support exists

• Gemini API supports response_schema in generation config for JSON mode
• Gemini CLI may not expose this yet

Crush/OpenCode — Unknown/varies

Summary of Concerns

1. The PR relies entirely on prompt-engineering + post-hoc validation — the schema file is written to disk and env vars are set, but no engine is actually told to produce structured output. The agent must discover the env vars and voluntarily write JSON. There's no mechanism in the system prompt injection to instruct the agent about this requirement.

2. No runtime schema validation — The post-agent step only checks JSON.parse() succeeds. It does NOT validate against the declared schema. The schema is validated at compile time but never enforced at runtime. A conforming-JSON-but-wrong-schema response would pass silently.

3. Missing prompt injection — How does the agent know to write JSON to $GH_AW_STRUCTURED_OUTPUT_FILE? The env vars are set, but unless the system prompt or agent instructions reference them, most engines will produce freeform text. Is there a separate PR that adds the system prompt integration?

4. Native capabilities create stronger guarantees — For Claude and OpenAI/Codex, model-level JSON constraints prevent malformed output entirely (no retry needed). The current approach allows the agent to produce invalid output, fail validation, and waste the entire run.

Recommendation

The architecture is sound as a lowest-common-denominator approach, but the PR should:

• Add runtime schema validation (not just JSON.parse) in the post-agent step
• Document which engines need prompt-based instruction vs. which could use native structured output in future
• Clarify how the agent discovers and acts on the env vars (system prompt integration)

[pelikhan]

@copilot configure codex to use the cli argument --output-schema

https://www.mintlify.com/openai/codex/cli/exec?utm_source=chatgpt.com#param-output-schema

[pelikhan]

Aside from codex none of these are native, it will not work well. We need to take into the SDK and provide the Schema into the LLM inference calls.

[pelikhan]

Unlikely to work properly

[Copilot]

@copilot configure codex to use the cli argument --output-schema

https://www.mintlify.com/openai/codex/cli/exec?utm_source=chatgpt.com#param-output-schema

Done in 5aa6e43. When structured-output is configured and it's not a detection run, the Codex exec command now includes --output-schema /tmp/gh-aw/structured-output-schema.json — the same schema file written by the pre-agent setup step. This provides native model-level JSON constraint on top of the existing env-var + post-validation layer.

[pelikhan]

It's not well supported so far. Looks like a job for sub-agents.