[Safe Outputs Conformance] IMP-004: Checker false-positives on $ref-based ado-* safe-output schemas

[github-actions]

Conformance Check Failure

Check ID: IMP-004
Severity: HIGH
Category: Implementation

Problem Description

The IMP-004 check ("Safe Output Config Schema Coverage") in scripts/check-safe-outputs-conformance.sh reported 32 HIGH-severity findings claiming that Azure DevOps safe-output config properties (ado-assign-work-item, ado-comment-on-work-item, ado-create-work-item, ado-link-work-items, ado-update-work-item, ado-upload-workitem-attachment) are missing from pkg/parser/schemas/main_workflow_schema.json.

This is a false positive caused by a bug in the checker script itself, not a real schema gap. The properties are, in fact, present in the schema — they are declared inside named $defs entries (e.g. #/$defs/azure_devops_create_work_item) that the top-level safe-outputs properties reference via "$ref".

The Python helper embedded in the script, properties(node) (scripts/check-safe-outputs-conformance.sh:480-488), recursively merges properties/allOf/anyOf/oneOf but never resolves $ref. When it looks up safe_outputs.get(output_name) for an ado-* key, it gets back only {"$ref": "#/$defs/..."}, which has no "properties" key of its own, so properties() returns an empty dict and every field declared in the referenced $defs entry is (incorrectly) flagged as missing.

Affected Components

• Script: scripts/check-safe-outputs-conformance.sh (properties() helper, lines 480-488, used by check_safe_output_config_schema_coverage, lines 428-527)

• False-positive output (verified present in schema, e.g. pkg/parser/schemas/main_workflow_schema.json:13567-13714):

  • safe-outputs.ado-assign-work-item.{allowed,blocked,target}
  • safe-outputs.ado-comment-on-work-item.target
  • safe-outputs.ado-create-work-item.{allowed-tags,area-path,artifact-link,assignee,custom-fields,description-field,iteration-path,tags,work-item-type}
  • safe-outputs.ado-link-work-items.{allowed-link-types,target}
  • safe-outputs.ado-update-work-item.{allowed-area-prefixes,allowed-iteration-prefixes,allowed-tags,area-path,assignee,body,iteration-path,markdown-body,tag-prefix,tags,target,title,title-prefix}
  • safe-outputs.ado-upload-workitem-attachment.{allowed-extensions,comment-prefix,max-file-size,target}

  (Note: one property in this list, ado-update-work-item.status, is a genuine schema gap and is tracked separately.)

🔍 Current vs Expected Behavior

Current Behavior

properties(node) in the embedded Python only inspects node["properties"] and recurses into allOf/anyOf/oneOf children of node itself. It does not dereference $ref pointers, so any schema authored with "$ref": "#/$defs/x" (rather than an inline object) is treated as having zero properties.

Expected Behavior

Per the Safe Outputs Specification, IMP-004 should only flag properties that are genuinely absent from the effective (fully-resolved) JSON schema. The checker should resolve $ref pointers against schema["$defs"] before computing properties(), so config properties declared via $ref are correctly recognized as present.

Remediation Steps

This task can be assigned to a Copilot coding agent with the following steps:

1. In scripts/check-safe-outputs-conformance.sh, update the embedded Python properties(node) helper (around line 480) to resolve "$ref" before reading "properties"/allOf/anyOf/oneOf: when node contains a "$ref" key of the form #/$defs/<name>, look up schema["$defs"][<name>] and recurse into that resolved node instead.
2. Ensure the resolution handles the oneOf: [ {type: object, properties: {...}}, {type: "null"} ] wrapper pattern used throughout main_workflow_schema.json for these Azure DevOps definitions (the existing oneOf handling should compose correctly once $ref is resolved first).
3. Re-run the script and confirm the 32 false-positive ado-* findings listed above disappear, while genuine gaps (see the separate ado-update-work-item.status issue) are still correctly detected.
4. Add/adjust any local test fixture in the script (if one exists) to cover a $ref-based safe-output schema definition, so this regression can't silently reappear.

Verification

After remediation, verify the fix by running:

bash scripts/check-safe-outputs-conformance.sh

IMP-004 should no longer report the ado-* properties listed above as missing.

References

• Safe Outputs Specification: docs/src/content/docs/specs/safe-outputs-specification.md
• Conformance Checker: scripts/check-safe-outputs-conformance.sh
• Run ID: 33841457434
• Date: 2026-09-04

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by ✅ Daily Safe Outputs Conformance Checker · claude · agent · 88.1 AIC · ⌖ 8.16 AIC · ⊞ 7.5K · ◷

• expires on Sep 4, 2026, 9:48 PM UTC-08:00