Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/governance-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -997,6 +997,28 @@ jobs:
# governance jobs on every PR estate-wide. github.sha resolves to the
# same merge commit but is always fetchable.
ref: ${{ github.sha }}
- name: Checkout standards for the duplicate-key check
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: hyperpolymath/standards
ref: main
path: .standards-dupkey
sparse-checkout: scripts/check-workflow-duplicate-keys.py
sparse-checkout-cone-mode: false

- name: Duplicate YAML keys in workflows
run: |
# GitHub Actions REJECTS a workflow with duplicate keys: the run is
# `failure` with no jobs, no log and no check run. Nothing else here
# can see it, because yaml.safe_load silently keeps the LAST
# duplicate and reports success — so the file "parses" and every
# other lint passes. Measured 2026-08-05: nine workflows in hypatia
# were dead this way, including a CodeQL workflow with zero
# successful runs in its entire lifetime.
cp .standards-dupkey/scripts/check-workflow-duplicate-keys.py "$RUNNER_TEMP/"
rm -rf .standards-dupkey
python3 "$RUNNER_TEMP/check-workflow-duplicate-keys.py" .github/workflows

- name: Check SPDX headers + permissions
run: |
failed=0
Expand Down
100 changes: 100 additions & 0 deletions scripts/check-workflow-duplicate-keys.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MPL-2.0
"""Reject GitHub Actions workflows containing duplicate YAML keys.

WHY THIS EXISTS AS A SEPARATE CHECK
-----------------------------------
GitHub Actions rejects a workflow with duplicate keys outright. The run is
recorded as `failure` with NO jobs, NO log and NO check run — a red mark on
the board with nothing behind it to read, and `gh pr checks` shows no row at
all.

Nothing else in the toolchain can see this, because `yaml.safe_load` SILENTLY
KEEPS THE LAST duplicate and reports success. The file "parses". Every
ordinary validation — linters, formatters, our own sweep scripts — is
structurally blind to it.

Measured 2026-08-05: nine workflows in `hypatia` were in this state, including
a CodeQL workflow with 18 failures, 12 startup_failures and ZERO successes in
its lifetime. The repository had never once been scanned by its own scanner.

USAGE
-----
check-workflow-duplicate-keys.py [PATH ...] # default: .github/workflows

Exit 0 when clean, 1 when any duplicate is found.
"""
import glob
import os
import sys

import yaml


class StrictLoader(yaml.SafeLoader):
"""A SafeLoader that refuses duplicate mapping keys instead of silently
keeping the last one."""


def _no_duplicates(loader, node, deep=False):
mapping = {}
dupes = []
for key_node, value_node in node.value:
key = loader.construct_object(key_node, deep=deep)
if key in mapping:
dupes.append((key, key_node.start_mark.line + 1))
mapping[key] = loader.construct_object(value_node, deep=deep)
if dupes:
detail = ", ".join(f"{k!r} (line {ln})" for k, ln in dupes)
raise yaml.YAMLError(f"duplicate key(s): {detail}")
return mapping
Comment on lines +39 to +50

@gitar-bot gitar-bot Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Edge Case: Unhashable complex mapping keys crash with uncaught TypeError

_no_duplicates constructs each key and does key in mapping / mapping[key] = .... If a workflow uses a complex YAML key (a mapping or sequence key), construct_object returns an unhashable dict/list and the membership/assignment raises TypeError, which is not caught by the yaml.YAMLError/OSError handlers in check() and aborts the whole run. This is unlikely in real workflows but would make the linter itself crash rather than report cleanly; wrap the dict operations or catch TypeError.

Fix:

    except (yaml.YAMLError, TypeError) as exc:
        return str(exc).replace("
", " ")[:160]

Was this helpful? React with 👍 / 👎



StrictLoader.add_constructor(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _no_duplicates
)


def check(path):
"""Return a problem string, or None when the file is fine."""
try:
with open(path, encoding="utf-8") as fh:

Check failure on line 61 in scripts/check-workflow-duplicate-keys.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

LLMs running this code with faulty CLI arguments can escape file system restrictions. Refactor this code to validate the constructed path before accessing the file system.

See more on https://sonarcloud.io/project/issues?id=hyperpolymath_standards&issues=AZ_UyQMNf4sIwMWSgtEH&open=AZ_UyQMNf4sIwMWSgtEH&pullRequest=582
yaml.load(fh, StrictLoader)
except yaml.YAMLError as exc:
return str(exc).replace("\n", " ")[:160]
except OSError as exc:
return f"unreadable: {exc}"
return None
Comment on lines +60 to +67

@gitar-bot gitar-bot Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Edge Case: Any YAML syntax error is mislabeled as a duplicate key

check() catches every yaml.YAMLError (including plain syntax/scanner errors, not just the duplicate-key error raised by _no_duplicates), and the summary printed on failure asserts the files "contain duplicate keys" and that "yaml.safe_load does NOT catch this." A file with an unrelated parse error will therefore be reported as a duplicate-key violation, misdirecting the fix. Consider raising a dedicated exception subclass from _no_duplicates and only attributing the duplicate-key message when that specific error is caught, otherwise report it as a generic parse error.

Use a distinct exception so genuine syntax errors are not mislabeled.:

class DuplicateKeyError(yaml.YAMLError):
    pass

# in _no_duplicates:
    if dupes:
        detail = ", ".join(f"{k!r} (line {ln})" for k, ln in dupes)
        raise DuplicateKeyError(f"duplicate key(s): {detail}")

# in check(): distinguish DuplicateKeyError from other YAMLError to label
# the failure accurately.

Was this helpful? React with 👍 / 👎



def main(argv):
targets = argv[1:] or [".github/workflows"]
files = []
for t in targets:
if os.path.isdir(t):
for ext in ("yml", "yaml"):
files.extend(sorted(glob.glob(os.path.join(t, f"*.{ext}"))))

Check failure on line 76 in scripts/check-workflow-duplicate-keys.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

LLMs running this code with faulty CLI arguments can escape file system restrictions. Refactor this code to validate the constructed path before accessing the file system.

See more on https://sonarcloud.io/project/issues?id=hyperpolymath_standards&issues=AZ_UyQMNf4sIwMWSgtEI&open=AZ_UyQMNf4sIwMWSgtEI&pullRequest=582
else:
files.append(t)

failed = 0
for f in files:
problem = check(f)
if problem:
print(f"::error file={f}::{problem}")
print(f"FAIL {f}: {problem}")
failed += 1

if failed:
print(f"\n{failed} of {len(files)} workflow file(s) contain duplicate keys.")
print("GitHub Actions rejects these before any job is created — they")
print("fail with no log and no check run. yaml.safe_load does NOT")
print("catch this; it keeps the last duplicate and reports success.")
return 1

print(f"duplicate-key check: {len(files)} workflow file(s) clean")
return 0


if __name__ == "__main__":
sys.exit(main(sys.argv))
Loading