contributor-check returns UNKNOWN on transient failure with no retry, producing false needs-review flags

[imran-siddique]

Summary

contributor-check returns Profile: UNKNOWN on transient failures with no retry. Because the risk ordering is deliberately fail-closed, UNKNOWN outranks LOW, so a momentary GitHub API blip labels an established contributor as more suspicious than a genuinely low-risk one. This has already produced three false flags on real submissions.

Observed

Three issues from long-standing accounts were labeled needs-review:UNKNOWN, all reporting Profile: UNKNOWN, Credential: LOW:

Issue	Author	Account created	Public repos
agentrust-io/agent-manifest#272	Mayur021	2014-05	57
agentrust-io/trace-spec#116	lywinged	2019-02	24
agentrust-io/trace-spec#117	lywinged	2019-02	24

None of these are plausible risk signals. For contrast, the same action returned a real, correctly-computed Profile: HIGH on integrations#46, so the check works when the lookup succeeds.

I have removed the misleading label from all three. They should be re-run once this is fixed.

Root cause

.github/actions/contributor-check/action.yml does not run its own logic. It checks out microsoft/agent-governance-toolkit at pinned ref 359a6b8cf453f95d6bc9caf932057e1e78ccffd9 (main as of 2026-07-31), sparse-checkout scripts, and executes _agt/scripts/contributor_check_action.py.

In that file:

# scripts/contributor_check_action.py
result = subprocess.run(
    ...
    timeout=120,
)
risk = data.get("risk", "UNKNOWN")
...
except Exception:
    return "UNKNOWN"

Any exception at all, including a timeout or a single failed API call, collapses to UNKNOWN. There is no retry.

The fail-closed ordering is intentional and correct in principle:

# Fail-closed ordering: UNKNOWN ("could not be determined") must outrank LOW
RISK_ORDER = {"LOW": 1, "MEDIUM": 2, "UNKNOWN": 3, "HIGH": 4}

The problem is that nothing tries hard enough before giving up, so fail-closed fires on noise.

Why this is not fixed by microsoft/agent-governance-toolkit#3571

AGT#3571 tracks contributor_check.py and credential_audit.py having diverged into two copies. The exponential-backoff retry added in AGT#2196 went to the packaged agent_compliance/cli/ copy only, and was never backported to scripts/.

This action consumes the scripts/ copy, which is the one without the retry. The only backoff-adjacent code in scripts/contributor_check.py is a single Retry-After header read on HTTPError (line 88), not a general retry loop.

So the two are related but distinct: resolving AGT#3571 upstream may fix this, but only if the reconciliation lands the retry in scripts/ and this action's pinned ref is advanced past it. Today it is pinned to 2026-07-31.

Suggested fix

1. Add a bounded retry with exponential backoff around the profile lookup before returning UNKNOWN, so UNKNOWN means "checked and could not determine" rather than "one call failed".
2. Distinguish UNKNOWN from ERROR in the output, so a genuine indeterminate result and an infrastructure failure are not the same label. Only the former deserves a needs-review label.
3. Decide whether to keep consuming AGT scripts/ at a pinned SHA. That couples our contributor gating to an upstream repo's internal script layout and to whichever copy of a known-diverged pair happens to be there. Vendoring the checker, or depending on the published package instead, would decouple us.
4. Once fixed, re-run against Mayur021 and lywinged to confirm they resolve to a real level.

Items 1 and 2 are the fix. Item 3 is worth a separate decision.

[imran-siddique]

Second round of false flags, and they are all on one person.

Every open issue carrying needs-review:UNKNOWN right now was lywinged's: trace-spec#124, #128, #138, #142 and trace-tests#53. Removed all five.

Worth recording because it sharpens the cost. Three of those four trace-spec issues were accurate enough to become PRs today (#146, #147, #148), and #138 found a real signature-verification defect in trace-verify. The check is applying its most suspicious label, on every submission, to the most productive external contributor this project has, purely because a transient lookup outranks LOW under the fail-closed ordering.

That is now five false flags across two rounds, versus one correctly-computed HIGH on integrations#46. Items 1 and 2 in the description are still the fix; this comment is only to note the pattern is recurring rather than one-off, and that it lands on the same account every time.