Scorecard does not update after adding annotation for dangerous workflow

[matheuscscp]

Hi 👋

We just added this file to CNCF Flux:

https://github.com/fluxcd/flux2/blob/main/.scorecard.yml

According to these docs: https://github.com/ossf/scorecard/tree/main/config

This seems to have fixed the security issue in the repo last week: https://github.com/fluxcd/flux2/security/code-scanning/134

But our Scorecard is not updating the "dangerous workflow" issue, the Scorecard updated today and the issue is still there: https://scorecard.dev/viewer/?uri=github.com/fluxcd/flux2

[spencerschrock]

Thanks for the issue. You've unfortunately hit one of our known issues with Danergous-Workflow:

scorecard/docs/checks.md

Lines 289 to 290 in 0c205a0

	token in memory. This check does not detect whether untrusted code checkouts are
	used safely, for example, only on pull request that have been assigned a label.

This is a great time to use Maintainer Annotations to provide more context + silence the alert. But annotations aren't necessarily about changing the score. We've talked about the idea of presenting the actual score alongside an annotated score, but currently none of this is hooked up to the API, but you can see what it'd look like locally:

scorecard --repo fluxcd/flux2 --checks Dangerous-Workflow --show-details --format json --show-annotations | jq
INFO[0004] using maintainer annotations: .scorecard.yml 
{
  "date": "2025-04-11T10:14:44-06:00",
  "repo": {
    "name": "github.com/fluxcd/flux2",
    "commit": "7b551b0d3572714b8f77f4aa5e6575e79f7f3d20"
  },
  "scorecard": {
    "version": "v5.1.1",
    "commit": "unknown"
  },
  "score": 0.0,
  "checks": [
    {
      "details": [
        "Warn: untrusted code checkout '${{ github.event.pull_request.head.sha }}': .github/workflows/backport.yaml:18"
      ],
      "score": 0,
      "reason": "dangerous workflow patterns detected",
      "name": "Dangerous-Workflow",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#dangerous-workflow",
        "short": "Determines if the project's GitHub Action workflows avoid dangerous patterns."
      },
      "annotations": [
        "The check or probe is not applicable in this case."
      ]
    }
  ],
  "metadata": null
}

[matheuscscp]

So the annotations don't actually change the score? There's nothing we can do to change this score without changing the workflow?

[github-actions]

This issue has been marked stale because it has been open for 60 days with no activity.

[matheuscscp]

So the annotations don't actually change the score? There's nothing we can do to change this score without changing the workflow?

Still waiting for an answer here if possible 🙏

[spencerschrock]

So the annotations don't actually change the score? There's nothing we can do to change this score without changing the workflow?

Still waiting for an answer here if possible 🙏

The other alternative would be for Scorecard to try to parse the extra protections.

if: github.event.pull_request.state == 'closed' && github.event.pull_request.merged && (github.event_name != 'labeled' || startsWith('backport:', github.event.label.name))

Possible fields of interest include the fact that it's labeled or that it's merged. It may be easier for Scorecard to reason about if the pull_request_trigger was only for the labeled event, but I imagine you have closed too so the order of operations doesn't matter.