Scorecard action v2.0.0 fails with 500 Internal Server Error

[irongut]

The Scorecard Action fails for one of my projects with 500 Internal Sever Error after what apppears to be a successful scan. This has been happening since I upgraded to v2.0.0 yesterday, prior to that scans were working.

tlog entry created with index: 3457369
MEYCIQDDQkKJ7w/FDTNeipLY+rhyHvsgMl6rdUzmsvP3GLJuvAIhALVjmFXe0FgVA+mEKekcjgPy6Eob/dV6WnyghS5vp+/9
2022/09/10 18:54:33 error processing signature: http response 500, status: 500 Internal Server Error, error: {"code":500,"message":"something went wrong and we are looking into it."}

Project: irongut/EditRelease
Workflow: openssf-scorecards.yml
Failing Run: https://github.com/irongut/EditRelease/runs/8286780953?check_suite_focus=true

[azeemshaikh38]

Hi @irongut please see https://github.com/ossf/scorecard-action#breaking-changes-in-v2. Since we use GitHub's OIDC token to ensure workflow integrity we only allow specific steps in the same job as the scorecard-action. The lines causing the failure in your case are here: https://github.com/irongut/EditRelease/blob/master/.github/workflows/openssf-scorecards.yml#L27-L30. You'll have to either: (i) specify that step in a separate job or (ii) set publish_results: false for scorecard-action.

[irongut]

That step can't run in a separate job, it is a security agent that protects against several supply chain attack vectors and needs to run in the job that it is being protected.

So what you're telling me is that OpenSSF Scorecards won't work with StepSecurity's Harden Runner security agent? Scorecards suggests StepSecurity Harden Workflows, which adds Harden Runner automatically, as remediation for several issues. So on the one hand you recommend StepSecurity but on the other hand your tools are not actually compatible.

This makes OpenSSF Scorecards no longer usable because it reduces supply chain security. (at least if you wanted a badge)

[azeemshaikh38]

@laurentsimon @asraa could we do away with workflow verification altogether to make this more flexible? One idea is to make the API POST request require an API key. We could bake in the API key when building the Docker image in CloudBuild.

We'll also need to separately verify if the workflow which signed the payload is the one making the POST API call. For this, we send the OIDC JWT in the POST payload and verify it's the same token used to sign the payload. Wdyt?

[laurentsimon]

We can't bake in an API key: the binary will contain it and anyone will be able to extract it.

@irongut we recommend using the web UI to remediation, we never ask to use the hardened runner. Can you point to the documentation that recommends installing the hardened runner?

[azeemshaikh38]

I have added an exception for step-security/harden-runner. Closing this issue.

[jauderho]

This still seems to be problematic for me. Got here as I am finally getting around to investigating why the v2 action was not working for me.

After going through and adding the id-token: write perms, it still did not appear to be working. At this point, I've also commented out the harden-runner but am still getting the 500 error.

Here's the run: https://github.com/jauderho/ghafiles/actions/runs/3715223655/jobs/6300125369#step:4:1156
Workflow file: https://github.com/jauderho/ghafiles/blob/main/.github/workflows/scorecard.yml

Any ideas would be appreciated.

Error message is
error processing signature: http response 500, status: 500 Internal Server Error, error: {"code":500,"message":"something went wrong and we are looking into it."}

[laurentsimon]

Thanks for the report. This is a bug in our implementation https://github.com/ossf/scorecard-webapp/blob/main/app/server/verify_workflow.go#L94
You're setting a runner value which is very recent and have not accounted for. Set it to ubuntu-latest for now. I will send a PR to add ubuntu-22.04 as an accepted runner. Sorry for the inconvenience

[jauderho]

FWIW, ubuntu-latest is due to default to ubuntu-22.04. See actions/runner-images#6399

[laurentsimon]

I know! We just need to keep our list updated, because we can't dynamically infer what ubuntu-latest point to, AFAIK. But if you now how to reliably do that, please let us know.

[jauderho]

Unfortunately, no.

I do not like guessing what -latest points to either which is why I prefer to point to an explicit Ubuntu version.

[laurentsimon]

@jauderho I've applied the changes. Can you verify that it works with your setup?

[laurentsimon]

still not working in my test. Need to dig further

[laurentsimon]

working in my test now https://github.com/laurentsimon/scorecard-action-test-3/actions/runs/3733808955/jobs/6335072541.

Closing this issue. Re-open if it still triggers on your end. Thanks!