[bug] Creating and signing provenance fails because retrieving signed certificate fails.

[jenstroeger]

Describe the bug

Creating signed provenance fails with

##[debug]/usr/bin/bash --noprofile --norc -e -o pipefail /home/runner/work/_temp/12f3ed8f-b71f-4b24-816e-9adc5a6a9d03.sh
Retrieving signed certificate...

        Note that there may be personally identifiable information associated with this signed artifact.
        This may include the email address associated with the account with which you authenticate.
        This information will be used for signing this artifact and will be stored in public transparency logs and cannot be removed later.
validating log entry: unable to fetch Rekor public keys from TUF repository, and not trusting the Rekor API for fetching public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: invalid key
remote status:{
	"mirror": "sigstore-tuf-root",
	"metadata": {
		"root.json": {
			"version": 5,
			"len": 6388,
			"expiration": "18 Apr 23 18:13 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 53,
			"len": 1973,
			"expiration": "10 Nov 22 21:10 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 5,
			"len": 4188,
			"expiration": "18 Apr 23 18:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 53,
			"len": 719,
			"expiration": "03 Nov 22 21:10 UTC",
			"error": ""
		}
	}
}
Error: Process completed with exit code 1.
##[debug]Finishing: Create and sign provenance

To Reproduce

This happens in a private repository, based on the this job:

  provenance:
    needs: build
    # The generator should be referenced with a semantic version.
    # The build will fail if we reference it using the commit sha.
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.2.0
    with:
      base64-subjects: ${{ needs.build.outputs.artifacts-sha256 }}
    permissions:
      actions: read # To read the workflow path.
      id-token: write # To sign the provenance.
      contents: write # To add assets to a release.

Expected behavior

Finish successfully.

Screenshots

Additional context

I’m going to be honest: I’m trying to convince people in my org to integrate SLSA into the CI and build processes, but it’s getting harder when the actions break with different issues every other week.

I just updated to v1.2.1 and hope that’ll work.

[jenstroeger]

And v1.2.1 fails with

Error: some/repo: Repository is private. The workflow has halted in order to keep the repository name from being exposed in the public transparency log. Set 'private-repository' to override.

which isn’t documented. So based on the src I guess I need to use

    with:
      private-repository: true

and that means that the name of the repository “leaks”… where’s that transparency log accessible?

[ianlewis]

And v1.2.1 fails with

Error: some/repo: Repository is private. The workflow has halted in order to keep the repository name from being exposed in the public transparency log. Set 'private-repository' to override.

which isn’t documented. So based on the src I guess I need to use

    with:
      private-repository: true

and that means that the name of the repository “leaks”… where’s that transparency log accessible?

Yeah, forgetting to add that to the docs was an oversight on our part. The reason for it is that the repository name is uploaded as part of the transparency log entry on the public Rekor instance (rekor.sigstore.dev) and can be found via the Rekor API. This might be considered a "leak" if the repo is private but you can explicitly opt-in if you are ok with that. I will create a PR to document it today.

For those who don't want repo names to leak on the public Rekor instance, we have an issue to track support for using a private Rekor instance (#34) and we may prioritize adding that now that Rekor 1.0 was released and the API and public instance will hopefully be a bit more stable. It's been a big concern for us so we've been working with that team to figure out ways to catch issues earlier so they don't end up affecting users.

[jenstroeger]

@ianlewis thanks for the answer, adding private-repository worked.

However, I want to emphasize that this bug still applies to v1.2.0 and a fix update & bump to v1.2.1 should not leave behind a broken service!

[ianlewis]

Thanks for the feedback. We'll try to adhere to semantic versioning better going forward.

[jenstroeger]

@ianlewis what are you saying? That you folks intentionally broke v1.2.0? You’ve changed this issue to a documentation change… what’s with the invalid key that breaks the generator?

[ianlewis]

I'm saying that the private-repository input is working as intended. It's an opt in flag for users to indicate that it's ok for private repository names to be uploaded to the public transparency log.

However, since it wasn't a simple bugfix upgrade from v1.2.0 to v1.2.1 without adding that input we probably should have set the version at 1.3.0 to indicate that.

private-repository doesn't have anything to do with v1.2.0 though. It was added in v1.2.1.

[jenstroeger]

Regarding v1.2.0: is that version dead and deprecated, or is it supposed to be functioning (because it isn’t, hence this bug report).

[ianlewis]

Ok, I see. Sorry for the confusion. Yes. Version v1.2.0 is broken for reasons that are outside the scope of this project and is not likely to get fixed.

I'll try to add some more doc to communicate that better.

[jenstroeger]

Yes. Version v1.2.0 is broken for reasons that are outside the scope of this project and is not likely to get fixed.

So… the supply chain tools are broken by their own supply chain? And by being broken as v1.2.0 is it itself becomes a faulty link in the software supply chain.

Ironic.

[ianlewis]

Yes. Version v1.2.0 is broken for reasons that are outside the scope of this project and is not likely to get fixed.

So… the supply-chain tools are broken by their own supply chain? Ironic.

Indeed... We're working with them to improve things and catch them before they become a problem but they've been moving fast...

[ianlewis]

I added a note to the v1.2.0 release indicating that users should upgrade.

[jenstroeger]

Frankly, that’s very disagreeable and frustrating 🙁

I suggest keeping the title of this issue as it was so people who still use v1.2.0 and who will now run into this problem can find this issue.

Adding documentation to the v1.2.1 docs regarding the private-repository should be separate from this rather monumental problem here.

[ianlewis]

Ok, I'll create a separate issue for that. Sorry for all the confusion. I misunderstood your core problem and thought this was more of a documentation issue.

[ianlewis]

@jenstroeger Were you able to try it with v1.2.1? I'm now seeing failures like yours for v1.2.1 as well...

FAILED: SLSA verification failed: could not find a matching valid signature entry: got unexpected errors updating local metadata and targets: error updating to TUF remote mirror: tuf: invalid key

https://github.com/slsa-framework/slsa-github-generator/actions/runs/3334679611/jobs/5517919607

(not sure if you can actually see that url but just adding for posterity)

The workaround seems to be to add compile-generator: true again 😕

with:
  compile-generator: true

[jenstroeger]

@ianlewis we’ve enabled compile-generator ever since issue #942, so running v1.2.1 worked for me. And one more reason to document that option and make it a feature (or use it always).

[ianlewis]

@jenstroeger Yeah, I'll probably just go ahead and document it (#1166). It was really only meant for our pre-submit tests because this kind of issue with Rekor is never supposed to happen but here we are...

[asraa]

Were you able to try it with v1.2.1? I'm now seeing failures like yours for v1.2.1 as well..

I can chime in very quick but am travelling after a conference :|

Sigstore was using a non-compliant TUF rootversion, and a compliant version got pushed last night to the remote repository.

The old libs sigstore (Go) was using to get the root verification material did not support compliant versions. I can update later, but this problem wasn't detected in our release workflows because we don't test sigstore root layouts (@kpk47 testing against pre-prod would have caught these issues, but only given us a 3 day window to fix issues). Sigstore's TUF root is inherently moving and has breaking changes because of using broken underlying libraries.

We'll be able to fix, at least, I can relay and have plans to push the TUF root changes to staging (which gives a lot more of a window than pre-prod).

More info:
The fixes were present in v1.4.2+ sigstore/sigstore and v1.13.1+ of sigstore/cosign.

[gal-legit]

Getting the same kind of errors with the go builder for a private repository:

  # Trusted builder.
  build:
    permissions:
      id-token: write # To sign the provenance.
      contents: write # To upload assets to release.
      actions: read # To read the workflow path.
    needs: args
    uses: slsa-framework/slsa-github-generator/.github/workflows/builder_go_slsa3.yml@1.2.2
    with:
      go-version: 1.17
      # Optional: only needed if using ldflags.
      evaluated-envs: "COMMIT_DATE:${{needs.args.outputs.commit-date}}, COMMIT:${{needs.args.outputs.commit}}, VERSION:${{needs.args.outputs.version}}, TREE_STATE:${{needs.args.outputs.tree-state}}"
      private-repository: true

Referencing either v1.2.2 or main results in:

Run ./__BUILDER_CHECKOUT_DIR__/.github/actions/generate-builder/generate-builder.sh
Fetching the builder with ref: refs/tags/v1.2.2
./__BUILDER_CHECKOUT_DIR__/.github/actions/generate-builder/generate-builder.sh: line 22: .github/actions/generate-builder/builder-fetch.sh: No such file or directory
Error: Process completed with exit code 127.

Referencing v1.2.1 results in:

unable to fetch Rekor public keys from TUF repository: verifying tlog entry 24296fb24b8ad77a09fe522f9a80b80e39769fce01ccb678e76b740491c5c3a7b1e22978ff0fa541

[ianlewis]

@gal-legit 1.2.2 is a pre-release and not intended to be used until it's been fully released. We should probably clarify a bit in the docs but pre-releases are needed for our release process and aren't "beta" releases or anything. They are not-fully baked or tested.

[asraa]

Referencing v1.2.1 results in:

For this, see the above thread, but adding the compile-builder: true option under with: inputs will workaround.

[ianlewis]

v1.2.2 was released and includes fixes for this issue. Users should upgrade to this release to solve this issue. Users should no longer need to set the compile-builder input workaround after upgrade.

v1.2.1 is deprecated and should not be used.

[sgammon]

This is happening again, starting today... @ianlewis

Job log

Verifying artifact slsa-generator-generic-linux-amd64: FAILED: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}

[ianlewis]

This is happening again, starting today... @ianlewis

Job log

Verifying artifact slsa-generator-generic-linux-amd64: FAILED: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
	"mirror": "https://tuf-repo-cdn.sigstore.dev/",
	"metadata": {
		"root.json": {
			"version": 9,
			"len": 6766,
			"expiration": "12 Sep 24 06:53 UTC",
			"error": ""
		},
		"snapshot.json": {
			"version": 132,
			"len": 2302,
			"expiration": "09 Apr 24 16:16 UTC",
			"error": ""
		},
		"targets.json": {
			"version": 9,
			"len": 5478,
			"expiration": "12 Sep 24 06:13 UTC",
			"error": ""
		},
		"timestamp.json": {
			"version": 169,
			"len": 723,
			"expiration": "26 Mar 24 16:16 UTC",
			"error": ""
		}
	}
}

Yeah, Sigstore updates their TUF root keys for the public instance and expects everyone to upgrade to their latest version of the sigstore client. It makes it a bit difficult for downstream users like us to keep up or even know when breakages are coming (even when we are a sigstore adjacent project...)