4.6.0-0.okd-2020-11-27-200126: Invalid signature configmap, needs force: true

[devinmatte]

Describe the bug
Unable to upgrade to 4.6.0-0.okd-2020-12-12-135354 from 4.6.0-0.okd-2020-11-27-200126 due to error

The update cannot be verified: unable to locate a valid signature for one or more sources

Version
4.6.0-0.okd-2020-11-27-200126
Bare Metal Install

How reproducible
100% of the time

Log bundle
Must Gather tar

[vrutkovs]

That's odd, it has

2020-12-12T21:19:01.418695637Z I1212 21:19:01.418429       1 cvo.go:244] Verifying release authenticity: All release image digests must have GPG signatures from verifier-public-key-redhat (567E347AD0044ADE55BA8A5F199E2F91FD431D51: Red Hat, Inc. (release key 2) <security@redhat.com>, B08B659EE86AF623BC90E8DB938A80CAF21541EB: Red Hat, Inc. (beta key 2) <security@redhat.com>) - will check for signatures in containers/image format at serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", parallel signature store wrapping containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release, containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release

instead of expected

I1212 14:30:12.142255       1 cvo.go:244] Verifying release authenticity: All release image digests must have GPG signatures from verifier-public-key-redhat (567E347AD0044ADE55BA8A5F199E2F91FD431D51: Red Hat, Inc. (release key 2) <security@redhat.com>, B08B659EE86AF623BC90E8DB938A80CAF21541EB: Red Hat, Inc. (beta key 2) <security@redhat.com>) - will check for signatures in containers/image format at serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release

Seems OKD is accidentally using official signature store instead of CI

[vrutkovs]

Yup, 0000_90_cluster-update-keys_configmap.yaml in both payloads have invalid store list:

  store-openshift-official-release: https://storage.googleapis.com/openshift-release/official/signatures/openshift/release
  store-openshift-official-release-mirror: https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release

while born-in-4.5 clusters have:

  store-openshift-ci-release: >-
    https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release
  store-openshift-official-release: >-
    https://storage.googleapis.com/openshift-release/official/signatures/openshift/release
  store-openshift-official-release-mirror: 'https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release'

So this would affect born-in-4.6 clusters only. The workaround would be applying https://github.com/openshift/cluster-update-keys/blob/master/manifests/0000_90_cluster-update-keys_configmap.yaml configmap.

[titou10titou10]

So, is there a fix planned for this release and/or is it safe to use"--force"to upgrade?

[vrutkovs]

The fix is planned, but it can't be applied in this release - the problem is that previous 4.6 stable release applied the wrong keys and it went unnoticed.

While we sort it out please use force: true for now

[devinmatte]

Should we use --force or apply the config map linked above? Which is the safer option?

[vrutkovs]

Seems CVO won't accept the updated configmap (it extracts it from the payload iiuc). force: true is the only option which works now :/

How to verify the release manually:

$ curl -Ls https://raw.githubusercontent.com/openshift/cluster-update-keys/master/keys/verifier-public-key-openshift-ci | gpg --import
gpg: key 28B76E05B923888E: public key "openshift-ci" imported
gpg: Total number processed: 1
gpg:               imported: 1
$ oc adm release info quay.io/openshift/okd:4.6.0-0.okd-2020-12-12-135354                                   
Name:      4.6.0-0.okd-2020-12-12-135354
Digest:    sha256:01948f4c6bdd85cdd212eb40d96527a53d6382c4489d7da57522864178620a2c
...
$ export DIGEST="01948f4c6bdd85cdd212eb40d96527a53d6382c4489d7da57522864178620a2c"
$ curl -Ls https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/sha256\=${DIGEST}/signature-1 | gpg -d      
{
  "critical": {
    "type": "atomic container signature",
    "image": {
      "docker-manifest-digest": "sha256:01948f4c6bdd85cdd212eb40d96527a53d6382c4489d7da57522864178620a2c"
    },
    "identity": {
      "docker-reference": "quay.io/openshift/okd:4.6.0-0.okd-2020-12-12-135354"
    }
  },
  "optional": {
    "creator": "openshift release-controller",
    "timestamp": 1607787341
  }
}gpg: Signature made Sat 12 Dec 2020 16:35:41 CET
gpg:                using RSA key 28B76E05B923888E
gpg: Good signature from "openshift-ci" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: D047 61B1 1620 3B0C 0859  B616 28B7 6E05 B923 888E

So the encrypted message generated on CI has valid digest, signed with openshift-ci key and uploaded to storage.googleapis.com/openshift-ci-release.

The problem is that previous 4.6 release has changed the expected key to openshift-release and location to mirror.openshift.com (like in OCP) and this went unnoticed :/

[chriskuipers]

If you're like us, and started the upgrade from the Web interface, you first need to clear the upgrade and "downgrade" to the old version again, verify the image with the steps Vrutkovs proposed. If that checks out you can force the upgrade:

$ oc adm upgrade --clear
$ oc adm upgrade --force --to=4.6.0-0.okd-2020-11-27-200126 
$ oc adm upgrade --force --to=4.6.0-0.okd-2020-12-12-135354

[vrutkovs]

That's not necessary, edit ClusterVersion cluster object and set spec.desired.force: true

[ssams]

That's not necessary, edit ClusterVersion cluster object and set spec.desired.force: true

correct config entry is spec.desiredUpdate.force: true, but otherwise this seemed to work in my case

[vrutkovs]

Keeping open for openshift/cluster-update-keys#30 (and nightly) to be available

[vrutkovs]

https://amd64.origin.releases.ci.openshift.org/releasestream/4.6.0-0.okd/release/4.6.0-0.okd-2021-01-15-162431 should have updated expected keys.

However release-controller doesn't upload new signatures just yet - tracking this issue w/ infra folks

[vrutkovs]

This seems to work now:

$ oc adm release info registry.ci.openshift.org/origin/release:4.6.0-0.okd-2021-01-15-162431 | grep Digest
Digest:    sha256:765312811b89e90ec6e2bc4fe2a57f67dbef39443b1c44738b648054e71b64e1
$ curl -Ls https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/sha256\=765312811b89e90ec6e2bc4fe2a57f67dbef39443b1c44738b648054e71b64e1/signature-1 | gpg -d
{
  "critical": {
    "type": "atomic container signature",
    "image": {
      "docker-manifest-digest": "sha256:765312811b89e90ec6e2bc4fe2a57f67dbef39443b1c44738b648054e71b64e1"
    },
    "identity": {
      "docker-reference": "registry.ci.openshift.org/origin/release:4.6.0-0.okd-2021-01-15-162431"
    }
  },
  "optional": {
    "creator": "openshift release-controller",
    "timestamp": 1610734946
  }
}gpg: Signature made Fri 15 Jan 2021 19:22:26 CET
gpg:                using RSA key 28B76E05B923888E
gpg: Good signature from "openshift-ci" [unknown]