4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources #605

RobVerduijn · 2021-04-25T12:28:26Z

Describe the bug
Hi,

Today I used the update button to update from 4.7.0-0.okd-2021-04-11-124433 to 4.7.0-0.okd-2021-04-24-103438.
But it seems unable to verify the image.

apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
  creationTimestamp: '2021-04-21T18:21:29Z'
  generation: 2
  managedFields:
    - apiVersion: config.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          .: {}
          'f:channel': {}
          'f:clusterID': {}
          'f:upstream': {}
      manager: cluster-bootstrap
      operation: Update
      time: '2021-04-21T18:21:29Z'
    - apiVersion: config.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:desiredUpdate':
            .: {}
            'f:image': {}
            'f:version': {}
      manager: Mozilla
      operation: Update
      time: '2021-04-25T09:36:15Z'
    - apiVersion: config.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:desiredUpdate':
            'f:force': {}
        'f:status':
          .: {}
          'f:availableUpdates': {}
          'f:conditions': {}
          'f:desired':
            .: {}
            'f:image': {}
            'f:version': {}
          'f:history': {}
          'f:observedGeneration': {}
          'f:versionHash': {}
      manager: cluster-version-operator
      operation: Update
      time: '2021-04-25T09:36:19Z'
  name: version
  resourceVersion: '725979'
  selfLink: /apis/config.openshift.io/v1/clusterversions/version
  uid: dbfdb75c-0e09-41ec-9c37-736b3eba9440
spec:
  channel: stable-4
  clusterID: 6d5876e1-6fea-4f40-be9c-335882390845
  desiredUpdate:
    image: >-
      registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
    version: 4.7.0-0.okd-2021-04-24-103438
  upstream: 'https://origin-release.svc.ci.openshift.org/graph'
status:
  availableUpdates:
    - image: >-
        registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
      version: 4.7.0-0.okd-2021-04-24-103438
  conditions:
    - lastTransitionTime: '2021-04-21T19:15:17Z'
      message: Done applying 4.7.0-0.okd-2021-04-11-124433
      status: 'True'
      type: Available
    - lastTransitionTime: '2021-04-25T12:38:14Z'
      message: >-
        The update cannot be verified: unable to locate a valid signature for
        one or more sources
      reason: ImageVerificationFailed
      status: 'True'
      type: Failing
    - lastTransitionTime: '2021-04-25T09:36:19Z'
      message: >-
        Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe
        to use
      reason: ImageVerificationFailed
      status: 'True'
      type: Progressing
    - lastTransitionTime: '2021-04-25T12:35:03Z'
      status: 'True'
      type: RetrievedUpdates
  desired:
    image: >-
      registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
    version: 4.7.0-0.okd-2021-04-24-103438
  history:
    - completionTime: null
      image: >-
        registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
      startedTime: '2021-04-25T09:36:19Z'
      state: Partial
      verified: false
      version: 4.7.0-0.okd-2021-04-24-103438
    - completionTime: '2021-04-21T19:15:17Z'
      image: >-
        quay.io/openshift/okd@sha256:4414db9e8945504d1d5423ec11d64df9fa8b4aa60d978f8b4bdb3d0720800741
      startedTime: '2021-04-21T18:21:52Z'
      state: Completed
      verified: false
      version: 4.7.0-0.okd-2021-04-11-124433
  observedGeneration: 2
  versionHash: 4boqI_3hclA=

Version

How reproducible

use the update function in okd
Log bundle

Is a log needed ? the update hasn't done anything yet.

Rob

The text was updated successfully, but these errors were encountered:

GerbenWelter · 2021-04-25T15:28:22Z

Hey Rob,

Long time no see/hear. This has happened before. Some image signing process hasn't run yet. Maybe one of the admins need to to restart that process.

vrutkovs · 2021-04-25T17:37:27Z

Right, seems the signer has stuck. We'll restart it, until then please use 'oc adm upgrade --force' or set .spec.desired.force to true in ClusterVersion

RobVerduijn · 2021-04-25T21:16:40Z

I'll wait for the signer to do it's job.

danielchristianschroeter · 2021-04-26T10:39:34Z

We see this almost every month. Is there no sustainable solution for this?

bobby0724 · 2021-04-26T13:22:43Z

I have the same issue

bobby0724 · 2021-04-27T13:17:12Z

Right, seems the signer has stuck. We'll restart it, until then please use 'oc adm upgrade --force' or set .spec.desired.force to true in ClusterVersion

just for general info 'oc adm upgrade --force' didn't work

twling · 2021-04-27T14:41:34Z

Right, seems the signer has stuck. We'll restart it, until then please use 'oc adm upgrade --force' or set .spec.desired.force to true in ClusterVersion

just for general info 'oc adm upgrade --force' didn't work

The second option did work for me to force the update for an unverified image, specifically set .spec.desiredUpdate.force to true

vrutkovs · 2021-04-28T08:59:21Z

$ export RELEASE="4.7.0-0.okd-2021-04-24-103438"                                  
$ oc adm release info quay.io/openshift/okd:${RELEASE} | head -n 2 
Name:           4.7.0-0.okd-2021-04-24-103438
Digest:         sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
$ export DIGEST="1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0"                                                         
$ curl -Ls https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/sha256\=${DIGEST}/signature-1 | gpg -d
{
  "critical": {
    "type": "atomic container signature",
    "image": {
      "docker-manifest-digest": "sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0"
    },
    "identity": {
      "docker-reference": "quay.io/openshift/okd:4.7.0-0.okd-2021-04-24-103438"
    }
  },
  "optional": {
    "creator": "openshift release-controller",
    "timestamp": 1619554498
  }
}gpg: Signature made Tue 27 Apr 2021 22:14:58 CEST
gpg:                using RSA key 65C28371F55D29A9

However the previous signer key has expired, so you'd have to force upgrade this time (and, most likely, the next time too).

Sorry for the inconvenience

OleksandrShtepa · 2021-04-28T15:46:07Z

Got same problem, mitigated:

$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          70s     Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe to use
$ oc adm upgrade --clear=true
Cleared the update field, still at 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          8m4s    Working towards 4.7.0-0.okd-2021-04-11-124433: 601 of 669 done (89% complete)
...
version   4.7.0-0.okd-2021-04-11-124433   True        False         107s    Cluster version is 4.7.0-0.okd-2021-04-11-124433
$ oc adm upgrade --force --to=4.7.0-0.okd-2021-04-24-103438
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          5s      Working towards 4.7.0-0.okd-2021-04-24-103438: downloading update
...
version   4.7.0-0.okd-2021-04-11-124433   True        True          57m     Working towards 4.7.0-0.okd-2021-04-24-103438: 301 of 669 done (44% complete)
...
version   4.7.0-0.okd-2021-04-24-103438   True        False         20s     Cluster version is 4.7.0-0.okd-2021-04-24-103438

bobby0724 · 2021-04-28T20:04:01Z

thanks

bobby0724 · 2021-04-29T05:04:48Z

my upgrade --force take a little while but finally completes thanks

apmlima · 2021-05-11T12:09:23Z

Got same problem, mitigated:

$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          70s     Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe to use
$ oc adm upgrade --clear=true
Cleared the update field, still at 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          8m4s    Working towards 4.7.0-0.okd-2021-04-11-124433: 601 of 669 done (89% complete)
...
version   4.7.0-0.okd-2021-04-11-124433   True        False         107s    Cluster version is 4.7.0-0.okd-2021-04-11-124433
$ oc adm upgrade --force --to=4.7.0-0.okd-2021-04-24-103438
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          5s      Working towards 4.7.0-0.okd-2021-04-24-103438: downloading update
...
version   4.7.0-0.okd-2021-04-11-124433   True        True          57m     Working towards 4.7.0-0.okd-2021-04-24-103438: 301 of 669 done (44% complete)
...
version   4.7.0-0.okd-2021-04-24-103438   True        False         20s     Cluster version is 4.7.0-0.okd-2021-04-24-103438

I got:

oc get clusterversion

NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.0-0.okd-2021-02-14-205305 True True 64m Unable to apply 4.7.0-0.okd-2021-04-24-103438: the control plane is reporting an internal error

Can anyone help?

vrutkovs · 2021-05-15T09:00:21Z

openshift/cluster-update-keys#36 would resolve that. 4.8 nightlies signature verification works.

vrutkovs · 2021-05-22T08:32:59Z

Fixed in 4.7.0-0.okd-2021-05-22-050008.

Note that release verification needs to forced this one time.

kissoliver · 2021-06-03T07:41:24Z

I have the same issue:

The update cannot be verified: unable to locate a valid signature for one or more sources
Unable to apply 4.7.0-0.okd-2021-05-22-050008: the image may not be safe to use

vrutkovs changed the title ~~The update cannot be verified: unable to locate a valid signature for one or more sources~~ 4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources Apr 27, 2021

vrutkovs pinned this issue Apr 28, 2021

vrutkovs mentioned this issue May 9, 2021

[tracking] 4.7.0-0.okd-2021-05-22-050008 #631

Closed

1 task

vrutkovs closed this as completed May 22, 2021

devzeronull mentioned this issue Jun 7, 2021

[OKD4.7 UPI] Upgrade from "4.7.0-0.okd-2021-04-11-124433" to "any newer" fails on machine-config upgrade #666

Closed

vrutkovs mentioned this issue Jun 8, 2021

Image signature verifying error when upgrading to 4.7.0-0.okd-2021-04-24-103438 and laters #667

Closed

vrutkovs unpinned this issue Jun 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources #605

4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources #605

RobVerduijn commented Apr 25, 2021 •

edited

GerbenWelter commented Apr 25, 2021

vrutkovs commented Apr 25, 2021

RobVerduijn commented Apr 25, 2021

danielchristianschroeter commented Apr 26, 2021

bobby0724 commented Apr 26, 2021

bobby0724 commented Apr 27, 2021 •

edited

twling commented Apr 27, 2021

vrutkovs commented Apr 28, 2021 •

edited

OleksandrShtepa commented Apr 28, 2021 •

edited

bobby0724 commented Apr 28, 2021

bobby0724 commented Apr 29, 2021 •

edited

apmlima commented May 11, 2021

vrutkovs commented May 15, 2021

vrutkovs commented May 22, 2021

kissoliver commented Jun 3, 2021

4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources #605

4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources #605

Comments

RobVerduijn commented Apr 25, 2021 • edited

GerbenWelter commented Apr 25, 2021

vrutkovs commented Apr 25, 2021

RobVerduijn commented Apr 25, 2021

danielchristianschroeter commented Apr 26, 2021

bobby0724 commented Apr 26, 2021

bobby0724 commented Apr 27, 2021 • edited

twling commented Apr 27, 2021

vrutkovs commented Apr 28, 2021 • edited

OleksandrShtepa commented Apr 28, 2021 • edited

bobby0724 commented Apr 28, 2021

bobby0724 commented Apr 29, 2021 • edited

apmlima commented May 11, 2021

oc get clusterversion

vrutkovs commented May 15, 2021

vrutkovs commented May 22, 2021

kissoliver commented Jun 3, 2021

RobVerduijn commented Apr 25, 2021 •

edited

bobby0724 commented Apr 27, 2021 •

edited

vrutkovs commented Apr 28, 2021 •

edited

OleksandrShtepa commented Apr 28, 2021 •

edited

bobby0724 commented Apr 29, 2021 •

edited