Skip to content

add retry for skopeo and no_proxy for s3.us-east-1.amazonaws.com#55503

Merged
openshift-merge-bot[bot] merged 1 commit intoopenshift:masterfrom
jianzhangbjz:skopeo
Aug 14, 2024
Merged

add retry for skopeo and no_proxy for s3.us-east-1.amazonaws.com#55503
openshift-merge-bot[bot] merged 1 commit intoopenshift:masterfrom
jianzhangbjz:skopeo

Conversation

@jianzhangbjz
Copy link
Copy Markdown
Member

Add retry for Skopeo to address https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive/1822786217326940160/artifacts/aws-c2s-ipi-disc-priv-fips-f28-destructive/enable-qe-catalogsource-disconnected/build-log.txt

Copying blob sha256:c7965aa7086045a59bdb113a1fb8a19d7ccf7af4133e59af8ecefd39cda8e0b1
time="2024-08-12T01:14:18Z" level=fatal msg="copying image 1/4 from manifest list: reading blob sha256:c7965aa7086045a59bdb113a1fb8a19d7ccf7af4133e59af8ecefd39cda8e0b1: Get \"https://s3.us-east-1.amazonaws.com/quayio-production-s3/sha256/c7/c7965aa7086045a59bdb113a1fb8a19d7ccf7af4133e59af8ecefd39cda8e0b1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI5LUAQGPZRPNKSJA%2F20240812%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240812T011418Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=89b46f04473b3c5b4a9014966d44fe0a7e228b6199c05a1f3531fd2dee79deed\": Service Unavailable"
Error! skopeo copy catalog failed, abort !

@jianzhangbjz
Copy link
Copy Markdown
Member Author

/assign @rioliu-rh @anpingli

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 13, 2024
@jianzhangbjz
Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@@ -146,7 +146,26 @@ function mirror_optional_images () {
skopeo login quay.io/openshift-qe-optional-operators -u ${optional_auth_user} -p ${optional_auth_password}

echo "skopeo copy docker://${origin_index_image} oci://${work_dir}oci-local-catalog --remove-signatures"
Copy link
Copy Markdown
Contributor

@anpingli anpingli Aug 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a slash here? I made a mistake here.
echo "skopeo copy docker://${origin_index_image} oci://${work_dir}**/**oci-local-catalog --remove-signatures"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jianzhangbjz sorry, can you remove **?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be 'skopeo copy docker://${origin_index_image} oci://${work_dir}/oci-local-catalog --remove-signatures"

@rioliu-rh
Copy link
Copy Markdown
Contributor

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@rioliu-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianzhangbjz
Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianzhangbjz
Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianzhangbjz
Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianzhangbjz
Copy link
Copy Markdown
Member Author

Failed after 3 retries, https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/pr-logs/pull/openshift_release/55503/rehearse-55503-periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive/1823528139754573824/artifacts/aws-c2s-ipi-disc-priv-fips-f28-destructive/enable-qe-catalogsource-disconnected/build-log.txt

Date=20240814T021940Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f43943fe7edac449b6f986e2e9333d7f4bee9b26dd7c2eca8c36af395a96c8d7\": Service Unavailable"
Retry 2/3...
Copying 2 of 2 images in list
Copying image sha256:5d7f940e5704b34cb2330e1498d428fe95b0b6ef4afba04c87c698f20166e926 (1/2)
Copying blob sha256:fee6d4b2035aeacafcd858da273d628708fd9e5b24be62451893aa9e4159ff19
Copying blob sha256:2e9427b8c823ce724ae0b71b43b521e9729883236c4ec7d1e8e25f661e35fe24
Copying blob sha256:b7f09ba47d002b4216d2c0372502bff993e1ce93f45b9ff3a15d922da13ed319
Copying blob sha256:d2da84f6e5b72b37214fb486accfc8764a26f91a31d4cb6f6f593c4a8246a675
Copying blob sha256:fb59b26fc1696069c946779723a5dcc9c6c91dcda16a66703a11306c441b80f5
Copying blob sha256:166f5782bb9af344492ecb2070587653be097a0ff142dfb9dc78db770b23535f
time="2024-08-14T02:20:10Z" level=fatal msg="copying image 1/2 from manifest list: reading blob sha256:d2da84f6e5b72b37214fb486accfc8764a26f91a31d4cb6f6f593c4a8246a675: Get \"https://s3.us-east-1.amazonaws.com/quayio-production-s3/sha256/d2/d2da84f6e5b72b37214fb486accfc8764a26f91a31d4cb6f6f593c4a8246a675?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI5LUAQGPZRPNKSJA%2F20240814%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240814T022010Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=16b45b1200ad882af80c5eb0b2b75bc0779b42e3ad2351460b65e4a6d481d1a5\": Service Unavailable"
Retry 3/3...
Failed after 3 retries

@jianzhangbjz
Copy link
Copy Markdown
Member Author

Hi @anpingli , could you help have a look? Thanks!

[cloud-user@preserve-olm-env2 ~]$ podman pull quay.io/openshift-qe-optional-operators/aosqe-index:v1.25
Trying to pull quay.io/openshift-qe-optional-operators/aosqe-index:v1.25...
Error: initializing source docker://quay.io/openshift-qe-optional-operators/aosqe-index:v1.25: too many requests to registry

@jianzhangbjz
Copy link
Copy Markdown
Member Author

As for the retry mechanism, I think it's good. @kuiwang02 , could you help approve it? Thanks!
/pj-rehearse ack

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 14, 2024
@anpingli
Copy link
Copy Markdown
Contributor

anpingli commented Aug 14, 2024

@jianzhangbjz
Is this a proxy issue? Shall we add no_proxy to s3.us-east-1.amazonaws.com?

@jianzhangbjz
Copy link
Copy Markdown
Member Author

Is this a proxy issue? Shall we add no_proxy to s3.us-east-1.amazonaws.com?

Maybe. However, I cannot pull that index image, as above and follows,

[cloud-user@preserve-olm-env2 ~]$ podman --log-level debug pull quay.io/openshift-qe-optional-operators/aosqe-index:v1.25
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull quay.io/openshift-qe-optional-operators/aosqe-index:v1.25) 
DEBU[0000] Using conmon: "/usr/bin/conmon"       
...
Error: initializing source docker://quay.io/openshift-qe-optional-operators/aosqe-index:v1.25: too many requests to registry
DEBU[0000] Shutting down engines
...

@jianzhangbjz
Copy link
Copy Markdown
Member Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot removed the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 14, 2024
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@jianzhangbjz: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-windows-machine-config-operator-master-vsphere-disconnected-e2e-operator openshift/windows-machine-config-operator presubmit Registry content changed
pull-ci-openshift-windows-machine-config-operator-release-4.19-vsphere-disconnected-e2e-operator openshift/windows-machine-config-operator presubmit Registry content changed
pull-ci-openshift-windows-machine-config-operator-release-4.18-vsphere-disconnected-e2e-operator openshift/windows-machine-config-operator presubmit Registry content changed
pull-ci-openshift-windows-machine-config-operator-release-4.17-vsphere-disconnected-e2e-operator openshift/windows-machine-config-operator presubmit Registry content changed
pull-ci-openshift-windows-machine-config-operator-release-4.16-vsphere-disconnected-e2e-operator openshift/windows-machine-config-operator presubmit Registry content changed
periodic-ci-openshift-hypershift-release-4.14-periodics-mce-e2e-agent-disconnected-ovn-ipv6-metal3-conformance N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-gcp-ipi-disc-workers-rhel8-f28-destructive N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-stable-nutanix-ipi-disc-rt-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-4.16-upgrade-from-stable-4.16-ibmcloud-ipi-disc-priv-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-arm64-stable-4.17-upgrade-from-stable-4.16-aws-ipi-disc-priv-sts-ep-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-4.17-upgrade-from-stable-4.17-aws-ipi-disc-priv-sts-ep-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-vsphere-upi-disc-sdn-rhcos-rhel8-f28-destructive N/A periodic Registry content changed
periodic-ci-quay-quay-tests-master-omr-ocp417-unreleased-quay-omr-tests-omr-ocp417-disconnected-unreleased N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-arm64-nightly-4.17-upgrade-from-stable-4.17-gcp-ipi-disc-priv-oidc-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-azure-ipi-disc-f28-destructive N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-arm64-nightly-aws-ipi-disc-priv-sts-basecap-none-f28-destructive N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-amd64-stable-aws-ipi-disc-priv-sts-ep-fips-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-stable-gcp-ipi-disc-priv-oidc-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-stable-4.11-vsphere-upi-disc-secureboot-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-arm64-nightly-4.17-upgrade-from-stable-4.17-aws-ipi-disc-priv-sts-basecap-none-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.13-gcp-ipi-disc-priv-oidc-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-arm64-nightly-4.13-upgrade-from-stable-4.12-aws-ipi-disc-priv-sno-basecap-none-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-aws-c2s-ipi-disc-priv-fips-f28-destructive N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-4.16-upgrade-from-stable-4.15-vsphere-upi-disc-secureboot-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-stable-4.11-azure-ipi-disc-fullyprivate-f28 N/A periodic Registry content changed

A total of 756 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jianzhangbjz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 14, 2024
@jianzhangbjz jianzhangbjz changed the title add retry for skopeo add retry for skopeo and no_proxy for s3.us-east-1.amazonaws.com Aug 14, 2024
@jianzhangbjz
Copy link
Copy Markdown
Member Author

Hi @anpingli , could you help add /lgtm for it? Thanks!

@anpingli
Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 14, 2024
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Aug 14, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anpingli, jianzhangbjz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit 0f3547d into openshift:master Aug 14, 2024
@jianzhangbjz jianzhangbjz deleted the skopeo branch August 15, 2024 01:04
jadhaj pushed a commit to jadhaj/release that referenced this pull request Aug 18, 2024
jadhaj pushed a commit to jadhaj/release that referenced this pull request Aug 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants