testing: prow/gpu-operator.sh: use the GPU Operator must-gather image #294

kpouget · 2021-12-01T15:06:37Z

build/Dockerfile: unprepare the image for GPU Operator must-gather support

The GPU Operator image now provides its own must-gather script.

https://gitlab.com/nvidia/kubernetes/gpu-operator/-/blob/master/hack/must-gather.sh

testing: prow/gpu-operator.sh: use the GPU Operator must-gather image

... or our script toolbox/gpu-operator/must-gather.sh if it doesn't
exist. Support integrated in v1.9.0.

…pport The GPU Operator image now provides its own must-gather script. https://gitlab.com/nvidia/kubernetes/gpu-operator/-/blob/master/hack/must-gather.sh

kpouget · 2021-12-01T15:06:47Z

/test gpu-operator-e2e

kpouget · 2021-12-01T16:53:45Z

*[2021-12-01T15:11:17Z]NoClusters: No clusters in pool are ready to be claimed

/test gpu-operator-e2e

…

On Wed, Dec 1, 2021 at 5:41 PM openshift-ci[bot] ***@***.***> wrote: @kpouget <https://github.com/kpouget>: The following test *failed*, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests: Test name Commit Details Required Rerun command ci/prow/gpu-operator-e2e 9228bf2 <9228bf2> link <https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/294/pull-ci-openshift-psap-ci-artifacts-master-gpu-operator-e2e/1466061224595689472> true /test gpu-operator-e2e Full PR test history <https://prow.ci.openshift.org/pr-history?org=openshift-psap&repo=ci-artifacts&pr=294>. Your PR dashboard <https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:kpouget>. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/guide/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. I understand the commands that are listed here <https://go.k8s.io/bot-commands>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZVQIX5HB6PYN5SILIPVEDUOZF4DANCNFSM5JEWXXGA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

kpouget · 2021-12-01T20:00:44Z

/test gpu-operator-e2e

kpouget · 2021-12-01T21:06:39Z

/test gpu-operator-e2e

kpouget · 2021-12-02T08:27:02Z

/test gpu-operator-e2e

kpouget · 2021-12-02T12:05:39Z

/test gpu-operator-e2e

kpouget · 2021-12-02T15:55:34Z

/test gpu-operator-e2e
/test test-commit

kpouget · 2021-12-02T18:26:52Z

/test gpu-operator-e2e
/test test-commit

kpouget · 2021-12-02T20:51:59Z

new behavior:

calls gpu-operator capture_deployment_state is must-gather not available in the image (eg, 1.8.2)
calls must-gather in the image is available
calls gpu-operator capture_deployment_state after, until we're sure we're happy with the must-gather image.

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/294/pull-ci-openshift-psap-ci-artifacts-master-gpu-operator-e2e/1466473963252092928/artifacts/gpu-operator-e2e/presubmit-operatorhub/artifacts/
--> see must-gather script being broken in v1.9.0-beta

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/294/pull-ci-openshift-psap-ci-artifacts-master-gpu-operator-e2e/1466473963252092928/artifacts/gpu-operator-e2e/presubmit-master/artifacts/

--> see must-gather script being fixed in master

kpouget · 2021-12-02T20:52:12Z

/approve

kpouget · 2021-12-07T10:06:05Z

@omertuc PTAL

testing/prow/gpu-operator.sh

kpouget · 2021-12-08T12:26:49Z

thanks for the feedback @omertuc , I followed the suggestions

/test gpu-operator-e2e
/test test-commit

omertuc · 2021-12-08T12:39:55Z

/lgtm
/hold wait for job

kpouget · 2021-12-08T12:42:58Z

did you see the force push when you LGTM? 🙃 seems to be that both happened at the same time :)

anyway:

/test gpu-operator-e2e
/test test-commit

openshift-ci · 2021-12-08T13:23:03Z

@kpouget: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-commit	`d40638d`	link	true	`/test test-commit`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kpouget · 2021-12-08T13:28:50Z

@kpouget: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/test-commit d40638d link true /test test-commit

Full PR test history. Your PR dashboard.

our test passed:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/294/pull-ci-openshift-psap-ci-artifacts-master-test-commit/1468561748058443776/artifacts/test-commit/test-commit/artifacts/prow_gpu-operator_test_operatorhub_1.8.2_v1.8/

the failure occured in the must-gather step:

   * could not run steps: step test-commit failed: "test-commit" post steps failed: "test-commit" pod "test-commit-gather-extra" failed: the pod ci-op-m549vkn3/test-commit-gather-extra failed after 59s (failed containers: test): ContainerFailed one or more containers exited

... or our script `toolbox/gpu-operator/must-gather.sh` if it doesn't exist. Support integrated in v1.9.0. test-path: prow gpu-operator test_operatorhub 1.8.2 v1.8

kpouget · 2021-12-08T14:19:38Z

/test gpu-operator-e2e

omertuc · 2021-12-08T14:27:29Z

/lgtm

kpouget · 2021-12-08T14:28:13Z

/test gpu-operator-e2e

…

On Wed, Dec 8, 2021 at 3:25 PM openshift-ci[bot] ***@***.***> wrote: New changes are detected. LGTM label has been removed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZVQIW6JVSIXFJYRKO5ARDUP5TEZANCNFSM5JEWXXGA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

kpouget · 2021-12-08T17:14:45Z

must-gather looks good!
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/294/pull-ci-openshift-psap-ci-artifacts-master-gpu-operator-e2e/1468588232127025152/artifacts/gpu-operator-e2e/presubmit-master/artifacts/011__gpu-operator__must-gather/
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/294/pull-ci-openshift-psap-ci-artifacts-master-gpu-operator-e2e/1468588232127025152/artifacts/gpu-operator-e2e/presubmit-operatorhub/artifacts/012__gpu-operator__must-gather/

I'll remote the rmdir in another PR

Running exit finalizers ...
Running finalizer 'collect_must_gather' ...
Running the GPU Operator must-gather image ...
Operator image: nvcr.io/nvidia/gpu-operator@sha256:57e7be259af342ccdf281d23d79e15d474e2e6f506f84c0cdae3db2a199b3395
Copying must-gather results to /logs/artifacts/012__gpu-operator__must-gather ...
rmdir: failed to remove '/tmp/gpu-operator_Rlqi': Directory not empty
Running gpu_operator capture_deployment_state ...
Running gpu_operator capture_deployment_state ... done.

/hold cancel
/approve

openshift-ci · 2021-12-08T17:14:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kpouget

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kpouget]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

build/Dockerfile: unprepare the image for GPU Operator must-gather su…

51cb094

…pport The GPU Operator image now provides its own must-gather script. https://gitlab.com/nvidia/kubernetes/gpu-operator/-/blob/master/hack/must-gather.sh

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 1, 2021

openshift-ci bot requested review from dagrayvid and jmencak December 1, 2021 15:06

kpouget force-pushed the must-gather branch from 9228bf2 to 2dd8f09 Compare December 1, 2021 21:06

kpouget force-pushed the must-gather branch from 2dd8f09 to e87e6c9 Compare December 2, 2021 08:26

kpouget force-pushed the must-gather branch from 5eba215 to 6a0b84a Compare December 2, 2021 15:54

kpouget force-pushed the must-gather branch from 6a0b84a to 8310793 Compare December 2, 2021 18:26

kpouget changed the title ~~WIP: testing: prow/gpu-operator.sh: use the GPU Operator must-gather image~~ testing: prow/gpu-operator.sh: use the GPU Operator must-gather image Dec 2, 2021

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 2, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 2, 2021

omertuc reviewed Dec 8, 2021

View reviewed changes

testing/prow/gpu-operator.sh Outdated Show resolved Hide resolved

omertuc reviewed Dec 8, 2021

View reviewed changes

testing/prow/gpu-operator.sh Outdated Show resolved Hide resolved

omertuc reviewed Dec 8, 2021

View reviewed changes

testing/prow/gpu-operator.sh Outdated Show resolved Hide resolved

kpouget force-pushed the must-gather branch from 8310793 to 1b4af4a Compare December 8, 2021 12:26

kpouget force-pushed the must-gather branch from 1b4af4a to d40638d Compare December 8, 2021 12:39

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 8, 2021

openshift-ci bot assigned omertuc Dec 8, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2021

testing: prow/gpu-operator.sh: use the GPU Operator must-gather image

6f0b8ef

... or our script `toolbox/gpu-operator/must-gather.sh` if it doesn't exist. Support integrated in v1.9.0. test-path: prow gpu-operator test_operatorhub 1.8.2 v1.8

kpouget force-pushed the must-gather branch from d40638d to 6f0b8ef Compare December 8, 2021 14:24

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2021

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 8, 2021

openshift-merge-robot merged commit b88b1e3 into openshift-psap:master Dec 8, 2021

kpouget deleted the must-gather branch December 8, 2021 17:16

kpouget mentioned this pull request Dec 14, 2021

ci-artifacts maintenance overview #291

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testing: prow/gpu-operator.sh: use the GPU Operator must-gather image #294

testing: prow/gpu-operator.sh: use the GPU Operator must-gather image #294

kpouget commented Dec 1, 2021

kpouget commented Dec 1, 2021

kpouget commented Dec 1, 2021 via email •

edited

Loading

kpouget commented Dec 1, 2021

kpouget commented Dec 1, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 7, 2021

kpouget commented Dec 8, 2021

omertuc commented Dec 8, 2021

kpouget commented Dec 8, 2021

openshift-ci bot commented Dec 8, 2021

kpouget commented Dec 8, 2021

kpouget commented Dec 8, 2021

omertuc commented Dec 8, 2021

kpouget commented Dec 8, 2021 via email

kpouget commented Dec 8, 2021

openshift-ci bot commented Dec 8, 2021

testing: prow/gpu-operator.sh: use the GPU Operator must-gather image #294

testing: prow/gpu-operator.sh: use the GPU Operator must-gather image #294

Conversation

kpouget commented Dec 1, 2021

kpouget commented Dec 1, 2021

kpouget commented Dec 1, 2021 via email • edited Loading

kpouget commented Dec 1, 2021

kpouget commented Dec 1, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 2, 2021

kpouget commented Dec 7, 2021

kpouget commented Dec 8, 2021

omertuc commented Dec 8, 2021

kpouget commented Dec 8, 2021

openshift-ci bot commented Dec 8, 2021

kpouget commented Dec 8, 2021

kpouget commented Dec 8, 2021

omertuc commented Dec 8, 2021

kpouget commented Dec 8, 2021 via email

kpouget commented Dec 8, 2021

openshift-ci bot commented Dec 8, 2021

kpouget commented Dec 1, 2021 via email •

edited

Loading