PSAP-1210: Update code for automate OCPBUGS-18052 #28382

liqcui · 2023-11-06T13:43:37Z

This test case use for automate for the bug OCPBUGS-18052
https://issues.redhat.com/browse/OCPBUGS-18052

The log of test cases:
make WHAT=cmd/openshift-tests

  Nov 23 12:01:23.354: INFO: Enabling in-tree volume drivers
  Running Suite: OpenShift e2e suite - /opt/origin
  ================================================
  Random Seed: 1700740877 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  [sig-node-tuning] NTO should SNO installation does not finish due to wait for non-existing machine-config [Early] [Suite:openshift/conformance/parallel]
  github.com/openshift/origin/test/extended/node_tuning/node_tuning.go:123
    STEP: Creating a kubernetes client @ 11/23/23 12:01:23.987
  Nov 23 12:01:24.027: INFO: the cluster type is SingleReplica
  Nov 23 12:01:24.102: INFO: the first masterNode is ip-10-0-8-108.us-east-2.compute.internal
  Nov 23 12:01:24.102: INFO: assert the status of mcp master
  Nov 23 12:01:54.256: INFO: the status of mcp master restore to normal state
  Nov 23 12:01:54.256: INFO: assert the status of co machine-config
  Nov 23 12:01:54.385: INFO: desiredCO Name is machine-config
  Nov 23 12:01:54.385: INFO: desiredCO.status.conditions of machine-config is [{Progressing False 2023-11-23 02:22:07 +0000 UTC  Cluster version is 4.14.3} {Degraded False 2023-11-23 02:22:06 +0000 UTC  } {Available True 2023-11-23 02:21:10 +0000 UTC AsExpected Cluster has deployed [{operator 4.14.3}]} {Upgradeable True 2023-11-23 02:32:02 +0000 UTC AsExpected }]
  Nov 23 12:02:24.386: INFO: the status of status of cluster operator machine-config restore to normal state
  Nov 23 12:02:24.386: INFO: assert the status of co node-tuning 
  Nov 23 12:02:24.551: INFO: desiredCO Name is node-tuning
  Nov 23 12:02:24.551: INFO: desiredCO.status.conditions of node-tuning is [{Available True 2023-11-23 02:19:54 +0000 UTC AsExpected Cluster has deployed "4.14.3"} {Progressing False 2023-11-23 02:31:31 +0000 UTC AsExpected Cluster version is "4.14.3"} {Degraded False 2023-11-23 02:19:53 +0000 UTC AsExpected DaemonSet "tuned" available}]
  Nov 23 12:02:54.551: INFO: the status of status of cluster operator node-tuning restore to normal state
  Nov 23 12:02:54.590: INFO: mcpConfigDaemonsetPodName machine-config-daemon-psl5d
  Nov 23 12:02:54.590: INFO: Get pod logs for machine-config-daemon-psl5d
  Nov 23 12:02:54.637: INFO: Check if the log of machine-config-daemon-psl5d contains keyword [Marking Degraded due to|not found]
  Nov 23 12:02:54.637: INFO: the keywords [Marking Degraded due to|not found] of pod isn't found ...
  • [90.656 seconds]
  ------------------------------

  Ran 1 of 1 Specs in 90.656 seconds
  SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped

jmencak

Thank you for the PR. I haven't tested the code, but it seems a bit error-prone. I believe it would be much better to use native clients and query k8s objects directly rather than using the oc command.

test/extended/node_tuning/node_tuning_utils.go

test/extended/node_tuning/node_tuning.go

openshift-trt-bot · 2023-11-22T06:34:41Z

Job Failure Risk Analysis for sha: 674d77b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade	Medium [sig-network-edge] ns/openshift-authentication route/oauth-openshift disruption/ingress-to-oauth-server connection/new should be available throughout the test This test has passed 95.70% of 93 runs on release 4.15 [amd64 aws ha ovn upgrade upgrade-micro] in the last week.

jmencak

Thank you for the changes, Liquan. The code looks better than the previous one, but I'm not sure I understand a few things, the 10 minute wait at the beginning in particular.

test/extended/node_tuning/node_tuning_utils.go

test/extended/node_tuning/node_tuning.go

test/extended/node_tuning/node_tuning_utils.go

test/extended/util/annotate/generated/zz_generated.annotations.go

test/extended/node_tuning/node_tuning.go

test/extended/node_tuning/node_tuning_utils.go

test/extended/node_tuning/node_tuning.go

liqcui · 2023-11-25T12:06:04Z

/test e2e-aws-ovn-single-node-serial

openshift-trt-bot · 2023-11-25T14:17:26Z

Job Failure Risk Analysis for sha: 220ebcb

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	Medium [Serial] [sig-auth][Feature:OAuthServer] [RequestHeaders] [IdP] test RequestHeaders IdP [apigroup:config.openshift.io][apigroup:user.openshift.io][apigroup:apps.openshift.io] [Suite:openshift/conformance/serial] This test has passed 93.33% of 45 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-ovn-single-node-serial'] in the last 14 days. --- [sig-imageregistry][Feature:ImageTriggers][Serial] ImageStream API TestImageStreamWithoutDockerImageConfig [apigroup:image.openshift.io] [Suite:openshift/conformance/serial] This test has passed 95.56% of 45 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-ovn-single-node-serial'] in the last 14 days.

liqcui · 2023-11-26T07:51:31Z

/test e2e-aws-ovn-single-node-serial

jmencak

Thank you for the changes. I didn't review the new logic yet, but left some suggestions for improvement.

test/extended/node_tuning/node_tuning_utils.go

openshift-trt-bot · 2023-11-29T23:14:20Z

Job Failure Risk Analysis for sha: 6e51038

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (654): IncompleteTests

jmencak

The code is getting better. I pointed out a few nits. Also pointed out pottential issues in two functions (early exit). But it can be lack of my understanding of the test case which I still do not fully understand.

test/extended/node_tuning/node_tuning.go

test/extended/node_tuning/node_tuning_utils.go

liqcui · 2023-12-01T08:13:10Z

/retest

jmencak

Thank you for the explanation. Pointed out some remaining nits I found. After that, I believe this PR is probably ready for another pair of eyes who can review/approve it.

test/extended/node_tuning/node_tuning_utils.go

test/extended/node_tuning/node_tuning.go

update code per pr comments

openshift-trt-bot · 2023-12-05T04:12:18Z

Job Failure Risk Analysis for sha: c001861

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (26) are below the historical average (2042): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

liqcui · 2023-12-19T04:54:53Z

/test e2e-gcp-ovn-builds

openshift-trt-bot · 2023-12-19T04:56:57Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1633): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (530): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

liqcui · 2023-12-19T04:57:26Z

@bparees Do you know who I can contact with for failed e2e-gcp-ovn-builds, it failed so many times, actually my new code skipped execute in the job.

seems like it has passed now

/retest

I have executed at least 20 times, all failed. it block the pr merged

openshift-trt-bot · 2023-12-19T06:36:32Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1633): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (530): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2023-12-19T07:11:54Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	IncompleteTests Tests for this run (99) are below the historical average (613): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1620): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (530): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

liqcui · 2023-12-19T07:14:32Z

/test e2e-gcp-ovn-builds

openshift-trt-bot · 2023-12-19T09:11:25Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1620): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (533): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

liqcui · 2023-12-19T11:58:41Z

/test e2e-gcp-ovn-builds

bparees · 2023-12-19T14:23:06Z

I have executed at least 20 times, all failed. it block the pr merged

sorry i was looking at e2e-gcp-ovn.

@coreydaley owns the e2e-gcp-ovn-builds job.

openshift-trt-bot · 2023-12-19T15:11:47Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1617): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (532): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-ci-robot · 2023-12-19T22:38:22Z

/retest-required

Remaining retests: 0 against base HEAD 14c37c2 and 1 for PR HEAD 887428b in total

openshift-trt-bot · 2023-12-20T01:11:51Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1570): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (522): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

liqcui · 2023-12-20T01:22:16Z

/test e2e-gcp-ovn-builds

liqcui · 2023-12-20T01:24:07Z

@coreydaley could you please help us to check the job e2e-gcp-ovn-builds job, it failed many times, actually my new code will skipped im the job, it block the pr get merged, thanks!

openshift-ci-robot · 2023-12-20T03:59:08Z

/retest-required

Remaining retests: 0 against base HEAD 2736628 and 0 for PR HEAD 887428b in total

openshift-trt-bot · 2023-12-20T05:56:47Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds	High
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1570): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (522): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2023-12-20T06:11:36Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1570): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (522): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-ci-robot · 2023-12-20T06:18:39Z

/hold

Revision 887428b was retested 3 times: holding

liqcui · 2023-12-21T05:36:18Z

/test e2e-gcp-ovn-builds

liqcui · 2023-12-21T05:36:48Z

/assign @coreydaley

openshift-trt-bot · 2023-12-21T08:11:11Z

Job Failure Risk Analysis for sha: 887428b

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (800) are below the historical average (1481): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (24) are below the historical average (514): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

liqcui · 2023-12-21T12:22:17Z

/unhold

liqcui · 2023-12-21T12:22:40Z

/test e2e-aws-ovn-single-node-serial

liqcui · 2023-12-21T12:22:58Z

/test e2e-aws-ovn-single-node-upgrade

openshift-ci · 2023-12-21T15:28:04Z

@liqcui: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node-serial	`887428b`	link	false	`/test e2e-aws-ovn-single-node-serial`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2023-12-21T18:27:41Z

[ART PR BUILD NOTIFIER]

This PR has been included in build openshift-enterprise-tests-container-v4.16.0-202312211733.p0.gaeaf534.assembly.stream for distgit openshift-enterprise-tests.
All builds following this will include this PR.

openshift-ci bot requested review from jmencak and shahsahil264 November 6, 2023 13:50

openshift-ci bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 6, 2023

liqcui force-pushed the ntomcp branch 3 times, most recently from 467beeb to c022419 Compare November 16, 2023 14:18

jmencak reviewed Nov 20, 2023

View reviewed changes

test/extended/node_tuning/node_tuning_utils.go Outdated Show resolved Hide resolved

test/extended/node_tuning/node_tuning.go Outdated Show resolved Hide resolved

liqcui changed the title ~~Update code for automate OCPBUGS-18052~~ Update code for automate OCPBUGS-18052[Updating Code] Nov 22, 2023

liqcui force-pushed the ntomcp branch from 674d77b to 8bad0b4 Compare November 23, 2023 12:46

liqcui changed the title ~~Update code for automate OCPBUGS-18052[Updating Code]~~ Update code for automate OCPBUGS-18052 Nov 23, 2023

jmencak reviewed Nov 23, 2023

View reviewed changes

liqcui force-pushed the ntomcp branch 2 times, most recently from 6f85557 to 1e2616f Compare November 24, 2023 15:07

jmencak reviewed Nov 27, 2023

View reviewed changes

liqcui force-pushed the ntomcp branch 3 times, most recently from def0faa to 6e51038 Compare November 28, 2023 08:41

jmencak reviewed Nov 30, 2023

View reviewed changes

liqcui force-pushed the ntomcp branch from 6e51038 to a892382 Compare December 1, 2023 04:13

jmencak reviewed Dec 1, 2023

View reviewed changes

update code to automate nto bugs

c001861

update code per pr comments

liqcui force-pushed the ntomcp branch from 10fd6d3 to c001861 Compare December 1, 2023 11:26

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 20, 2023

openshift-ci bot assigned coreydaley Dec 21, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 21, 2023

openshift-merge-bot bot merged commit aeaf534 into openshift:master Dec 21, 2023
21 of 23 checks passed

PSAP-1210: Update code for automate OCPBUGS-18052 #28382

PSAP-1210: Update code for automate OCPBUGS-18052 #28382

Conversation

liqcui commented Nov 6, 2023 • edited

jmencak left a comment

Choose a reason for hiding this comment

openshift-trt-bot commented Nov 22, 2023

jmencak left a comment

Choose a reason for hiding this comment

liqcui commented Nov 25, 2023

openshift-trt-bot commented Nov 25, 2023

liqcui commented Nov 26, 2023

jmencak left a comment

Choose a reason for hiding this comment

openshift-trt-bot commented Nov 29, 2023

jmencak left a comment

Choose a reason for hiding this comment

liqcui commented Dec 1, 2023

jmencak left a comment • edited

Choose a reason for hiding this comment

openshift-trt-bot commented Dec 5, 2023

liqcui commented Dec 19, 2023

openshift-trt-bot commented Dec 19, 2023

liqcui commented Dec 19, 2023

openshift-trt-bot commented Dec 19, 2023

openshift-trt-bot commented Dec 19, 2023

liqcui commented Dec 19, 2023

openshift-trt-bot commented Dec 19, 2023

liqcui commented Dec 19, 2023

bparees commented Dec 19, 2023

openshift-trt-bot commented Dec 19, 2023

openshift-ci-robot commented Dec 19, 2023

openshift-trt-bot commented Dec 20, 2023

liqcui commented Dec 20, 2023

liqcui commented Dec 20, 2023

openshift-ci-robot commented Dec 20, 2023

openshift-trt-bot commented Dec 20, 2023

openshift-trt-bot commented Dec 20, 2023

openshift-ci-robot commented Dec 20, 2023

liqcui commented Dec 21, 2023

liqcui commented Dec 21, 2023

openshift-trt-bot commented Dec 21, 2023

liqcui commented Dec 21, 2023

liqcui commented Dec 21, 2023

liqcui commented Dec 21, 2023

openshift-ci bot commented Dec 21, 2023

openshift-bot commented Dec 21, 2023

liqcui commented Nov 6, 2023 •

edited

jmencak left a comment •

edited