Improved the quality of output when the test fails #105860

NikhilSharmaWe · 2021-10-24T10:06:11Z

What type of PR is this?

Enhancement

What this PR does / why we need it:

Improved the quality of output when the test fails

Which issue(s) this PR fixes:

Part of #105678

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2021-10-24T10:06:12Z

@NikhilSharmaWe: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-10-24T10:06:19Z

@NikhilSharmaWe: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-10-24T10:06:19Z

Hi @NikhilSharmaWe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-10-24T10:06:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: NikhilSharmaWe
To complete the pull request process, please assign xing-yang after the PR has been reviewed.
You can assign the PR to them by writing /assign @xing-yang in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

test/e2e/storage/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pohly · 2021-10-25T08:29:32Z

Can you combine more of these changes into a single PR? A good granularity is to go by OWNERS file.

In this case, everything under test/e2e/storage can go into a single PR.

NikhilSharmaWe · 2021-10-25T08:51:56Z

Can you combine more of these changes into a single PR? A good granularity is to go by OWNERS file.

In this case, everything under test/e2e/storage can go into a single PR.

Should I combine this and #105774 PR together.

pohly · 2021-10-25T09:36:37Z

Yes, please do.

NikhilSharmaWe · 2021-10-25T09:38:55Z

Yes, please do.

Done #105875

pohly · 2021-10-25T09:39:00Z

There are also other places that should be fixed:

$ git grep 'Expect.*Equal([^,]*, *\(true\|false\)' -- test/e2e/storage/ | wc -l
63

pohly · 2021-10-25T09:40:38Z

Sorry, I missed that the other PR is outside of test/e2e/storage. Let's use this PR here for that directory and #105774 for fixes under test/e2e/upgrades/apps.

NikhilSharmaWe · 2021-10-25T09:47:31Z

Sorry, I missed that the other PR is outside of test/e2e/storage. Let's use this PR here for that directory and #105774 for fixes under test/e2e/upgrades/apps.

The above command tells that how many changes needed, can you tell me that how can I find in which files the changes needed or it has to be done manually.

pohly · 2021-10-25T09:51:33Z

$ git grep -n 'Expect.*Equal([^,]*, *\(true\|false\)' -- test/e2e/storage/
test/e2e/storage/csi_mock_volume.go:686:                                framework.ExpectEqual(*sc.AllowVolumeExpansion, true, "failed creating sc with allowed expansion")
test/e2e/storage/csi_mock_volume.go:779:                                framework.ExpectEqual(*sc.AllowVolumeExpansion, true, "failed creating sc with allowed expansion")
test/e2e/storage/drivers/in_tree.go:128:        framework.ExpectEqual(ok, true, "Failed to cast test volume to NFS test volume")
test/e2e/storage/drivers/in_tree.go:140:        framework.ExpectEqual(ok, true, "Failed to cast test volume to NFS test volume")
test/e2e/storage/drivers/in_tree.go:268:        framework.ExpectEqual(ok, true, "Failed to cast test volume to Gluster test volume")
test/e2e/storage/drivers/in_tree.go:283:        framework.ExpectEqual(ok, true, "Failed to cast test volume to Gluster test volume")
test/e2e/storage/drivers/in_tree.go:403:        framework.ExpectEqual(ok, true, "Failed to cast test volume to iSCSI test volume")
test/e2e/storage/drivers/in_tree.go:421:        framework.ExpectEqual(ok, true, "Failed to cast test volume to iSCSI test volume")
test/e2e/storage/drivers/in_tree.go:580:        framework.ExpectEqual(ok, true, "Failed to cast test volume to RBD test volume")
test/e2e/storage/drivers/in_tree.go:602:        framework.ExpectEqual(ok, true, "Failed to cast test volume to RBD test volume")
test/e2e/storage/drivers/in_tree.go:703:        framework.ExpectEqual(ok, true, "Failed to cast test volume to Ceph test volume")
test/e2e/storage/drivers/in_tree.go:719:        framework.ExpectEqual(ok, true, "Failed to cast test volume to Ceph test volume")
test/e2e/storage/drivers/in_tree.go:877:        framework.ExpectEqual(ok, true, "Failed to cast test volume to Hostpath Symlink test volume")
test/e2e/storage/drivers/in_tree.go:1097:       framework.ExpectEqual(ok, true, "Failed to cast test volume to Cinder test volume")
test/e2e/storage/drivers/in_tree.go:1113:       framework.ExpectEqual(ok, true, "Failed to cast test volume to Cinder test volume")
test/e2e/storage/drivers/in_tree.go:1310:       framework.ExpectEqual(ok, true, "Failed to cast test volume to GCE PD test volume")
test/e2e/storage/drivers/in_tree.go:1325:       framework.ExpectEqual(ok, true, "Failed to cast test volume to GCE PD test volume")
test/e2e/storage/drivers/in_tree.go:1443:       framework.ExpectEqual(ok, true, "Failed to cast test volume to vSphere test volume")
test/e2e/storage/drivers/in_tree.go:1463:       framework.ExpectEqual(ok, true, "Failed to cast test volume to vSphere test volume")
test/e2e/storage/drivers/in_tree.go:1572:       framework.ExpectEqual(ok, true, "Failed to cast test volume to Azure test volume")
test/e2e/storage/drivers/in_tree.go:1592:       framework.ExpectEqual(ok, true, "Failed to cast test volume to Azure test volume")
test/e2e/storage/drivers/in_tree.go:1716:       framework.ExpectEqual(ok, true, "Failed to cast test volume to AWS test volume")
test/e2e/storage/drivers/in_tree.go:1731:       framework.ExpectEqual(ok, true, "Failed to cast test volume to AWS test volume")
test/e2e/storage/drivers/in_tree.go:1972:       framework.ExpectEqual(ok, true, "Failed to cast test volume to local test volume")
test/e2e/storage/external/external.go:289:              framework.ExpectEqual(ok, true, "storage class from %s", d.StorageClass.FromFile)
test/e2e/storage/flexvolume_mounted_volume_resize.go:100:               framework.ExpectEqual(*resizableSc.AllowVolumeExpansion, true)
test/e2e/storage/flexvolume_online_resize.go:93:                framework.ExpectEqual(*resizableSc.AllowVolumeExpansion, true)
test/e2e/storage/mounted_volume_resize.go:91:           framework.ExpectEqual(*sc.AllowVolumeExpansion, true)
test/e2e/storage/persistent_volumes-local.go:836:               framework.ExpectEqual(ok, true)
test/e2e/storage/regional_pd.go:230:            framework.ExpectEqual(podutil.IsPodReadyConditionTrue(pod.Status), true, "The statefulset pod has the following conditions: %s", pod.Status.Conditions)
test/e2e/storage/regional_pd.go:280:            framework.ExpectEqual(podutil.IsPodReadyConditionTrue(pod.Status), true, "The statefulset pod has the following conditions: %s", pod.Status.Conditions)
test/e2e/storage/testsuites/provisioning.go:491:                framework.ExpectEqual(found, true)
test/e2e/storage/testsuites/volumeperf.go:220:          framework.ExpectEqual(ok, true)
test/e2e/storage/testsuites/volumeperf.go:272:                  framework.ExpectEqual(ok, true, "PVC %s should exist in interval map already", newPVC.Name)
test/e2e/storage/testsuites/volumeperf.go:300:                          framework.ExpectEqual(ok, true)
test/e2e/storage/testsuites/volumeperf.go:302:                          framework.ExpectEqual(ok, true)
test/e2e/storage/utils/pod.go:152:              framework.ExpectEqual(isPidChanged, true, "Kubelet PID remained unchanged after restarting Kubelet")
test/e2e/storage/volume_metrics.go:271:                 framework.ExpectEqual(found, true, "PVC %s, Namespace %s not found for %s", pvcName, pvcNamespace, kubeletKeyName)
test/e2e/storage/volume_metrics.go:335:                 framework.ExpectEqual(found, true, "PVC %s, Namespace %s not found for %s", pvcBlock.Name, pvcBlock.Namespace, kubeletKeyName)
test/e2e/storage/volume_metrics.go:435:         framework.ExpectEqual(ok, true, "Key %q not found in A/D Controller metrics", forceDetachKey)
test/e2e/storage/volume_metrics.go:720:         framework.ExpectEqual(ok, true, "Error getting updated latency metrics for %s", metricName)
test/e2e/storage/volume_metrics.go:723: framework.ExpectEqual(ok, true, "Error getting updated status metrics for %s", metricName)
test/e2e/storage/vsphere/persistent_volumes-vsphere.go:109:             framework.ExpectEqual(isAttached, true, "disk is not attached with the node")
test/e2e/storage/vsphere/pv_reclaimpolicy.go:132:                       framework.ExpectEqual(isVolumeAttached, true)
test/e2e/storage/vsphere/vsphere_scale.go:87:           framework.ExpectNotEqual(numberOfInstances > 5, true, "Maximum allowed instances are 5")
test/e2e/storage/vsphere/vsphere_scale.go:88:           framework.ExpectNotEqual(numberOfInstances > volumeCount, true, "Number of instances should be less than the total volume count")
test/e2e/storage/vsphere/vsphere_statefulsets.go:114:                           framework.ExpectEqual(apierrors.IsNotFound(err), true)
test/e2e/storage/vsphere/vsphere_statefulsets.go:147:                                   framework.ExpectEqual(volumesBeforeScaleDown[vSpherediskPath] == "", false)
test/e2e/storage/vsphere/vsphere_statefulsets.go:149:                                   framework.ExpectEqual(isVolumeAttached, true)
test/e2e/storage/vsphere/vsphere_stress.go:73:          framework.ExpectEqual(instances <= volumesPerNode*len(nodeList.Items), true, fmt.Sprintf("Number of Instances should be less or equal: %v", volumesPerNode*len(nodeList.Items)))
test/e2e/storage/vsphere/vsphere_stress.go:74:          framework.ExpectEqual(instances > len(scNames), true, "VCP_STRESS_INSTANCES should be greater than 3 to utilize all 4 types of storage classes")
test/e2e/storage/vsphere/vsphere_stress.go:77:          framework.ExpectEqual(iterations > 0, true, "VCP_STRESS_ITERATIONS should be greater than 0")
test/e2e/storage/vsphere/vsphere_stress.go:161:         framework.ExpectEqual(isVolumeAttached, true)
test/e2e/storage/vsphere/vsphere_utils.go:381:          framework.ExpectEqual(isAttached, true, fmt.Sprintf("disk %v is not attached with the node", pv.Spec.VsphereVolume.VolumePath))
test/e2e/storage/vsphere/vsphere_utils.go:791:  framework.ExpectEqual(isAttached, true, fmt.Sprintf("disk: %s is not attached with the node", volumePath))
test/e2e/storage/vsphere/vsphere_volume_diskformat.go:153:      framework.ExpectEqual(isAttached, true)
test/e2e/storage/vsphere/vsphere_volume_diskformat.go:199:      framework.ExpectEqual(diskFound, true, "Failed to find disk")
test/e2e/storage/vsphere/vsphere_volume_fstype.go:150:  framework.ExpectEqual(isFound, true, "Unable to verify MountVolume.MountDevice failure")
test/e2e/storage/vsphere/vsphere_volume_node_delete.go:56:              framework.ExpectEqual(len(nodeList.Items) > 1, true, "At least 2 nodes are required for this test")
test/e2e/storage/vsphere/vsphere_volume_node_poweroff.go:62:            framework.ExpectEqual(len(nodeList.Items) > 1, true, "At least 2 nodes are required for this test")
test/e2e/storage/vsphere/vsphere_volume_node_poweroff.go:114:           framework.ExpectEqual(isAttached, true, "Disk is not attached to the node")
test/e2e/storage/vsphere/vsphere_volume_placement.go:375:               framework.ExpectEqual(isAttached, true, "disk:"+volumePath+" is not attached with the node")
test/e2e/storage/vsphere/vsphere_volume_vsan_policy.go:342:     framework.ExpectEqual(isVMPresentFlag, false, errorMsg)

chetak123 · 2021-10-25T12:47:51Z

hey @NikhilSharmaWe Actually I was making the req. changes locally and it would be really helpful if you can please specify which all files you are changing so we can collaborate our work

pohly · 2021-10-25T14:02:11Z

@NikhilSharmaWe: @chetak123 makes a good point here. To avoid duplication of efforts we assign issues to people and then no-one else should work on the same thing.

I hadn't checked who the assignee is for #105678, sorry.

@chetak123: do you want to take the lead on splitting up that issue by OWNERS? I don't think we need sub-issues for that. Just having a comment with a proposed split should be enough. Then others can comment to sign up and you can edit your comment to keep track of who's doing what.

chetak123 · 2021-10-26T01:44:28Z

That would be good

pohly · 2021-10-29T19:13:03Z

/ok-to-test

k8s-ci-robot · 2021-10-29T20:03:53Z

@NikhilSharmaWe: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-verify-govet-levee	`3297f83`	link	true	`/test pull-kubernetes-verify-govet-levee`
pull-kubernetes-e2e-gce-storage-snapshot	`3297f83`	link	false	`/test pull-kubernetes-e2e-gce-storage-snapshot`
pull-kubernetes-e2e-gce-csi-serial	`3297f83`	link	false	`/test pull-kubernetes-e2e-gce-csi-serial`
pull-kubernetes-e2e-gce-storage-slow	`3297f83`	link	false	`/test pull-kubernetes-e2e-gce-storage-slow`
pull-kubernetes-e2e-kind	`3297f83`	link	true	`/test pull-kubernetes-e2e-kind`
pull-kubernetes-e2e-gce-100-performance	`3297f83`	link	true	`/test pull-kubernetes-e2e-gce-100-performance`
pull-kubernetes-typecheck	`3297f83`	link	true	`/test pull-kubernetes-typecheck`
pull-kubernetes-conformance-kind-ipv6-parallel	`3297f83`	link	false	`/test pull-kubernetes-conformance-kind-ipv6-parallel`
pull-kubernetes-e2e-kind-ipv6	`3297f83`	link	true	`/test pull-kubernetes-e2e-kind-ipv6`
pull-kubernetes-e2e-gce-ubuntu-containerd	`3297f83`	link	true	`/test pull-kubernetes-e2e-gce-ubuntu-containerd`
pull-kubernetes-conformance-kind-ga-only-parallel	`3297f83`	link	true	`/test pull-kubernetes-conformance-kind-ga-only-parallel`
pull-kubernetes-verify	`3297f83`	link	true	`/test pull-kubernetes-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

pohly · 2021-11-19T12:44:17Z

@NikhilSharmaWe: there are syntax errors in this PR.

dims · 2022-01-04T23:27:46Z

@NikhilSharmaWe will you be revisiting this PR? or can we close it?

NikhilSharmaWe · 2022-01-06T14:21:32Z

@dims Apologies for the inconvenience , I will keep better track of my PR's in future.

Improved the quality of output when the test fails

bc24d35

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 24, 2021

k8s-ci-robot requested review from gnufied and pohly October 24, 2021 10:06

k8s-ci-robot added area/test sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 24, 2021

Update vsphere_volume_diskformat.go

3297f83

chetak123 mentioned this pull request Oct 26, 2021

replace ExpectEqual(someBoolean, ...) #105678

Open

22 tasks

NikhilSharmaWe mentioned this pull request Oct 27, 2021

updated the code for developers to get better and more informative ou… #105875

Closed

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 29, 2021

NikhilSharmaWe closed this Jan 6, 2022

pohly mentioned this pull request May 31, 2022

enhance assertions in test/e2e/apps #110293

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved the quality of output when the test fails #105860

Improved the quality of output when the test fails #105860

NikhilSharmaWe commented Oct 24, 2021 •

edited

k8s-ci-robot commented Oct 24, 2021

k8s-ci-robot commented Oct 24, 2021

k8s-ci-robot commented Oct 24, 2021

k8s-ci-robot commented Oct 24, 2021

pohly commented Oct 25, 2021

NikhilSharmaWe commented Oct 25, 2021

pohly commented Oct 25, 2021

NikhilSharmaWe commented Oct 25, 2021

pohly commented Oct 25, 2021

pohly commented Oct 25, 2021

NikhilSharmaWe commented Oct 25, 2021

pohly commented Oct 25, 2021

chetak123 commented Oct 25, 2021

pohly commented Oct 25, 2021

chetak123 commented Oct 26, 2021

pohly commented Oct 29, 2021

k8s-ci-robot commented Oct 29, 2021

pohly commented Nov 19, 2021

dims commented Jan 4, 2022

NikhilSharmaWe commented Jan 6, 2022

Improved the quality of output when the test fails #105860

Improved the quality of output when the test fails #105860

Conversation

NikhilSharmaWe commented Oct 24, 2021 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Oct 24, 2021

k8s-ci-robot commented Oct 24, 2021

k8s-ci-robot commented Oct 24, 2021

k8s-ci-robot commented Oct 24, 2021

pohly commented Oct 25, 2021

NikhilSharmaWe commented Oct 25, 2021

pohly commented Oct 25, 2021

NikhilSharmaWe commented Oct 25, 2021

pohly commented Oct 25, 2021

pohly commented Oct 25, 2021

NikhilSharmaWe commented Oct 25, 2021

pohly commented Oct 25, 2021

chetak123 commented Oct 25, 2021

pohly commented Oct 25, 2021

chetak123 commented Oct 26, 2021

pohly commented Oct 29, 2021

k8s-ci-robot commented Oct 29, 2021

pohly commented Nov 19, 2021

dims commented Jan 4, 2022

NikhilSharmaWe commented Jan 6, 2022

NikhilSharmaWe commented Oct 24, 2021 •

edited