fix: wait for eventual deletion of installplan e2e test objects #1648

exdx · 2020-07-17T19:21:54Z

Description of the change:
Improves the performance of the install plan e2e test suite by:

Waiting for full deletion of objects via Eventually() before proceeding. These e2e tests keep deleting/recreating a subscription but with a new channel each time. On OpenShift the subscription reference would be to the older subscription object (which was not fully deleted), which would cause the install plan to not be found and the test would flake.
Add a server side apply to an install plan e2e test

Motivation for the change:
Reduce flakiness of these installplan e2e tests on OpenShift

Install Plan with deprecated version CRD Test upgrade CRD with deprecated version expand_less | 1m5s
-- | --
/go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:43  	
Error Trace:	installplan_e2e_test.go:1086 	            				
...
Error:      	
Received unexpected error: 	            	
installplans.operators.coreos.com "install-ndqtt" not found  /go/src/github.com/operator-framework/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

 • Failure [24.353 seconds]
Install Plan
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:39
  creation with pre existing CRD owners
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:211
    PreExistingCRDOwnerIsReplaced [It]
    /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:336
    	Error:      	
Received unexpected error:
    	            	installplans.operators.coreos.com "install-ggdbc" not found

Reviewer Checklist

Implementation matches the proposed design, or proposal is updated to match implementation
Sufficient unit test coverage
Sufficient end-to-end test coverage
Docs updated or added to /docs
Commit messages sensible and descriptive

openshift-ci-robot · 2020-07-17T19:21:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: exdx
To complete the pull request process, please assign ecordell
You can assign the PR to them by writing /assign @ecordell in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Bowenislandsong · 2020-07-20T16:13:42Z

This PR failed tests for 1 times with 1 individual failed tests and 4 skipped tests. A test is considered flaky if failed on multiple commits.

totaltestcount: 1
flaketestcount: 1
skippedtestcount: 4
flaketests:

classname: End-to-end
name: 'Install Plan with CRD schema change Test existing version is present
in new CRD (deprecated field)'
counts: 1
details:

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:43

	Error Trace:	installplan_e2e_test.go:867
	            				value.go:460
	            				value.go:321
	            				table_entry.go:37
	            				runner.go:113
	            				runner.go:64
	            				it_node.go:26
	            				spec.go:215
	            				spec.go:138
	            				spec_runner.go:200
	            				spec_runner.go:170
	            				spec_runner.go:66
	            				suite.go:62
	            				ginkgo_dsl.go:226
	            				ginkgo_dsl.go:214
	            				e2e_test.go:54
	Error:      	Received unexpected error:
	            	clusterserviceversions.operators.coreos.com "nginx-l8jn6-beta" not found

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

meandurationsec: 28.313588
skippedtests:

classname: End-to-end
name: 'Catalog image update'
counts: 1
details: []
meandurationsec: 0.022407
classname: End-to-end
name: 'Subscription updates existing install plan'
counts: 1
details: []
meandurationsec: 0.275107
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should install the operator
successfully'
counts: 1
details: []
meandurationsec: 2.187222
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should have created the expected
prometheus objects'
counts: 1
details: []
meandurationsec: 2.122605

exdx · 2020-07-20T18:13:54Z

/retest

exdx · 2020-07-20T18:21:47Z

/retest

Bowenislandsong · 2020-07-20T18:49:27Z

This PR failed tests for 1 times with 2 individual failed tests and 4 skipped tests. A test is considered flaky if failed on multiple commits.

totaltestcount: 1
flaketestcount: 2
skippedtestcount: 4
flaketests:

classname: End-to-end
name: 'Operator Group cleanup csvs with bad owner operator groups'
counts: 1
details:

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/operator_groups_e2e_test.go:1800

	Error Trace:	operator_groups_e2e_test.go:2040
	            				runner.go:113
	            				runner.go:64
	            				it_node.go:26
	            				spec.go:215
	            				spec.go:138
	            				spec_runner.go:200
	            				spec_runner.go:170
	            				spec_runner.go:66
	            				suite.go:62
	            				ginkgo_dsl.go:226
	            				ginkgo_dsl.go:214
	            				e2e_test.go:54
	Error:      	Received unexpected error:
	            	timed out waiting for the condition

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

meandurationsec: 612.260778

classname: End-to-end
name: 'Catalog delete gRPC registry pod triggers recreation'
counts: 1
details:

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:614

	Error Trace:	catalog_e2e_test.go:679
	            				runner.go:113
	            				runner.go:64
	            				it_node.go:26
	            				spec.go:215
	            				spec.go:138
	            				spec_runner.go:200
	            				spec_runner.go:170
	            				spec_runner.go:66
	            				suite.go:62
	            				ginkgo_dsl.go:226
	            				ginkgo_dsl.go:214
	            				e2e_test.go:54
	Error:      	Received unexpected error:
	            	pods "catalog-ktxs2-6c2hs" not found

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

meandurationsec: 40.911598
skippedtests:

classname: End-to-end
name: 'Catalog image update'
counts: 1
details: []
meandurationsec: 0.01556
classname: End-to-end
name: 'Subscription updates existing install plan'
counts: 1
details: []
meandurationsec: 0.021062
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should have created the expected
prometheus objects'
counts: 1
details: []
meandurationsec: 2.173725
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should install the operator
successfully'
counts: 1
details: []
meandurationsec: 2.056814

…e proceeding.

Bowenislandsong · 2020-07-20T19:35:17Z

This PR failed tests for 1 times with 1 individual failed tests and 4 skipped tests. A test is considered flaky if failed on multiple commits.

totaltestcount: 1
flaketestcount: 1
skippedtestcount: 4
flaketests:

classname: End-to-end
name: 'Install Plan creation with pre existing CRD owners PreExistingCRDOwnerIsReplaced'
counts: 1
details:

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:336

	Error Trace:	installplan_e2e_test.go:476
	            				runner.go:113
	            				runner.go:64
	            				it_node.go:26
	            				spec.go:215
	            				spec.go:138
	            				spec_runner.go:200
	            				spec_runner.go:170
	            				spec_runner.go:66
	            				suite.go:62
	            				ginkgo_dsl.go:226
	            				ginkgo_dsl.go:214
	            				e2e_test.go:54
	Error:      	Received unexpected error:
	            	installplans.operators.coreos.com "install-mvq88" not found

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

meandurationsec: 37.162097
skippedtests:

classname: End-to-end
name: 'Subscription updates existing install plan'
counts: 1
details: []
meandurationsec: 1.5692
classname: End-to-end
name: 'Catalog image update'
counts: 1
details: []
meandurationsec: 0.01466
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should install the operator
successfully'
counts: 1
details: []
meandurationsec: 1.489317
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should have created the expected
prometheus objects'
counts: 1
details: []
meandurationsec: 3.549195

openshift-ci-robot · 2020-07-20T20:35:58Z

@exdx: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-olm	`dc9963d`	link	`/test e2e-aws-olm`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

exdx · 2020-07-21T13:33:39Z

Closing in favor of #1659

exdx requested review from benluddy, dinhxuanvu and njhale July 17, 2020 19:21

openshift-ci-robot requested review from gallettilance and kevinrizza July 17, 2020 19:21

exdx force-pushed the fix/installplan-crd-e2e branch from 564efa6 to c7912ff Compare July 20, 2020 15:28

exdx force-pushed the fix/installplan-crd-e2e branch from c7912ff to b925ab7 Compare July 20, 2020 17:33

fix: wait for eventual deletion of installplan e2e test objects befor…

dc9963d

…e proceeding.

exdx force-pushed the fix/installplan-crd-e2e branch from b925ab7 to dc9963d Compare July 20, 2020 19:00

exdx closed this Jul 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: wait for eventual deletion of installplan e2e test objects #1648

fix: wait for eventual deletion of installplan e2e test objects #1648

Uh oh!

exdx commented Jul 17, 2020 •

edited

Loading

Uh oh!

openshift-ci-robot commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

exdx commented Jul 20, 2020

Uh oh!

exdx commented Jul 20, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

openshift-ci-robot commented Jul 20, 2020

Uh oh!

exdx commented Jul 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fix: wait for eventual deletion of installplan e2e test objects #1648

fix: wait for eventual deletion of installplan e2e test objects #1648

Uh oh!

Conversation

exdx commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

exdx commented Jul 20, 2020

Uh oh!

exdx commented Jul 20, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

openshift-ci-robot commented Jul 20, 2020

Uh oh!

exdx commented Jul 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

exdx commented Jul 17, 2020 •

edited

Loading