Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1909836: Re-enable and update operator-install-* tests to add catalog-source #7699

Merged
merged 1 commit into from Jan 12, 2021

Conversation

rhamilto
Copy link
Member

@rhamilto rhamilto commented Jan 4, 2021

…alog-source

@openshift-ci-robot openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. labels Jan 4, 2021
@openshift-ci-robot
Copy link
Contributor

@rhamilto: This pull request references Bugzilla bug 1909836, which is invalid:

  • expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

[WIP] Bug 1909836: Re-enable and update operator-install-* tests to add catalog-source

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. component/olm Related to OLM labels Jan 4, 2021
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 4, 2021
@rhamilto rhamilto force-pushed the bz1909836 branch 3 times, most recently from 9934ab9 to 6d2675b Compare January 5, 2021 13:43
cy.log('verify the packagemanifest has been created');
cy.visit('/k8s/ns/openshift-marketplace/packages.operators.coreos.com~v1~PackageManifest');
listPage.filter.byName(operatorPkgName);
listPage.rows.shouldExist(operatorName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So is this where it fails? Times out after 30 seconds (the default timeout)? If so, you might want to add a longer timeout to listPage.rows.shouldExist (90 seconds) like we did for listPage.rows.shouldNotExist.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I load the list of PackageManifests before all the PackageManifests are created, they never will all load (I guess we're not watching them here?). I put the wait of 60 seconds in after creation of the CatalogSource and everything works as expected. I realize this isn't ideal, but I'm trying to just get something that works at the moment.

If we had a CatalogSource with just the two Operators we needed for the tests, that would certainly help, but I'm not sure we want to do that since that would require the creation of another image that someone would have to maintain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm...ok, it might be flaky under slower CI, something to watch for.
If your interested :-) https://clearmeasure.com/polling-a-page-in-cypress/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I load the list of PackageManifests before all the PackageManifests are created, they never will all load (I guess we're not watching them here?).

@spadgett, any idea why /k8s/ns/openshift-marketplace/packages.operators.coreos.com~v1~PackageManifest is not dynamically updating? Seems like a bug?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented the polling technique @dtaylor113 mentioned.

@rhamilto rhamilto force-pushed the bz1909836 branch 2 times, most recently from a0f797a to 1ded875 Compare January 5, 2021 18:51
@dtaylor113
Copy link
Contributor

/retest

Copy link
Contributor

@dtaylor113 dtaylor113 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests look good to me. Only issue is the Cypress anti-pattern of Having-tests-rely-on-the-state-of-previous-tests.

@@ -104,15 +130,11 @@ xdescribe(`Interacting with a global install mode Operator (${operatorName})`, (
);
});

it(`uninstalls the operator`, () => {
cy.log('navigate to the operator uninstall modal in OperatorHub');
it(`uninstalls the Operator`, () => {
Copy link
Contributor

@dtaylor113 dtaylor113 Jan 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @rhamilto, we are running into the Cypress anti-pattern of Having-tests-rely-on-the-state-of-previous-tests. Here you have an it to create the operator, another it to test it's details, and a final it to uninstall the operator. While this seems very logical, it is frowned upon by the Cypress gods :-) All its should be able to run independently; you should be able to specify a its.only for any test and it should be able to run.
To resolve I think these three tests should be in a single it test with alot of cy.log statements.

From Cypress best practices about this:
"This ... is ideal because now we are resetting the state between each test and ensuring nothing in previous tests leaks into subsequent ones.
We’re also paving the way to make it less complicated to write multiple tests against the “default” state of the form. That way each test stays lean but each can be run independently and pass."

I think this also may come into play when/if we ever get to running tests in parallel.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not opposed to this, but I was trying to maintain focus on getting the tests re-enabled. This would need to be one it in order to comply with the best practice since the first it installs the operator.

});

it(`uninstalls the operator from ${testName} namespace`, () => {
cy.log('navigate to the operator uninstall modal in OperatorHub');
it(`uninstalls the Operator from ${testName} namespace`, () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about running into the Cypress anti-pattern of Having-tests-rely-on-the-state-of-previous-tests.

@@ -13,6 +13,28 @@ const operatorInstance = 'StorageCluster';
const openshiftOperatorsNS = 'openshift-operators';
const operandLink = 'portworx';

const verifyPackageManifest = (opName: string, csName: string, depth: number = 0) => {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dtaylor113, I made an attempt at adding polling per #7699 (comment). What do you think?

If this is good, I will move this and the rest of CatalogSource install and uninstall out to a separate file so it can be shared between the two tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I like it!

@rhamilto rhamilto changed the title [WIP] Bug 1909836: Re-enable and update operator-install-* tests to add catalog-source Bug 1909836: Re-enable and update operator-install-* tests to add catalog-source Jan 6, 2021
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 6, 2021
@rhamilto
Copy link
Member Author

rhamilto commented Jan 6, 2021

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jan 6, 2021
@openshift-ci-robot
Copy link
Contributor

@rhamilto: This pull request references Bugzilla bug 1909836, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@rhamilto
Copy link
Member Author

rhamilto commented Jan 6, 2021

@dtaylor113, @spadgett PTAL.

.get(`[data-test-rows="resource-row"]`)
.should('exist')
.then(() => {
// the opName can exist in multiple CatalogSources, so check for the csName
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use the new "Operators" tab on the CatalogSource added in #7430. I'd think it would simplify the logic here and add more test coverage for that feature.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion. @spadgett. I've implemented it. Please take another look.

@openshift-ci-robot openshift-ci-robot added the component/core Related to console core functionality label Jan 8, 2021
@dtaylor113
Copy link
Contributor

/retest

Copy link
Member

@spadgett spadgett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
/lgtm

@spadgett spadgett added this to the v4.7 milestone Jan 12, 2021
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 12, 2021
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhamilto, spadgett

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 77aa4b6 into openshift:master Jan 12, 2021
@openshift-ci-robot
Copy link
Contributor

@rhamilto: All pull requests linked via external trackers have merged:

Bugzilla bug 1909836 has been moved to the MODIFIED state.

In response to this:

Bug 1909836: Re-enable and update operator-install-* tests to add catalog-source

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@rhamilto rhamilto deleted the bz1909836 branch January 12, 2021 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. component/core Related to console core functionality component/olm Related to OLM lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants