Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Dev Catalog e2e tests to Jenkins CI #1082

Merged
merged 1 commit into from
Jan 29, 2019
Merged

Add Dev Catalog e2e tests to Jenkins CI #1082

merged 1 commit into from
Jan 29, 2019

Conversation

dtaylor113
Copy link
Contributor

@dtaylor113 dtaylor113 commented Jan 15, 2019 •
edited

  • added test for when no service classes loaded in Dev Catalog
  • updated catalog-page to update when clusterServiceVersions change
  • renamed catalog.scenario.ts to developer-catalog.scenario.ts and added it to serviceCatalog test suite, which is run during Jenkins CI

@openshift-ci-robot openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 15, 2019
spadgett
spadgett reviewed Jan 15, 2019
View reviewed changes
Copy link
Member

@spadgett spadgett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dtaylor113

jenkins.sh Outdated Show resolved Hide resolved
frontend/integration-tests/protractor.conf.ts Outdated
@@ -96,7 +96,7 @@ export const config: Config = {
olmUpgrade: ['tests/base.scenario.ts', 'tests/olm/update-channel-approval.scenario.ts'],
performance: ['tests/base.scenario.ts', 'tests/performance.scenario.ts'],
serviceCatalog: ['tests/base.scenario.ts', 'tests/service-catalog/service-catalog.scenario.ts', 'tests/service-catalog/service-broker.scenario.ts', 'tests/service-catalog/service-class.scenario.ts', 'tests/service-catalog/service-binding.scenario.ts'],
catalog: ['tests/base.scenario.ts', 'tests/catalog.scenario.ts'],
developerCatalog: ['tests/base.scenario.ts', 'tests/developer-catalog.scenario.ts'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest just making this part of the serviceCatalog suite, particularly if it requires service catalog enabled to pass.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@dtaylor113 dtaylor113 changed the title Add OperatorHub and Dev Catalog e2e tests to Jenkins CI Add Dev Catalog e2e tests to Jenkins CI Jan 15, 2019
spadgett
spadgett approved these changes Jan 16, 2019
View reviewed changes
Copy link
Member

@spadgett spadgett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dtaylor113

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2019
@dtaylor113
Copy link
Contributor Author

/retest

@openshift openshift deleted a comment from spadgett Jan 17, 2019
@openshift openshift deleted a comment from spadgett Jan 17, 2019
@openshift openshift deleted a comment from spadgett Jan 17, 2019
@openshift openshift deleted a comment from spadgett Jan 17, 2019
@openshift openshift deleted a comment from spadgett Jan 17, 2019
@dtaylor113
Copy link
Contributor Author

Trying to verify Service Catalog e2e test w/ dev catalog passes locally, shared cluster seems to be unstable

@spadgett
Copy link
Member

This error can happen periodically when looping over DOM nodes in tests

1) Catalog : filters catalog tiles by 'Service Class' Type correctly
   StaleElementReferenceError: stale element reference: element is not attached to the page document
     (Session info: headless chrome=67.0.3396.62)

@dtaylor113
Copy link
Contributor Author

This error can happen periodically when looping over DOM nodes in tests

I know, it's on lines like: expect(catalogPageView.catalogTileFor('.NET Core').isDisplayed()).toBe(true); which are also on the operator hub e2e tests and have worked for a while now. Verified the css selector hasn't changed. Still investigating....

@spadgett
Copy link
Member

Yeah, it won't happen every time. It will just flake occasionally.

which are also on the operator hub e2e tests and have worked for a while now.

Those tests aren't currently gating merges, so it's hard to say how often they flake.

@spadgett
Copy link
Member

jenkins rebuild

3 similar comments
@spadgett
Copy link
Member

jenkins rebuild

@spadgett
Copy link
Member

jenkins rebuild

@jhadvig
Copy link
Member

jhadvig commented Jan 18, 2019

jenkins rebuild

@spadgett
Copy link
Member

Last failure was flake https://jira.coreos.com/browse/CONSOLE-1214

@spadgett
Copy link
Member

jenkins rebuild

@spadgett
Copy link
Member

Last failed on flake https://jira.coreos.com/browse/CONSOLE-1219

@spadgett
Copy link
Member

jenkins rebuild

@spadgett
Copy link
Member

/hold

Holding for now due to the StaleElementReferenceError. I think we should try to figure that out before merging, particularly since our queue has recently been flakey.

We should determine why the catalog is re-rendering so frequently, but it'd be good to get away from using filter() if possible as well.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 18, 2019
@dtaylor113
Copy link
Contributor Author

We should determine why the catalog is re-rendering so frequently

https://jira.coreos.com/browse/CONSOLE-1223

@openshift-ci-robot openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 18, 2019
@dtaylor113
Copy link
Contributor Author

jenkins rebuild

@dtaylor113
Copy link
Contributor Author

@spadgett This PR seems ready to go! -thanks

spadgett
spadgett approved these changes Jan 24, 2019
View reviewed changes
Copy link
Member

@spadgett spadgett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

There's some follow on work required to enable service catalog in the console-e2e tests so we can continue to gate merges on the service catalog scenarios. The Jenkins tests are going away soon.

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2019
@spadgett
Copy link
Member

jenkins rebuild

1 similar comment
@dtaylor113
Copy link
Contributor Author

jenkins rebuild

@dtaylor113
Copy link
Contributor Author

e2e failure:

Failed Specs:

1) Visiting Overview page : shows an emtpy list when no resources exist
   Error: Failed expectation
       at Object.<anonymous> /go/src/github.com/openshift/console/frontend/integration-tests/tests/overview/overview.scenario.ts:25:65
       at step /go/src/github.com/openshift/console/frontend/integration-tests/tests/overview/overview.scenario.ts:32:23
       at Object.next /go/src/github.com/openshift/console/frontend/integration-tests/tests/overview/overview.scenario.ts:13:53
       at fulfilled /go/src/github.com/openshift/console/frontend/integration-tests/tests/overview/overview.scenario.ts:4:58
Level { name_: 'SEVERE', value_: 1000 } 'http://localhost:9000/path-ZHAFIxCrKH/api/kubernetes/api/v1/namespaces/openshift-logging/configmaps/sharing-config - Failed to load resource: the server responded with a status of 404 (Not Found)'

@spadgett
Copy link
Member

jenkins rebuild

FYI, this isn't needed anymore (and will start some unnecessary tests)

@spadgett
Copy link
Member

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 25, 2019
@spadgett
Copy link
Member

e2e failed because the install failed

/retest

@spadgett
Copy link
Member

/retest

@openshift openshift deleted a comment from spadgett Jan 29, 2019
@openshift openshift deleted a comment from spadgett Jan 29, 2019
@openshift-merge-robot openshift-merge-robot merged commit af38aff into openshift:master Jan 29, 2019
@dtaylor113 dtaylor113 deleted the dev-catalog-e2e-updates branch August 22, 2019 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spadgett spadgett spadgett approved these changes

@jeff-phillips-18 jeff-phillips-18 jeff-phillips-18 approved these changes

Assignees

@spadgett spadgett

Labels
lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@dtaylor113 @spadgett @jhadvig @jeff-phillips-18 @openshift-ci-robot @openshift-merge-robot