-
Notifications
You must be signed in to change notification settings - Fork 702
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate other e2e test engines #2898
Comments
https://agouti.org/ also looks very interesting (found from ginko website). |
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Cypress results so farInitially looked very good, super easy to code steps for testing. Less verbose than current tests. A test for a login to Kubeapps looks as simple as this: describe("Kubeapps CI auth", () => {
it("Test UI login", () => {
const hostname = Cypress.env('INTEGRATION_ENTRYPOINT');
cy.visit(hostname);
cy.get('cds-button').contains("Login via OIDC Provider").click();
cy.get('.dex-container button').contains("Log in with Email").click();
cy.get('input[id="login"]').type("kubeapps-user@example.com").should('have.value', "kubeapps-user@example.com");
cy.get('input[id="password"]').type("password").should('have.value', "password");
cy.get('#submit-login').contains("Login").click();
cy.get('.dex-container button[type="submit"]').contains("Grant Access").click();
// Breaks at this point with Kubeapps throwing a 403
});
}); But Cypress uses iframes for performing the tests, and that breaks OIDC flow because we are changing hosts (Kubeapps -> Dex -> Kubeapps). Then the CSP directive is applied to the iframe, and Kubeapps shows a 403 when calling back. No luck so far on completing a login flow. Tried with obtaining token through REST with DEX, no luck. |
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Playwright results so farSyntax is similar to Pupeteer. Really easy to set up and build tests. Pros
Cons
Built a POC of 4 tests in this branch. Work is in progress. |
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
* Introducing Playwright as e2e test engine (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying new integration image (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Set a higher resource class for Playwright (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Increasing resources for CI (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Removed Kubeops for freeing resources (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Set back replicaCount for FE and Dashboard (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Enabling Kubeops (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Migrate all tests but 1 to Playwright (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Finished migrating e2e tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixing tests folder (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying to improve e2e tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixes and improvements for e2e tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Improvements on e2e tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Tweaking e2e tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fine tuning e2e testing with Playwright (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Use official e2e tests runner image (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Workaround for test timeout (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Improving Playwright tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixing Operators testing (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Remove double quotes around test args (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix tests dependencies paths (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Keep integration image free of test files (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * E2E test improvements, clean-up and add SPDX headers (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Some clean up and small adjustments (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Finally Playwright has been chosen and E2E tests are implemented with it in #4185. |
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
* Fix bearer token login on E2E tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Added positive and negative permissions tests for regular user (#4081) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Re-add rolebinding for kubeapps-user in its namespace (#4081) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
* Trying new CI setup Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix bearer token login on E2E tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Added positive and negative permissions tests for regular user (#4081) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Re-add rolebinding for kubeapps-user in its namespace (#4081) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixing tests for GKE env (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix for E2E running in GKE (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying to fix E2E tests for GKE (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Improving E2E tests setup (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix for CI pipeline in GKE Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixes for E2E tests in GKE Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying E2E tests with lower cluster resources Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Removed temporary comments in CI config Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Temporary fix for e2e CI development Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying test without Docker credentials Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Use new integration image. Small fixes for E2E. Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * E2E Tests fix Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixing E2E tests Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Debugging CI Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Improving GKE CI setup Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Debugging E2E CI tests Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Adding missing timeout param for local E2E Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Tidy up after debugging E2E test Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
* Fakerelease (#4242) * Trying new CI setup Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix bearer token login on E2E tests (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Added positive and negative permissions tests for regular user (#4081) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Re-add rolebinding for kubeapps-user in its namespace (#4081) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixing tests for GKE env (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix for E2E running in GKE (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying to fix E2E tests for GKE (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Improving E2E tests setup (#2898) Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fix for CI pipeline in GKE Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixes for E2E tests in GKE Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying E2E tests with lower cluster resources Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Removed temporary comments in CI config Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Temporary fix for e2e CI development Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Trying test without Docker credentials Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Use new integration image. Small fixes for E2E. Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * E2E Tests fix Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Fixing E2E tests Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Debugging CI Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Improving GKE CI setup Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Debugging E2E CI tests Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Adding missing timeout param for local E2E Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Tidy up after debugging E2E test Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Forcing execution of E2E tests in GKE Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Added some comments to E2E files Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com> * Renamed timeout variables and revert to prerelease branch Signed-off-by: Rafa Castelblanque <rcastelblanq@vmware.com>
Due to the number of the false-positive CI errors we usually get, it could be worthwhile to look at other test engines, like Michael's suggestion [1] , instead of continuing using Puppeteer.
Besides, the API is pretty similar, and the change appears to be straightforward [2]
[1] https://playwright.dev/
[2] https://medium.com/@davert/puppeteer-to-playwright-migration-guide-6c86ea66e85e
Other options to consider:
[3] https://www.cypress.io/
[4] https://agouti.org/ - For a go-based integration test experience, together with ginko etc.
The text was updated successfully, but these errors were encountered: