Multi-user feature UI integration tests are unstable.

[ximon18]

Note: This relates to the multi-user feature in the v0.8.1-bis branch.

The test in multi_user_config_file_with_ta.js sometimes fails with:

AssertionError: Timed out retrying after 4000ms: Expected to find element: `Click here to refresh`, but never found it.

I suspect that the test needs to wait for Krill to issue the resources and for them to appear in the UI.

[ximon18]

Unfortunately the test is now failing either more often or every time, so the attempt to fix it needs more work.

Config File Users with TA
       Add ROA for CA ca_readonly as readonly user should fail:
     CypressError: Timed out retrying after 5000ms: `cy.wait()` timed out waiting `5000ms` for the 1st request to the route: `analyzeRoutes`. No request ever occurred.

[ximon18]

Update: the "Add ROA" test does fail but not every time.

[ximon18]

Interestingly the same test suite sometimes fails a different way in one of the other tests:

  1) Config File Users with TA
       Register CA ca_readonly with parent as rohelper user should succeed:
     CypressError: The following error originated from your application code, not from Cypress.

  > ResizeObserver loop limit exceeded

When Cypress detects uncaught errors originating from your application it will automatically fail the current test.

This behavior is configurable, and you can choose to turn this off by listening to the `uncaught:exception` event.

https://on.cypress.io/uncaught-exception-from-application
      at cypressErr (https://localhost:3000/__cypress/runner/cypress_runner.js:172812:18)
      at Object.errByPath (https://localhost:3000/__cypress/runner/cypress_runner.js:172863:10)
      at Object.createUncaughtException (https://localhost:3000/__cypress/runner/cypress_runner.js:161554:45)
      at $Cy.onUncaughtException (https://localhost:3000/__cypress/runner/cypress_runner.js:170376:26)
      at onError (https://localhost:3000/__cypress/runner/cypress_runner.js:169436:39)
  From Your Spec Code:
      at Context.eval (https://localhost:3000/__cypress/tests?p=tests/ui/cypress_specs/multi_user_config_file_with_ta.js:398:41)

[ximon18]

I'm now aware of at least four test failures that are sometimes occuring:

Test Script	Test Name	Observed Failure	Examples	Status
multi_user_config_file_with_ta.js	"Add ROA ..."	AssertionError: Timed out retrying after 4000ms: Expected to find element: Click here to refresh, but never found it.	n/a	test disabled
multi_user_config_file_with_ta.js	"Add ROA ..."	CypressError: Timed out retrying after 5000ms: cy.wait() timed out waiting 5000ms for the 1st request to the route: analyzeRoutes. No request ever occurred.	own dev machine	investigating
multi_user_config_file_with_ta.js	"Register CA ..."	ResizeObserver loop limit exceeded *1	example example example example	investigating
multi_user_openid_connect.js	"Login receives short-lived refreshable token"	AssertionError: Timed out retrying after 4000ms: Expected to find element: #userinfo, but never found it.	example
multi_user_openid_connect.js	"Login receives short-lived that cannot be refreshed"	CypressError: Timed out retrying after 5000ms: cy.wait() timed out waiting 5000ms for the 2nd request to the route: isAuthorized. No request ever occurred	example	fixed by b4c9994
multi_user_config_file.js	n/a	docker: Error response from daemon: Get https://registry-1.docker.io/v2/cypress/included/manifests/6.2.0: received unexpected HTTP status: 502 Bad Gateway.	example	new

I can't yet reproduce these locally, presumably because my development machine is faster than the GitHub Actions runner VMs and so the state is as expected rather than not yet having reached the required state.

*1: According to this we should configure cypress to ignore this error. For more information see this as well. Example JS Cypress code that shows how to catch uncaught exceptions can be seen here.

[ximon18]

Update: There have been very few (none?) failures since the ResizeObserver workaround was tuned and the "Add ROAs" test was disabled. The "Add ROAs" test still needs work to make it deterministic however.

[ximon18]

Closing as the UI tests in the dev branch have been stable for a week now. Will re-open if another failure occurs.