Bug 1880118: Cypress better headless logging

[dtaylor113]

This PR improves logging in headless mode. Cypress best-practice of test independence results in larger independent tests, as opposed to previously many small dependent tests. In headless mode, cypress only outputs the test names as they run; as a result of fewer tests, there isn't a lot of output:

BEFORE

  Monitoring: Alerts
No BRIDGE_KUBEADMIN_PASSWORD set, skipping login
    ✓ displays and filters the Alerts list page, links to detail pages (4419ms)
No BRIDGE_KUBEADMIN_PASSWORD set, skipping logout
    ✓ creates and expires a Silence (2885ms)

In the gui Cypress Test Runner we use cy.log(...)'s to help determine test progression:

This PR exports these cy.log messages to headless terminal output, as well as improving logging messages:

AFTER

Monitoring: Alerts
  skipping login, no BRIDGE_KUBEADMIN_PASSWORD set
      create project
      use sidebar nav to goto Monitoring -> Alerting
      filter Alerts
      drills down to Alert details page
      drill down to the Alerting Rule details page
      drill back up to the Alert details page
    ✓ displays and filters the Alerts list page, links to detail pages (8727ms)
      filter to Watchdog alert
      silence Watchdog alert
*     shows the silenced Alert in the Silenced Alerts list 
*     shows the newly created Silence in the Silenced By list
*     expires the Silence
      delete project
  skipping logout
    ✓ creates and expires a Silence (25350ms)

* These log messages will only appear after #6628 merges (flake)!

In addition this PR:

• updates retries in headless to 1, and 0 when in test runner.
• changes mochawesomeReporter to quiet mode which removes: [mochawesome] Report JSON saved to /home/dtaylor/repos/openshift/origin/src/github.com/openshift/console/frontend/gui_test_screenshots/cypress_report_004.json output
• turns off 'verbose' copying of artifacts to gui_test_screenshots dir

[dtaylor113]

/retest

[spadgett]

Hey, @dtaylor113. Are you concerned with running headless in CI or running headless locally?

To be honest, I prefer the current behavior. I find it confusing to interleave the log messages and test results in the console.

[dtaylor113]

Hey, @dtaylor113. Are you concerned with running headless in CI or running headless locally?

To be honest, I prefer the current behavior. I find it confusing to interleave the log messages and test results in the console.

Adding the log msgs to the output allowed me to discover a flake and IMO really helps see what is occuring so that when a failure happens I can clearly and quickly see the last log message.
I realize we have screenshots and movies, but it takes some clicks to get to those artifacts and sometimes not clear what movie/screenshot to view; whereas the test log will have these log messages and I can easily see what and where something failed.
Without the log messages I find just the test results very sparse. Also, we control what log messages we use. For example, I removed all log messages from the crud tests because the tests themselves give enough info on what is occuring.

[dtaylor113]

Hi @spadgett , indented the log messages so that the test names stands out more:

 Namespace
  skipping login, no BRIDGE_KUBEADMIN_PASSWORD set
      create project
      test Namespace list page
      creates the Namespace
      delete the Namespace
      delete project
  skipping logout
    ✓ lists, creates, and deletes (21814ms)

Monitoring: Alerts
  skipping login, no BRIDGE_KUBEADMIN_PASSWORD set
      create project
      use sidebar nav to goto Monitoring -> Alerting
      filter Alerts
      drills down to Alert details page
      drill down to the Alerting Rule details page
      drill back up to the Alert details page
    ✓ displays and filters the Alerts list page, links to detail pages (8727ms)
      filter to Watchdog alert
      silence Watchdog alert
      shows the silenced Alert in the Silenced Alerts list 
      shows the newly created Silence in the Silenced By list
      expires the Silence
      delete project
  skipping logout
    ✓ creates and expires a Silence (25350ms)

[openshift-ci-robot]

@dtaylor113: This pull request references Bugzilla bug 1880118, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.6.0) matches configured target release for branch (4.6.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1880118: Cypress better headless logging

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spadgett]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtaylor113, spadgett

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [spadgett]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dtaylor113]

/retest

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci-robot]

@dtaylor113: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

• openshift/console#6698 is open

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state.

Bugzilla bug 1880118 has not been moved to the MODIFIED state.

In response to this:

Bug 1880118: Cypress better headless logging

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.