Improve Test Runner error experience

[jennifer-shehane]

Goal

The goal is to make the process of debugging tests and code easier.

Why

1. Spending less time debugging and deciphering errors means more time for users to do what they want to be doing (e.g. writing more tests, writing more code, stopping writing code because they’ve achieved their goals).
2. When users can solve their own problems, this reduces the time our team spends on support.
3. Better errors makes software easier to use for everyone, especially beginners or non-traditional users who might not have context on how to debug issues outside of what error messages instruct them to do.

Implementation

Types of errors

Current design of error within Command Log

Current design applied to types of errors

WIP Proposed internal error structure

Cypress Error (one code frame)

{
  name: "CypressError",
  message: "", // Error message (logged in console, used in run mode)
  stack: "",
  mdMessage: "", // Error message with escaped markdown (used in GUI)
  codeFrames: [
    {
      frame: "", // Code frame string
      filePath: "" // Path to open file
    }
  ],
  docs: "", // Either full URL or path after on.cypress.io
  template: "cypress-single-frame" // Template type, used to compose full error
}

Cypress Error (two code frames)

{
  name: "CypressError",
  message: "",
  stack: "",
  mdMessage: "",
  codeFrames: [
    { // Code frame to be displayed first, likely the spec code
      frame: "",
      filePath ""
    },
    { // Code frame to be displayed second, such as the plugins code
      frame: "",
      filePath ""
    }
  ],
  docs: "",
  template: "cypress-multiple-frames"
}

Assertion Error

{
  name: "AssertionError",
  message: "",
  stack: "",
  expected: "",
  actual: "",
  diff: "",
  codeFrames: [
    {
      frame: "",
      filePath: ""
    }
  ],
  docs: "",
  template: "assertion"
}

Spec/Application Error

{
  name: "",
  message: "",
  stack: "",
  codeFrames: [
    {
      frame: "",
      filePath: ""
    }
  ],
  docs: "",
  template: "other"
}

Existing Issues

There are a number of existing issues that might make sense to address under the umbrella of “error improvement" (I’m sure there are more):

• Add retry to baseUrl not present warning Be more user-friendly when checking that baseUrl server is responding #3284
• Better messaging when failing by type Better messaging when failing by type  #2050
• Failed assertion UI in open mode Show diffs for assertions logged in browser console #3129
• Module API doesn’t fail the promise with Cypress error message Module API doesn't fail the promise with Cypress failure messages #3216
• Root level after hooks don’t error Exceptions in after() don't show up in UI #3348
• Improve printing error messages in Test Runner Improve printing error messages in Test Runner #1664
• better stack trace/ sourceMaps better stack trace/ sourceMaps #881
• asterisks appearing in command log asterisks displaying instead of single quotes in Command Log assertion #1360
• Add support for backticks in command log Add support for backticks in reporter error messages #3608
• Exceptions in global 'after' are not shown correctly in UI Exceptions in global 'after' are not shown correctly in UI #3904
• Exceptions in after() don't show up in UI Exceptions in after() don't show up in UI #3348
• Exception in root level after() doesn't fail test run in interactive mode Exception in root level after() doesn't fail test run in interactive mode #2296
• Improve uncaught errors Improve Uncaught Errors #1761
• TypeError: Cannot read property 'replace' of undefined" at Object.appendErrMsg TypeError: Cannot read property 'replace' of undefined" at Object.appendErrMsg #1669
• .should('deep.equal') does not show a full diff comparison on failures .should('deep.equal') does not show a full diff comparison on failures #4084
• err when cy.visit('file://..') On cy.visit('file://...) throw err to explain they should visit relative html file #4452
• Show warning message in command log Show warning messages in command log #4917
• Add warning to command log for 'uncaught exception' Issue Command Log 'warning' event during uncaught:exception #4933
• Capture stack traces from spec frame Capture stack traces from spec frame #4932

[scharf]

see Stack overflow question How to figure out in which file and code line a cypress test has failed?

[tedpennings]

I would love to see the current browser URL recorded in Cypress assertion errors and appear in the test error output to stdout/stderr.

Some of our URLs are a little long and with important IDs cutoff in screenshots/video. It would be easier to investigate errors from CI (non-interactive) with URLs logged. (Locally this isn't a problem.) Here's an example from the Cypress.io dashboard -- all of the ids for repro are after the /service-diagram

Thank you!

[chrisbreiding]

More ideas for improvements from discussion with Brian on 1/17:

• Add learn more link to docs in modal too (like in desktop-gui)
• Output code frame in run mode stdout
• Don't link cypress:// error files in stack trace
• Send more data to dashboard (server/lib/reporter.coffee)
  • full error data with code frame, source mapped stack, etc
  • source mapped body
  • ask Jeff

[cypress-bot]

The code for this is done in cypress-io/cypress#6724, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 4.3.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v4.3.0, please open a new issue.

[jennifer-shehane]

This was partially released in 4.3.0. The full release with code frames is still in progress at #3930

[cypress-bot]

The code for this is done in cypress-io/cypress#3930, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 4.6.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v4.6.0, please open a new issue.