Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Test Runner error experience #3762

Open
jennifer-shehane opened this issue Mar 21, 2019 · 1 comment

Comments

@jennifer-shehane
Copy link
Member

commented Mar 21, 2019

Goal

The goal is to make the process of debugging tests and code easier.

Why

  1. Spending less time debugging and deciphering errors means more time for users to do what they want to be doing (e.g. writing more tests, writing more code, stopping writing code because they’ve achieved their goals).
  2. When users can solve their own problems, this reduces the time our team spends on support.
  3. Better errors makes software easier to use for everyone, especially beginners or non-traditional users who might not have context on how to debug issues outside of what error messages instruct them to do.

Implementation

Types of errors

Current design of error within Command Log

error-design-7

Current design applied to types of errors

error-designs-8

WIP Proposed internal error structure

Cypress Error (one code frame)

{
  name: "CypressError",
  message: "", // Error message (logged in console, used in run mode)
  stack: "",
  mdMessage: "", // Error message with escaped markdown (used in GUI)
  codeFrames: [
    {
      frame: "", // Code frame string
      filePath: "" // Path to open file
    }
  ],
  docs: "", // Either full URL or path after on.cypress.io
  template: "cypress-single-frame" // Template type, used to compose full error
}

Cypress Error (two code frames)

{
  name: "CypressError",
  message: "",
  stack: "",
  mdMessage: "",
  codeFrames: [
    { // Code frame to be displayed first, likely the spec code
      frame: "",
      filePath ""
    },
    { // Code frame to be displayed second, such as the plugins code
      frame: "",
      filePath ""
    }
  ],
  docs: "",
  template: "cypress-multiple-frames"
}

Assertion Error

{
  name: "AssertionError",
  message: "",
  stack: "",
  expected: "",
  actual: "",
  diff: "",
  codeFrames: [
    {
      frame: "",
      filePath: ""
    }
  ],
  docs: "",
  template: "assertion"
}

Spec/Application Error

{
  name: "",
  message: "",
  stack: "",
  codeFrames: [
    {
      frame: "",
      filePath: ""
    }
  ],
  docs: "",
  template: "other"
}

Existing Issues

There are a number of existing issues that might make sense to address under the umbrella of “error improvement" (I’m sure there are more):

  • Add retry to baseUrl not present warning #3284
  • Better messaging when failing by type #2050
  • Failed assertion UI in open mode #3129
  • Module API doesn’t fail the promise with Cypress error message #3216
  • Root level after hooks don’t error #3348
  • Improve printing error messages in Test Runner #1664
  • better stack trace/ sourceMaps #881
  • asterisks appearing in command log #1360
  • Add support for backticks in command log #3608
  • Exceptions in global 'after' are not shown correctly in UI #3904
  • Exceptions in after() don't show up in UI #3348
  • Exception in root level after() doesn't fail test run in interactive mode #2296
  • Improve uncaught errors #1761
  • TypeError: Cannot read property 'replace' of undefined" at Object.appendErrMsg #1669
  • .should('deep.equal') does not show a full diff comparison on failures #4084
@scharf

This comment has been minimized.

Copy link

commented Apr 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.