Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Test Runner error experience #3762

Closed
jennifer-shehane opened this issue Mar 21, 2019 · 9 comments
Closed

Improve Test Runner error experience #3762

jennifer-shehane opened this issue Mar 21, 2019 · 9 comments

Comments

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented Mar 21, 2019

Goal

The goal is to make the process of debugging tests and code easier.

Why

  1. Spending less time debugging and deciphering errors means more time for users to do what they want to be doing (e.g. writing more tests, writing more code, stopping writing code because they’ve achieved their goals).
  2. When users can solve their own problems, this reduces the time our team spends on support.
  3. Better errors makes software easier to use for everyone, especially beginners or non-traditional users who might not have context on how to debug issues outside of what error messages instruct them to do.

Implementation

Types of errors

Current design of error within Command Log

error-design-7

Current design applied to types of errors

error-designs-8

WIP Proposed internal error structure

Cypress Error (one code frame)

{
  name: "CypressError",
  message: "", // Error message (logged in console, used in run mode)
  stack: "",
  mdMessage: "", // Error message with escaped markdown (used in GUI)
  codeFrames: [
    {
      frame: "", // Code frame string
      filePath: "" // Path to open file
    }
  ],
  docs: "", // Either full URL or path after on.cypress.io
  template: "cypress-single-frame" // Template type, used to compose full error
}

Cypress Error (two code frames)

{
  name: "CypressError",
  message: "",
  stack: "",
  mdMessage: "",
  codeFrames: [
    { // Code frame to be displayed first, likely the spec code
      frame: "",
      filePath ""
    },
    { // Code frame to be displayed second, such as the plugins code
      frame: "",
      filePath ""
    }
  ],
  docs: "",
  template: "cypress-multiple-frames"
}

Assertion Error

{
  name: "AssertionError",
  message: "",
  stack: "",
  expected: "",
  actual: "",
  diff: "",
  codeFrames: [
    {
      frame: "",
      filePath: ""
    }
  ],
  docs: "",
  template: "assertion"
}

Spec/Application Error

{
  name: "",
  message: "",
  stack: "",
  codeFrames: [
    {
      frame: "",
      filePath: ""
    }
  ],
  docs: "",
  template: "other"
}

Existing Issues

There are a number of existing issues that might make sense to address under the umbrella of “error improvement" (I’m sure there are more):

  • Add retry to baseUrl not present warning #3284
  • Better messaging when failing by type #2050
  • Failed assertion UI in open mode #3129
  • Module API doesn’t fail the promise with Cypress error message #3216
  • Root level after hooks don’t error #3348
  • Improve printing error messages in Test Runner #1664
  • better stack trace/ sourceMaps #881
  • asterisks appearing in command log #1360
  • Add support for backticks in command log #3608
  • Exceptions in global 'after' are not shown correctly in UI #3904
  • Exceptions in after() don't show up in UI #3348
  • Exception in root level after() doesn't fail test run in interactive mode #2296
  • Improve uncaught errors #1761
  • TypeError: Cannot read property 'replace' of undefined" at Object.appendErrMsg #1669
  • .should('deep.equal') does not show a full diff comparison on failures #4084
  • err when cy.visit('file://..') #4452
  • Show warning message in command log #4917
  • Add warning to command log for 'uncaught exception' #4933
  • Capture stack traces from spec frame #4932
@scharf
Copy link

@scharf scharf commented Apr 17, 2019

@tedpennings
Copy link

@tedpennings tedpennings commented Jun 7, 2019

I would love to see the current browser URL recorded in Cypress assertion errors and appear in the test error output to stdout/stderr.

Some of our URLs are a little long and with important IDs cutoff in screenshots/video. It would be easier to investigate errors from CI (non-interactive) with URLs logged. (Locally this isn't a problem.) Here's an example from the Cypress.io dashboard -- all of the ids for repro are after the /service-diagram
cypress-screenshot

Thank you!

@chrisbreiding
Copy link
Contributor

@chrisbreiding chrisbreiding commented Jan 21, 2020

More ideas for improvements from discussion with Brian on 1/17:

  • Add learn more link to docs in modal too (like in desktop-gui)
  • Output code frame in run mode stdout
  • Don't link cypress:// error files in stack trace
  • Send more data to dashboard (server/lib/reporter.coffee)
    • full error data with code frame, source mapped stack, etc
    • source mapped body
    • ask Jeff

@cypress-bot
Copy link

@cypress-bot cypress-bot bot commented Mar 17, 2020

The code for this is done in cypress-io/cypress#6724, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@cypress-bot
Copy link

@cypress-bot cypress-bot bot commented Mar 30, 2020

Released in 4.3.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v4.3.0, please open a new issue.

@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators Mar 30, 2020
@cypress-io cypress-io unlocked this conversation May 4, 2020
@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented May 4, 2020

This was partially released in 4.3.0. The full release with code frames is still in progress at #3930

@cypress-bot
Copy link

@cypress-bot cypress-bot bot commented May 6, 2020

The code for this is done in cypress-io/cypress#3930, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@cypress-bot
Copy link

@cypress-bot cypress-bot bot commented May 20, 2020

Released in 4.6.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v4.6.0, please open a new issue.

@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators May 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.