Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: test replay docs #5349

Merged
merged 28 commits into from
Aug 22, 2023
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
4f1eb4d
sidebar content shuffle
jaffrepaul Jun 23, 2023
4c99796
main feature content & link updates
jaffrepaul Jun 23, 2023
7d07942
add screenshots
jaffrepaul Jun 26, 2023
eacccf9
add url to vpn whitelist
jaffrepaul Jun 28, 2023
6e1cd0f
add TR banner
jaffrepaul Jul 10, 2023
9ea688c
add & reorder faq
jaffrepaul Jul 6, 2023
870bbb3
build test replay partial
jaffrepaul Jun 9, 2023
42edbbe
add TR partial to related content
jaffrepaul Jul 12, 2023
1d73521
add TR link to related "see more" sections
jaffrepaul Jul 12, 2023
e3154b8
content tweaks
jaffrepaul Jul 13, 2023
1af882f
include TR content in body copy
jaffrepaul Jul 14, 2023
c52cce5
add troubleshooting
jaffrepaul Jul 17, 2023
4b27ff5
remove old images
jaffrepaul Jul 26, 2023
6ded7ef
add PR feedback
jaffrepaul Aug 2, 2023
61e1082
add command log notes
jaffrepaul Aug 2, 2023
6b7a157
remove dead link
jaffrepaul Aug 3, 2023
9e557c7
update images
jaffrepaul Aug 16, 2023
eb9c299
replace runs page screenshots
jaffrepaul Aug 16, 2023
b2cad3c
Apply suggestions from code review
jaffrepaul Aug 17, 2023
c4aab0f
add more faqs
jaffrepaul Aug 18, 2023
6293a93
restore missing partial export
jaffrepaul Aug 21, 2023
e0fbb71
fix links
jaffrepaul Aug 21, 2023
b4f41a7
swap in screenshot of new network panel
jaffrepaul Aug 21, 2023
746bd62
Apply suggestions from code review
jaffrepaul Aug 21, 2023
690ef5c
fix command line merge issue
jaffrepaul Aug 21, 2023
1fce3d2
lint
jaffrepaul Aug 21, 2023
323de1f
Update docs/guides/cloud/debugging/recorded-runs.mdx
jaffrepaul Aug 21, 2023
fc799c0
Apply suggestions from code review
jaffrepaul Aug 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/api/commands/screenshot.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -356,9 +356,10 @@ following:
## See also

- [After Screenshot API](/api/plugins/after-screenshot-api)
- [`cy.debug()`](/api/commands/debug)
- [`Cypress.Screenshot`](/api/cypress-api/screenshot-api)
- [Cypress Cloud](/guides/cloud/introduction)
- [`cy.debug()`](/api/commands/debug)
- [`.pause()`](/api/commands/pause)
- [Cypress Cloud](/guides/cloud/introduction)
- [Test Replay](/guides/cloud/debugging/test-replay)
- [Screenshots and Videos](/guides/guides/screenshots-and-videos)
- [Visual Testing](/guides/tooling/visual-testing)
121 changes: 106 additions & 15 deletions docs/faq/questions/cloud-faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,21 +27,34 @@ You can read more [here](/guides/cloud/introduction).

### <Icon name="angle-right" /> What does Cypress record?

We capture the following:
Cypress captures the following:
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

- Standard Output
- Test Failures
When you run Cypress via `cypress run` passing the `--record` flag the following
data is sent to the Cloud for every run:

- Standard output in the terminal
- Test results
- Test definitions
- Cypress configuration (minus Cypress environment variables)
- Screenshots
- Video
- Videos
- OS environment variables related to CI and git information

You have the option to
[delete videos or screenshots](/guides/guides/screenshots-and-videos#Control-which-videos-to-keep-and-upload-to-Cypress-Cloud)
before being sent to the Cloud so those are not captured.

We have already begun the implementation for capturing even more things from
your run such as:
When [Test Replay](#Test-Replay) capturing is enabled, the additional data outlined
below will be sent to the Cloud. You can
[disable capturing](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) this data.

- Commands
- Network Traffic
- Browser Console Logs
- The rendered DOM and CSS styles for the application under test
- Cypress commands and events represented in the Command Log
- Network traffic within your application under test
- Browser console logs

These will be added in subsequent releases.
Cypress Cloud does not capture anything related to the code of your application
under test or the code within any associated repositories that are associated with your project.

### <Icon name="angle-right" /> How is this different than CI?

Expand Down Expand Up @@ -74,7 +87,7 @@ the number of tests you record each month in your organization.

Tests are recorded when [cypress run](/guides/guides/command-line#cypress-run)
is called with the `--record` flag while supplying the record `--key`. This
means your [test run data](/guides/cloud/recorded-runs#Latest-Runs) is being
means your [test run data](/guides/cloud/debugging/recorded-runs#Latest-Runs) is being
"recorded" to Cypress Cloud.

We consider each time the `it()` function is called to be a single test. So you
Expand Down Expand Up @@ -151,7 +164,7 @@ any test results.
### <Icon name="angle-right" /> What is the projectId for?

The `projectId` is a 6 character string that helps identify your project once
you've [set up your tests to record](/guides/cloud/recorded-runs#Latest-Runs).
you've [set up your tests to record](/guides/cloud/debugging/recorded-runs#Latest-Runs).
It's generated by Cypress and typically is found in your
[Cypress configuration](/guides/references/configuration).

Expand All @@ -172,7 +185,7 @@ section of the [Cypress Cloud](/guides/cloud/introduction) docs.
### <Icon name="angle-right" /> What is a Record Key?

A _Record Key_ is a GUID that's generated automatically by Cypress once you've
[set up your tests to record](/guides/cloud/recorded-runs#Latest-Runs). It helps
[set up your tests to record](/guides/cloud/debugging/recorded-runs#Latest-Runs). It helps
identify your project and authenticate that your project is even _allowed_ to
record tests.

Expand All @@ -191,14 +204,14 @@ section of the [Cypress Cloud](/guides/cloud/introduction) docs.
### <Icon name="angle-right" /> How do I record my tests?

1. First [set up the project to record](/guides/cloud/getting-started#Setup).
2. Then [record your runs](/guides/cloud/recorded-runs#Latest-Runs).
2. Then [record your runs](/guides/cloud/debugging/recorded-runs#Latest-Runs).

After recording your tests, you will see them in
[Cypress Cloud](https://on.cypress.io/cloud) and in the Cypress App [Runs](/guides/core-concepts/cypress-app#Runs) tab.

### <Icon name="angle-right" /> Can I delete a run from Cypress Cloud?

You can [archive a run](/guides/cloud/recorded-runs#Archive-run) so that it does
You can [archive a run](/guides/cloud/debugging/recorded-runs#Archive-run) so that it does
not display in the runs list or in analytics.

**Note:** Archiving the recorded runs has no effect on the amount of tests
Expand Down Expand Up @@ -267,6 +280,7 @@ Cloud.
- `https://assets.cypress.io` - **Asset CDN** (Org logos, icons, videos,
screenshots, etc.)
- `https://authenticate.cypress.io` - **Authentication API**
- `https://capture.cypress.io` - **Cypress Test Replay**
- `https://cloud.cypress.io` - **Cypress Cloud**
- `https://docs.cypress.io` - **Cypress documentation**
- `https://download.cypress.io` - **CDN download of Cypress binary**
Expand Down Expand Up @@ -311,3 +325,80 @@ You can delete your Cypress account from
[your Cypress Cloud profile](https://cloud.cypress.io/profile). Deleting your
account cannot be undone! By deleting your Cypress account, all associated data
in your account will be permanently deleted.

## Test Replay

### <Icon name="angle-right" /> What is Test Replay?

[Test Replay](/guides/cloud/debugging/test-replay) enables you and your team to troubleshoot and debug failed tests faster. After updating to Cypress `v13`, the new feature will automatically record all browser events and allow you to “replay” what your application under test looked like.

Wind back the clock to any point in an application's execution and directly interact with tests as they happened in CI. Debug complex problems as if you were there when they first happened. You can:
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

- **Inspect the DOM** at the exact time of a test failure to debug tests quickly
- **View important debugging dimensions** such as network requests, console logs, and more
- **Save time recreating errors in CI locally** - replay tests as they happened in CI

<DocsImage
src="/img/guides/cloud/test-replay/test-replay-ui.png"
alt="Test Replay Demo"
width="80%"
/>

### <Icon name="angle-right" /> What types of testing can I debug with Test Replay?

Anything you can do with Cypress - E2E, Component, API, etc.

### <Icon name="angle-right" /> How is Cypress Test Replay different from other "replay" services?

There is an important distinction between _session_ replay services (LogRocket, FullStory, DataDog, etc) and Cypress Test Replay. In session replay user actions are captured as the application is used and delivered back in a replayable format, usually video or stitched DOM snapshots. These are valuable tools for gathering product insight such as user behavior, per session.
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

Cypress Test Replay captures every detail of your test runs as they happen in CI. Remember, these tests are running in a headless manner on a virtual machine with no UI being rendered. Sometimes tests will error or fail indicating an issue in your application. Since Cypress Cloud is monitoring the health of your CI test suite, each Test Replay offers a chance to step back in time to analyze and leverage [time travel debugging](/guides/core-concepts/cypress-app#Time-traveling), network requests, console logs, JavaScript errors, and element rendering to address problems directly. This is incredibly valuable for developer and team productivity. No more hours wasted recreating CI issues on your local machine!

### <Icon name="angle-right" /> Does the replay reflect the real time it took for the test to run?

Yes! The test is captured in Cypress Cloud as it executed in your CI run. That means the replay will show things like [slow typing](/api/cypress-api/keyboard-api#Slow-down-typing-by-increasing-the-keystroke-delay) or any [`cy.wait()`](/api/commands/wait) included in your test.

### <Icon name="angle-right" /> Are there performance implications when using Test Replay?

Expect an even performance exchange with Test Replay vs. video. There is more activity to capture and replay tests but without the overhead of recording, compressing, and storing video assets. `video` capture is set to `false` by default starting in Cypress `v13`. See the full [v13 changelog](/guides/references/changelog#12-0-0).
AtofStryker marked this conversation as resolved.
Show resolved Hide resolved

### <Icon name="angle-right" /> Is every aspect of my stack included in Test Replay?

Our aim is to create an impactful debugging experience that covers the most ground for users. However, if you have a need that would make Test Replay even more valuable, please [submit an issue](https://github.com/cypress-io/cypress/issues/new?assignees=&labels=&projects=&template=4-feature.yml).

We will continue to expand and can see a need to support (to name a few):

- Web sockets
- Canvas elements
- Shadow DOM
- Server side events

### <Icon name="angle-right" /> Can I use Test Replay for tests recorded in different browsers?

Test Replay leverages [Chrome DevTools Protocol(CDP)](https://chromedevtools.github.io/devtools-protocol/), so currently supports Chromium-based browsers (Chrome, Edge, and Electron) only.

Test Replay would be disabled, with a message that it's only available on Chromium, for tests run in Firefox or Webkit (Safari). You can still record and capture test [artifacts](/guides/cloud/debugging/recorded-runs#Artifacts) (screenshots, videos and CI logs) via other browsers in separate [run groups](/guides/cloud/smart-orchestration/parallelization#Grouping-test-runs).

### <Icon name="angle-right" /> Can I replay tests from historical Cypress Cloud runs?

Test Replay is available for tests recorded using Cypress `v13` and up. Tests recorded prior to this will not have Test Replay enabled in Cypress Cloud. You will still have any artifacts collected during the pre `v13` test runs.

### <Icon name="angle-right" /> Can I enable or disable Test Replay for specific, individual tests, or is it a global setting? Can I configure my settings so that Test Replay is only enabled for failing test retries?

At this time, users can only opt out of Test Replay via [project-level settings](/guides/cloud/debugging/test-replay#Opt-out-of-Test-Replay) in Cypress Cloud. There is no local option for opting out at this time.
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

### <Icon name="angle-right" /> Can I share my test replays?

Yes! You can share Test Replays to any team members who have access to view your Cypress Cloud runs like QA, marketing, design, etc. Open the replay, copy the url from the browser address bar and pass it along.

### <Icon name="angle-right" /> Will Test Replay write any assets to the file system?

Yes, but it is stored in a temporary directory and deleted after the run. There is nothing to `.gitignore`.

### <Icon name="angle-right" /> Is the network tab feature available exclusively in Test Replay?

All of the data within Test Replay's Developer Tools in Cypress Cloud (including network events) are already available to you when running tests locally in the Cypress app via your browser's built-in developer tools (e.g. Chrome DevTools). When reviewing a test run that occurred in CI, you can review the network events within Test Reaplay in Cypress Cloud.
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

### <Icon name="angle-right" /> Can I `console.log()` data in the console view of the Developer Tools panel?

Yes. This displays console logs from within the application under test or your spec files. You may need to use `JSON.stringify(Object)` to display deeply nested data.
6 changes: 6 additions & 0 deletions docs/guides/cloud/debugging/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"label": "Debugging",
"position": 30,
"collapsible": true,
"collapsed": true
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Recorded Runs
sidebar_position: 30
sidebar_position: 10
---

Recorded runs capture the results from your test runs. Each specific area of
Expand Down Expand Up @@ -119,9 +119,8 @@ It orders results by:
- **Flaky** - when a test failed one or more attempts before finally passing
- **Modified** - when the test body has changed from a previously recorded tests

The panel shows these tests broken down by spec, with easy access to run
artifacts, and a badge clearly indicating the reason the result was considered
notable.
The panel shows these tests broken down by spec, with a badge clearly indicating the reason the result was considered notable. Access run artifacts with a mouse hover to quickly expose the [Test Replay](/guides/cloud/debugging/test-replay) or [screenshots](/guides/guides/screenshots-and-videos#Screenshots),
[videos](/guides/guides/screenshots-and-videos#Videos) and CI logs generated by this test.
jaffrepaul marked this conversation as resolved.
Show resolved Hide resolved

Clicking on a specific test opens the
[test detail sidebar](#Test-detail-sidebar).
Expand Down Expand Up @@ -228,7 +227,7 @@ to raise tickets on integrated project management systems or view artifacts.
### Attempts and errors

Stack traces, error message, and point-of-failure code frames for each attempt
of a test.
of a test. View or debug artifacts of each particular attempt.

<DocsImage
src="/img/guides/cloud/runs/sidebar-errors.png"
Expand All @@ -247,6 +246,9 @@ successes on the current branch.

### Artifacts

[Test Replay](/guides/cloud/debugging/test-replay) to view the
test as it executed in CI with full debug capability.

[Screenshots](/guides/guides/screenshots-and-videos#Screenshots),
[videos](/guides/guides/screenshots-and-videos#Videos) and CI logs generated by
this test. Both screenshots taken during failures and screenshots taken using
Expand Down