Introducing in-app test results: A streamlined, local workflow for debugging CI failures

[pstakoun]

Today, with the release of Cypress 12.5.0, we are excited to introduce a game-changing Cloud workflow to Cypress, that allows you to easily debug failed CI test runs right from your local app. With the new Debug page, you can review, rerun, and debug failed CI test runs that are recorded to Cypress Cloud – all without leaving the comfort of your local Cypress app.

We encourage you to provide any feedback that you may have, as we are actively iterating on and improving this feature. Please use this thread to provide feedback, request features, or just comment on the new Debug page in general.

If you have a larger bug to report, please file a dedicated issue.

[lucasdonato]

Hi, I loved the new debug functionality, however in the local Cypres App only the last execution of the Cypress Cloud is displayed, in my case I run the tests in parallel and I cannot see all the errors in the debug page.

[tbiethman]

Hi @lucasdonato, thank you for your feedback. Can you confirm that you are setting unique --group flags for your parallel recordings? Doing so will allow Cypress to combine both of your recordings into a single run and ensure that all your test results are present in the run shown in Debug page. More info about grouping can be found in in our docs.

Or is there a reason why you would prefer to continue generating separate runs for these workflows?

[lucasdonato]

@tbiethman I used --group and it worked perfectly, thank you very much ;)

[spyglass24]

Hi @pstakoun , I have just added tag "--group daily" and it's failing on the bibucket run, only one rundom command is executing without teardown error. Rest of the runs have the following error message:
Searching for test report files in directories named [test-results, failsafe-reports, test-reports, TestResults, surefire-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.

Here is the .yml file

screen from bitbucket pipe

Any idea ?

[tbiethman]

Hi @spyglass24, thank you for reaching out. Do you see any warnings/errors in the collapsed npm run cypress:pip-smk || true step above your Build teardown step? I see that step only took 6s, which leads me to believe it errored for some reason.

[spyglass24]

Hi @tbiethman, Here is the error.

[tbiethman]

Thanks @spyglass24, the error indicates that you're using the same group name ('Daily') for your multiple jobs here. The groups need to be unique across the different cypress run --record executions in within the same bitbucket pipeline. Your first screenshot of your various run commands doesn't show which --group values you're using, though, so let me know if my assumption here is incorrect.

Based on your tag usage, it could make sense to duplicate the tag as the group name to guarantee this uniqueness. For reference, here are our docs around run grouping, which includes some typical grouping strategies.

[nicooprat]

In case it helps someone else: I had to disable the Brave browser shield protection to be able to login into Cypress Cloud & link it to the local app:

Without this, I could login but it never went back to the app.

[Deop]

Agreed with what @lucasdonato mentioned, I'd like to have an option to debug previous run. We do run multiple different suites from single branch during the day, so I would like to be able to choose specific run to debug.

[pstakoun]

@Deop thanks for your feedback! We're actively working on supporting this workflow. Out of curiosity, can you elaborate a bit on the workflow that involves multiple runs at different times? Is the reason you must have separate runs because they happen too far apart and therefore cannot be grouped via the --group parameter?

[Deop]

Yeah we have different runs during the day. We utilize --group for runs that supposed to be the part of one suite. But we could do full regression and couple of smoke suites runs (for example), so would be nice to be able to switch between them in the in-app debugger

[lucasdonato]

It would be nice :)

[spyglass24]

+++

[masomaananna]

If login process in google is possible by script like->

describe('Login to Gmail', () => {
it('Should be able to login', () => {
cy.visit('https://gmail.com')
cy.get('input[type="email"]').type('cypress@gmail.com')
cy.get('#identifierNext').click()
cy.get('input[type="password"]').type('Cypress@Cy#')
cy.get('#passwordNext').click()
cy.contains('Inbox').should('be.visible')
})

Will be helpful for us. In selenium it is easy to login in google through the script. But in cypress it is difficult to do google authentication, which is difficult to understand.

[pstakoun]

@masomaananna we recommend logging in with Google programmatically, due to their heavy use of automation detection. Those docs can be found here: https://docs.cypress.io/guides/end-to-end-testing/google-authentication. You can also find patterns for logging in via the UI here: https://docs.cypress.io/guides/end-to-end-testing/social-authentication, but we do not recommend this approach for use in CI.

[Phonesis]

Keen to use this feature but it doesn't appear to work with the Cypress CircleCI Orb where you have multiple parallel stages with unique group names. For example, circleCI config using the official Cypress Orb:

   - cypress/run:
          name: cypress-tests-chrome
          executor: with-chrome-firefox-edge
          browser: chrome
          spec: cypress/e2e/specs/ui/**/*
          command-prefix: npx @percy/cli@latest exec --
          start: npm run start:test
          wait-on: '-c .circleci/wait-on-config.json http://localhost:3000'
          record: true
          parallel: true
          parallelism: 9
          group: 'Chrome tests'

   - cypress/run:
          name: cypress-tests-firefox
          executor: with-chrome-firefox-edge
          browser: firefox
          spec: cypress/e2e/specs/ui/**/*
          command-prefix: npx @percy/cli@latest exec --
          start: npm run start:test
          wait-on: '-c .circleci/wait-on-config.json http://localhost:3000'
          record: true
          parallel: true
          parallelism: 9
          group: 'Firefox tests'

Say 1 test fails for Chrome, but Firefox all passes and finishes last, you will only see the last result in the debug tab and it will show that all tests passed (because it will be the firefox group)

Is this a known issue? Any fix?

[tbiethman]

Hi @Phonesis, thank you for posting your findings. Generally, Cypress knows which recordings to group into a single run based on an environment variable named CI_BUILD_ID. While you can provide this value yourself, we infer a reasonable value from your CI environment automatically; for CircleCI, that value comes from the CIRCLE_WORKFLOW_ID env that is shared across all jobs within a given workflow. Based on how your config looks here, I would expect your grouping to behave correctly, as both cypress/run jobs should share the same CIRCLE_WORKFLOW_ID value.

Can you confirm whether you see these executions grouped into a single run within the Cypress Cloud itself? Or do you see separate runs generated for each job here, one for each group?

You can reference the CI_BUILD_ID value used for any given run by looking at a run's metadata dropdown. Example:

[Phonesis]

Hi @tbiethman so think got to bottom of this based on your reply. Our repo was using a unique CI_BUILD_ID for each group. This seemed logical but guess caused problems with the way the debug tab combines the groups?

We were using this:

          ci-build-id: 'testing-commit-${CIRCLE_SHA1}-${CIRCLE_BUILD_NUM}'

I think CIRCLE_BUILD_NUM is always unique.

I amended it to this and it seems to work as expected now (a failure in any group will be shown regardless of order of run):

          ci-build-id: 'testing-commit-${CIRCLE_SHA1}'

[tbiethman]

I'm glad you got it tracked down. CIRCLE_BUILD_NUM does seem to be unique for each job, based on CircleCI's docs.

If you find yourself recording multiple runs for the same commit, due to re-running failed jobs or as part of a nightly build, I'd recommend using the CIRCLE_WORKFLOW_ID (or just falling back to the default) to guarantee more uniqueness in the build id. But if your new implementation using just the commit sha works for you, I say roll with it.

[Phonesis]

Thanks, CIRCLE_WORKFLOW_ID seems to do the trick for us

[Phonesis]

Further update: we actually need to use a unique ci-build-id for each group for reasons linked to manually triggering a re-run of a pipeline.

So I think this debug feature need some additional work to allow us to see the history of multiple groups of tests per a branch run. At the moment, it only shows the last one to complete.

[pstakoun]

@Phonesis we are indeed working on enabling navigation between runs for the Debug page. I am curious to hear a bit more about your use case for having a unique ci-build-id for each group, and why you use manual re-runs over Cypress test retries where applicable.

[Phonesis]

Sure, we use a unique id because this is the only way we can trigger Cypress to re-run tests in CircleCI when we use the re-run / retry failed steps feature of CircleCI.

If we do not include CIRCLE_BUILD_NUM (which is always unique) in the ID then I think Cypress determines nothing needs to be re-run because the commit ID is identical / no new code has been pushed to the branch.

This is not the same as the in-built Cypress retry feature. It is specific to CircleCI.

[Phonesis]

As mentioned above, looks like the new is to just leave it as the default and use CIRCLE_WORKFLOW_ID

[Deop]

Hey, it's looks good and thanks for adding ability to select previous runs. One more thing I'd find useful is to add button to rerun full .spec next to 'Run failures' button. Usually, when debugging tests we rerun full spec file.

[warrensplayer]

@Deop Thanks for the suggestion. For now, you can click the "Run Failures" button and then remove the "debug" filter by clicking the X in the filter. That will run the entire spec.