Cypress 13 - Module API missing skipped tests data

[debugbit]

Current behavior

Cypress 13 Module API returns bogus results that breaks our scripts. The changelog mentions a breaking change, but the related issue #23805 is different from what I am reporting here.

EDIT: confirmed - the only issue is with missing skipped stats

For example:

• totalDuration is NaN
• totalSkipped is NaN
• startedTestsAt is undefined
• endedTestsAt is undefined
• runs.stats.duration is undefined
• runs.stats.endedAt is undefined
• runs.stats.startedAt is undefined
• runs.stats.skipped is missing

{
  ~totalDuration: NaN,~
  totalSuites: 1,
  totalPending: 0,
  totalFailed: 2,
  ~totalSkipped: NaN,~
  totalPassed: 1,
  totalTests: 3,
  ~startedTestsAt: undefined,~
  ~endedTestsAt: undefined,~
  runs: [
    {
      stats: {
        ~duration: undefined~,
        ~endedAt: undefined~,
        failures: 2,
        passes: 1,
        pending: 0,
        ~startedAt: undefined~,
        suites: 1,
        tests: 3
      },
      reporter: 'spec',
      reporterStats: {
        suites: 1,
        tests: 3,
        passes: 1,
        pending: 0,
        failures: 2,
        start: '2023-08-29T17:44:14.712Z',
        end: '2023-08-29T17:44:27.024Z',
        duration: 12312
      },
      spec: {
        absolute: 'xxx',
        fileExtension: '.js',
        fileName: 'xxx',
        name: 'xxx.js',
        relative: 'cypress/e2e/xxx.js'
      },
      error: null,
      video: '/Users/debugbit/frontend/cypress/videos/xxx.js.mp4',
      shouldUploadVideo: true,
      hooks: undefined,
      tests: [
        // ...
      ]
    }
  ]
}

Desired behavior

Properly set the return value of cypress.run

Test code to reproduce

Ever invocation of cypress.run

Cypress Version

13.0.0

Node version

v16.17.1

Operating System

macOS 13.0

Debug Logs

No response

Other

Thanks a lot for your help!

[jennifer-shehane]

@debugbit Thanks for reporting so quickly. We're looking into this now.

[jennifer-shehane]

@debugbit Could you provide a reproducible example of the tests you ran? It may be specific to some way your test results are generating, so I'd like to have that info upfront if possible.

[jennifer-shehane]

@debugbit We did verify that the runs.stats.skipped is inadvertently missing. We will issue a fix for that. We aren't able to reproduce the other undefined/NaN values. Could you run the Module API in 12.17.4 and print the results of that? To verify this was not a pre-existing issue for those values?

[debugbit]

@jennifer-shehane thanks for quick response!

Here's an output from 12.17.4 for the same spec file.

{
  totalDuration: 12159,
  totalSuites: 1,
  totalPending: 0,
  totalFailed: 2,
  totalSkipped: 0,
  totalPassed: 1,
  totalTests: 3,
  startedTestsAt: '2023-08-29T18:28:25.733Z',
  endedTestsAt: '2023-08-29T18:28:37.892Z',
  runs: [
    {
      stats: {
        suites: 1,
        tests: 3,
        passes: 1,
        pending: 0,
        skipped: 0,
        failures: 2,
        wallClockStartedAt: '2023-08-29T18:28:25.733Z',
        wallClockEndedAt: '2023-08-29T18:28:37.892Z',
        wallClockDuration: 12159,
        startedAt: '2023-08-29T18:28:25.733Z',
        endedAt: '2023-08-29T18:28:37.892Z',
        duration: 12159
      },
      reporter: 'spec',
      reporterStats: {
        suites: 1,
        tests: 3,
        passes: 1,
        pending: 0,
        failures: 2,
        start: '2023-08-29T18:28:25.735Z',
        end: '2023-08-29T18:28:37.897Z',
        duration: 12162
      },
      spec: {
        fileExtension: '.js',
        baseName: 'xxx.js',
        fileName: 'xxx',
        specFileExtension: '.js',
        relativeToCommonRoot: 'xxx.js',
        specType: 'integration',
        name: 'cypress/e2e/xxx.js',
        relative: 'cypress/e2e/xxx.js',
        absolute: '/Users/debugbit/frontend/cypress/e2e/xxx.js'
      },
      error: null,
      video: '/Users/debugbit/frontend/cypress/videos/xxx.mp4',
      shouldUploadVideo: true,
      hooks: [
        // ...
      ],
      tests: [
        // ...
      ]
    }
  ],  
}

[cypress-bot]

Released in 13.1.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.1.0, please open a new issue.