Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run-ct returns with exit code 0 when tests fail - breaking CI detection of failed tests #17752

Closed
cellog opened this issue Aug 16, 2021 · 31 comments · Fixed by #17894
Closed

Comments

@cellog
Copy link

cellog commented Aug 16, 2021

Current behavior

This is a pretty big bug. All CI jobs running with run-ct will report success even if tests fail due to this bug.

In this screenshot, you are seeing the output of yarn cypress:component:cli --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx which runs this command:

    "cypress:component:cli": "cypress run-ct --project integration-tests",

Screen Shot 2021-08-16 at 3 12 44 PM

Note that at the bottom, we see Done in 69.93s but no error. This Done is reported by yarn. When a test fails, it should look more like this (I added expect(false).toBe(true) to a jest test to simulate a failure):

Screen Shot 2021-08-16 at 3 18 30 PM

cypress.json:

{
  "baseUrl": "http://localhost:6009",
  "retries": {
    "runMode": 2,
    "openMode": 0
  },
  "integrationFolder": "../packages",
  "testFiles": "**/integration_tests/*.test.tsx",
  "component": {
    "componentFolder": "../packages",
    "testFiles": "**/component_tests/*.test.tsx"
  }
}

Desired behavior

when a test fails, the shell error code should be non-zero. cypress run does this correctly.

Test code to reproduce

  1. write a failing component test
  2. run it with cypress run-ct
  3. watch to see that the exit code is 0

Cypress Version

8.3.0

Other

tested with 7.6.0 as well, upgraded to be sure it was not already fixed.

@cellog
Copy link
Author

cellog commented Aug 18, 2021

hello Cypress! This is a critical failure at the most core level of the product: the test runner reports success when tests fail. Is there anyone interested in investigating this? Is there a workaround?

@cellog
Copy link
Author

cellog commented Aug 23, 2021

tagging @JessicaSachs because you responded to a component test 5 days ago as well, hope you can route it to the right person!

@DamienCassou
Copy link

I have exactly the same problem. My CI reported success for a few days even though we broke the tests.

@DamienCassou
Copy link

DamienCassou commented Aug 25, 2021

The component runner API has the same problem: the promise always resolves instead of rejecting.

@DamienCassou
Copy link

The component runner API promise always resolves with

{
  status: 'failed',
  failures: 0,
  message: 'Could not find Cypress test run results'
}

regardless of the success or failure of tests.

Here is an example with only passing tests:


  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      2                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 second                                                                         │
  │ Spec Ran:     accordionWidget_spec.js                                                          │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

{
  status: 'failed',
  failures: 0,
  message: 'Could not find Cypress test run results'
}

And another one with failing tests:


  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  3                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     accordionWidget_spec.js                                                          │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  .../cypress/screenshots/accordionWidget_spec.js/An uncaught error was detected outside of               
      a test (failed).png                                                    


  (Video) [...]        

{
  status: 'failed',
  failures: 0,
  message: 'Could not find Cypress test run results'
}

The problem is the same if an assertion within the test fails.

Here is how I use the API:

async function main() {
	let result = await cypress.run({
		testingType: "component",
		spec: "cypress/component/**/*_spec.js",
		reporter: "junit",
		reporterOptions: "mochaFile=tests-component.[hash].xml,toConsole=true",
	});

	console.log(result);

	if (result && result.status === "failed") {
		throw new Error(result.message);
	}
}

Stack:

  • cypress 8.3.0
  • @cypress/webpack-dev-server: 1.4.0

@lmiller1990
Copy link
Contributor

@cellog your post says

"cypress:cli": "cypress run --project integration-tests",

Should this be "cypress:cli": "cypress run-ct --project integration-tests"? Assuming yes, since your issue is about run-ct.

@jennifer-shehane
Copy link
Member

Can you run the DEBUG logs for the cypress:cli during cypress run? This will print the exact exit code that Cypress is issuing at the end of the tests. Please provide the entire set of logs. A reproducible example would be helpful also.

DEBUG=cypress:cli* cypress run
DEBUG=cypress:cli* cypress run-ct

@cellog
Copy link
Author

cellog commented Aug 25, 2021

excellent, thank you, I'll report back shortly, probably a few hours once these meetings are dun

@cellog
Copy link
Author

cellog commented Aug 25, 2021

@cellog your post says

"cypress:cli": "cypress run --project integration-tests",

Should this be "cypress:cli": "cypress run-ct --project integration-tests"? Assuming yes, since your issue is about run-ct.

yes thank you for the catch, I copy/pasted the wrong line. Updated in the original comment

@lmiller1990
Copy link
Contributor

lmiller1990 commented Aug 25, 2021

@cellog thanks!

@DamienCassou I think this is a separate issue, seems there is a problem with webpack when using run-ct and the Cypress module API. I was able to reproduce with the module API (but not when running with yarn cypress run-ct. I reproduced it here: lmiller1990/cypress-react-webpack-5-example#1. I am tracking it here: #17885. I'll use this issue to track the original problem (could be the same root cause).

I'll look into both of these today.

@wesgro
Copy link

wesgro commented Aug 25, 2021

I'm also seeing this using Vite as the dev server. Cypress always returns 0 even with failing tests.

8.2.0 doesn't seem to have this bug

@cellog
Copy link
Author

cellog commented Aug 25, 2021

there was surprisingly little output:

➜  beacon-labs git:(renovate/major-all-other-braze) DEBUG=cypress:cli:* yarn cypress:component:cli --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx
yarn run v1.19.1
$ COMPONENT_TESTS=1 cypress run-ct --project integration-tests --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx
  cypress:cli:cli cli starts with arguments ["/Users/gregory.beaver/.nvm/versions/node/v14.15.4/bin/node","/Users/gregory.beaver/Documents/GitHub/beacon-labs/node_modules/.bin/cypress","run-ct","--project","integration-tests","--spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx"] +0ms
  cypress:cli:cli program parsing arguments +1ms
  cypress:cli:cli running Cypress run-ct +1ms
  cypress:cli:run processing run options { project: 'integration-tests', spec: 'packages/SearchInput/src/component_tests/ClearButton.test.tsx', testingType: 'component', key: null, reporter: null, reporterOptions: null } +0ms
  cypress:cli:run --key is not set, looking up environment variable CYPRESS_RECORD_KEY +2ms
  cypress:cli:run run to spawn.start args ["--run-project","integration-tests","--spec","packages/SearchInput/src/component_tests/ClearButton.test.tsx","--testing-type","component"] +0ms

then webpack compilation, then no other debug output at all, just this:

ℹ 「wds」: Project is running at http://localhost:58356/webpack-dev-server/
ℹ 「wds」: webpack output is served from /__cypress/src
ℹ 「wds」: Content not from webpack is served from /Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    7.6.0                                                                              │
  │ Browser:    Electron 89 (headless)                                                             │
  │ Specs:      1 found (SearchInput/src/component_tests/ClearButton.test.tsx)                     │
  │ Searched:   ../packages/SearchInput/src/component_tests/ClearButton.test.tsx                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  SearchInput/src/component_tests/ClearButton.test.tsx                            (1 of 1)
ℹ 「wdm」: wait until bundle finished: /__cypress/src/index.html
ℹ 「wdm」: Hash: 98ef4a6932450edfaf06
Version: webpack 4.46.0
Time: 22174ms
Built at: 08/25/2021 1:15:45 PM
                                                                                                           Asset       Size                                                                                                         Chunks             Chunk Names
                                                                                                            0.js   18.8 KiB                                                                                                              0  [emitted]  
                                                                                                            1.js   1.18 MiB                                                                                                              1  [emitted]  
                                                                                                            2.js   4.84 KiB                                                                                                              2  [emitted]  
                                                                                           fonts/Sailec-Bold.ttf   77.2 KiB                                                                                                                 [emitted]  
                                                                                          fonts/Sailec-Bold.woff   35.3 KiB                                                                                                                 [emitted]  
                                                                                         fonts/Sailec-Bold.woff2   28.2 KiB                                                                                                                 [emitted]  
                                                                                     fonts/Sailec-BoldItalic.ttf   80.3 KiB                                                                                                                 [emitted]  
                                                                                    fonts/Sailec-BoldItalic.woff   37.6 KiB                                                                                                                 [emitted]  
                                                                                   fonts/Sailec-BoldItalic.woff2   29.9 KiB                                                                                                                 [emitted]  
                                                                                        fonts/Sailec-Regular.ttf   79.2 KiB                                                                                                                 [emitted]  
                                                                                       fonts/Sailec-Regular.woff   35.9 KiB                                                                                                                 [emitted]  
                                                                                      fonts/Sailec-Regular.woff2     29 KiB                                                                                                                 [emitted]  
                                                                                  fonts/Sailec-RegularItalic.ttf   81.6 KiB                                                                                                                 [emitted]  
                                                                                 fonts/Sailec-RegularItalic.woff   37.9 KiB                                                                                                                 [emitted]  
                                                                                fonts/Sailec-RegularItalic.woff2   30.2 KiB                                                                                                                 [emitted]  
                                                                                                      index.html  393 bytes                                                                                                                 [emitted]  
                                                                                                         main.js   12.3 KiB                                                                                                           main  [emitted]  main
                                                                                                       spec-0.js   44.9 KiB                                                                                                         spec-0  [emitted]  spec-0
spec-0~spec-10~spec-11~spec-12~spec-21~spec-22~spec-23~spec-24~spec-25~spec-26~spec-27~spec-3~spec-4~e7700a5b.js    113 KiB  spec-0~spec-10~spec-11~spec-12~spec-21~spec-22~spec-23~spec-24~spec-25~spec-26~spec-27~spec-3~spec-4~e7700a5b  [emitted]  spec-0~spec-10~spec-11~spec-12~spec-21~spec-22~spec-23~spec-24~spec-25~spec-26~spec-27~spec-3~spec-4~e7700a5b
                                                                                               spec-0~spec-11.js     40 KiB                                                                                                 spec-0~spec-11  [emitted]  spec-0~spec-11
spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-19~spec-2~0fcaf07b.js   32.2 KiB  spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-19~spec-2~0fcaf07b  [emitted]  spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-19~spec-2~0fcaf07b
                                                                                  spec-0~spec-1~spec-4~spec-5.js   55.6 KiB                                                                                    spec-0~spec-1~spec-4~spec-5  [emitted]  spec-0~spec-1~spec-4~spec-5
                                                                                         spec-0~spec-4~spec-5.js   53.7 KiB                                                                                           spec-0~spec-4~spec-5  [emitted]  spec-0~spec-4~spec-5
                                                                                                spec-0~spec-5.js   92.5 KiB                                                                                                  spec-0~spec-5  [emitted]  spec-0~spec-5
                                                                                                       spec-1.js   18.7 KiB                                                                                                         spec-1  [emitted]  spec-1
                                                                                                      spec-10.js   18.4 KiB                                                                                                        spec-10  [emitted]  spec-10
                                                                                               spec-10~spec-3.js   18.7 KiB                                                                                                 spec-10~spec-3  [emitted]  spec-10~spec-3
                                                                                                      spec-11.js   34.4 KiB                                                                                                        spec-11  [emitted]  spec-11
                                                                                                      spec-12.js   34.1 KiB                                                                                                        spec-12  [emitted]  spec-12
                                                                                                      spec-13.js   7.81 KiB                                                                                                        spec-13  [emitted]  spec-13
                                                                                      spec-13~spec-14~spec-15.js    116 KiB                                                                                        spec-13~spec-14~spec-15  [emitted]  spec-13~spec-14~spec-15
                                                                              spec-13~spec-14~spec-15~spec-16.js   34.2 KiB                                                                                spec-13~spec-14~spec-15~spec-16  [emitted]  spec-13~spec-14~spec-15~spec-16
                                                                      spec-13~spec-14~spec-15~spec-16~spec-20.js    336 KiB                                                                        spec-13~spec-14~spec-15~spec-16~spec-20  [emitted]  spec-13~spec-14~spec-15~spec-16~spec-20
                                                                              spec-13~spec-14~spec-15~spec-20.js   25.6 KiB                                                                                spec-13~spec-14~spec-15~spec-20  [emitted]  spec-13~spec-14~spec-15~spec-20
                                                                                              spec-13~spec-15.js   26.4 KiB                                                                                                spec-13~spec-15  [emitted]  spec-13~spec-15
                                                                                                      spec-14.js   16.4 KiB                                                                                                        spec-14  [emitted]  spec-14
                                                                                                      spec-15.js   17.5 KiB                                                                                                        spec-15  [emitted]  spec-15
                                                                                                      spec-16.js   25.4 KiB                                                                                                        spec-16  [emitted]  spec-16
                                                                              spec-16~spec-17~spec-18~spec-19.js   92.3 KiB                                                                                spec-16~spec-17~spec-18~spec-19  [emitted]  spec-16~spec-17~spec-18~spec-19
                                                                                                      spec-17.js   35.4 KiB                                                                                                        spec-17  [emitted]  spec-17
                                                                                                      spec-18.js   32.5 KiB                                                                                                        spec-18  [emitted]  spec-18
                                                                                                      spec-19.js     25 KiB                                                                                                        spec-19  [emitted]  spec-19
                                                                                                       spec-2.js   29.7 KiB                                                                                                         spec-2  [emitted]  spec-2
                                                                                                      spec-20.js   30.1 KiB                                                                                                        spec-20  [emitted]  spec-20
                                                                                                      spec-21.js   19.9 KiB                                                                                                        spec-21  [emitted]  spec-21
                                                                                                      spec-22.js   20.4 KiB                                                                                                        spec-22  [emitted]  spec-22
                                                                                                      spec-23.js    334 KiB                                                                                                        spec-23  [emitted]  spec-23
                                                                                              spec-23~spec-26.js    170 KiB                                                                                                spec-23~spec-26  [emitted]  spec-23~spec-26
                                                                                                      spec-24.js   13.4 KiB                                                                                                        spec-24  [emitted]  spec-24
                                                                              spec-24~spec-25~spec-26~spec-27.js    142 KiB                                                                                spec-24~spec-25~spec-26~spec-27  [emitted]  spec-24~spec-25~spec-26~spec-27
                                                                                              spec-24~spec-26.js   25.1 KiB                                                                                                spec-24~spec-26  [emitted]  spec-24~spec-26
                                                                                                      spec-25.js    169 KiB                                                                                                        spec-25  [emitted]  spec-25
                                                                                              spec-25~spec-26.js   48.7 KiB                                                                                                spec-25~spec-26  [emitted]  spec-25~spec-26
                                                                                                      spec-26.js    584 KiB                                                                                                        spec-26  [emitted]  spec-26
                                                                                                      spec-27.js   16.6 KiB                                                                                                        spec-27  [emitted]  spec-27
                                                                                                       spec-3.js   46.8 KiB                                                                                                         spec-3  [emitted]  spec-3
                                                                                                spec-3~spec-8.js   50.8 KiB                                                                                                  spec-3~spec-8  [emitted]  spec-3~spec-8
                                                                                                       spec-4.js   8.16 KiB                                                                                                         spec-4  [emitted]  spec-4
                                                                                                       spec-5.js   18.4 KiB                                                                                                         spec-5  [emitted]  spec-5
                                                                                                       spec-6.js   15.3 KiB                                                                                                         spec-6  [emitted]  spec-6
                                                                                                       spec-7.js   46.9 KiB                                                                                                         spec-7  [emitted]  spec-7
                                                                                                       spec-8.js   13.6 KiB                                                                                                         spec-8  [emitted]  spec-8
                                                                                                       spec-9.js     37 KiB                                                                                                         spec-9  [emitted]  spec-9
                                                                                                 vendors~main.js     45 KiB                                                                                                   vendors~main  [emitted]  vendors~main
vendors~spec-0~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-2~spec-2~b4502d72.js   2.76 MiB  vendors~spec-0~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-2~spec-2~b4502d72  [emitted]  vendors~spec-0~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-2~spec-2~b4502d72
vendors~spec-0~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-20~spec-21~spec-22~spec-~76f5384f.js    295 KiB  vendors~spec-0~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-20~spec-21~spec-22~spec-~76f5384f  [emitted]  vendors~spec-0~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-20~spec-21~spec-22~spec-~76f5384f
vendors~spec-0~spec-10~spec-11~spec-12~spec-21~spec-22~spec-23~spec-24~spec-25~spec-26~spec-27~spec-~b705b335.js   2.07 MiB  vendors~spec-0~spec-10~spec-11~spec-12~spec-21~spec-22~spec-23~spec-24~spec-25~spec-26~spec-27~spec-~b705b335  [emitted]  vendors~spec-0~spec-10~spec-11~spec-12~spec-21~spec-22~spec-23~spec-24~spec-25~spec-26~spec-27~spec-~b705b335
vendors~spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-1~5c743c9e.js     28 KiB  vendors~spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-1~5c743c9e  [emitted]  vendors~spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-1~5c743c9e
vendors~spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-1~cc46fb12.js   3.44 MiB  vendors~spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-1~cc46fb12  [emitted]  vendors~spec-0~spec-1~spec-10~spec-11~spec-12~spec-13~spec-14~spec-15~spec-16~spec-17~spec-18~spec-1~cc46fb12
                                                                                               vendors~spec-1.js   20.2 KiB                                                                                                 vendors~spec-1  [emitted]  vendors~spec-1
                                                                                              vendors~spec-17.js   19.2 KiB                                                                                                vendors~spec-17  [emitted]  vendors~spec-17
                                                                                              vendors~spec-18.js   19.2 KiB                                                                                                vendors~spec-18  [emitted]  vendors~spec-18
                                                                                              vendors~spec-19.js   19.2 KiB                                                                                                vendors~spec-19  [emitted]  vendors~spec-19
                                               vendors~spec-21~spec-22~spec-23~spec-24~spec-25~spec-27~spec-8.js   3.19 MiB                                                 vendors~spec-21~spec-22~spec-23~spec-24~spec-25~spec-27~spec-8  [emitted]  vendors~spec-21~spec-22~spec-23~spec-24~spec-25~spec-27~spec-8
                                                                      vendors~spec-23~spec-24~spec-25~spec-26.js     86 KiB                                                                        vendors~spec-23~spec-24~spec-25~spec-26  [emitted]  vendors~spec-23~spec-24~spec-25~spec-26
                                                                                              vendors~spec-25.js    380 KiB                                                                                                vendors~spec-25  [emitted]  vendors~spec-25
Entrypoint main = vendors~main.js main.js
[../node_modules/@cypress/webpack-dev-server/dist/aut-runner.js] 483 bytes {vendors~main} [built]
[../node_modules/@cypress/webpack-dev-server/dist/browser.js] 89 bytes {vendors~main} [built]
[../node_modules/@cypress/webpack-dev-server/dist/loader.js!../node_modules/@cypress/webpack-dev-server/dist/browser.js] 13.1 KiB {vendors~main} [built]
[../packages/DateField/src/component_tests/DateFieldCalendarAndShortcutsPopover.test.tsx] 13.4 KiB {spec-0} [built]
[../packages/DateField/src/component_tests/DateFieldDay.test.tsx] 11.9 KiB {spec-1} [built]
[../packages/DateField/src/component_tests/DateFieldFocusRegistration.test.tsx] 2.99 KiB {spec-2} [built]
[../packages/DateField/src/component_tests/DateFieldInputRenderer.test.tsx] 8.45 KiB {spec-3} [built]
[../packages/DateField/src/component_tests/DateFieldMonth.test.tsx] 5.7 KiB {spec-4} [built]
[../packages/DateField/src/component_tests/DateFieldMonthChooser.test.tsx] 13.1 KiB {spec-5} [built]
[../packages/DateField/src/component_tests/DateFieldMonthHeading.test.tsx] 1.01 KiB {spec-6} [built]
[../packages/DateField/src/component_tests/DateFieldPrevNextButton.test.tsx] 2.49 KiB {spec-7} [built]
[../packages/DateField/src/component_tests/DateFieldRangeInputs.test.tsx] 9.35 KiB {spec-8} [built]
[../packages/DateField/src/component_tests/DateFieldShortcut.test.tsx] 2.31 KiB {spec-9} [built]
[../packages/DateField/src/component_tests/DateFieldShortcutInput.test.tsx] 7.67 KiB {spec-10} [built]
[../packages/DateField/src/component_tests/DateFieldShortcuts.test.tsx] 13.3 KiB {spec-11} [built]
    + 4485 hidden modules
Child HtmlWebpackCompiler:
                          Asset      Size               Chunks  Chunk Names
    __child-HtmlWebpackPlugin_0  4.96 KiB  HtmlWebpackPlugin_0  HtmlWebpackPlugin_0
    Entrypoint HtmlWebpackPlugin_0 = __child-HtmlWebpackPlugin_0
    [../node_modules/html-webpack-plugin/lib/loader.js!../node_modules/@cypress/webpack-dev-server/index-template.html] 561 bytes {HtmlWebpackPlugin_0} [built]
ℹ 「wdm」: Compiled successfully.


  SearchInput -> ClearButton
    ✓ should render a clear button (170ms)
    1) "after all" hook for "should render clear hover text"


  1 passing (1s)
  1 failing

  1) SearchInput -> ClearButton
       "after all" hook for "should render clear hover text":
     Error: pollBatchEnd is not a function

Because this error occurred during a `after all` hook we are skipping all of the remaining tests.

Although you have test retries enabled, we do not retry tests when `before all` or `after all` hooks fail
      at eval (webpack:///../node_modules/@applitools/eyes-cypress/src/browser/sendRequest.js?:9:17)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  1                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 second                                                                         │
  │ Spec Ran:     SearchInput/src/component_tests/ClearButton.test.tsx                             │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  /Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests/cypress/scr    (1920x1080)
     eenshots/SearchInput/src/component_tests/ClearButton.test.tsx/should render clea               
     r hover text -- after all hook (failed).png                                                    


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /Users/gregory.beaver/Documents/GitHub/beacon-labs/integrat    (4 seconds)
                          ion-tests/cypress/videos/SearchInput/src/component_tests/Cl               
                          earButton.test.tsx.mp4                                                    

✨  Done in 61.78s.

@cellog
Copy link
Author

cellog commented Aug 25, 2021

I re-ran it with cypress:* and got more output:

yarn run v1.19.1
$ COMPONENT_TESTS=1 cypress run-ct --project integration-tests --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx
  cypress:cli:cli cli starts with arguments ["/Users/gregory.beaver/.nvm/versions/node/v14.15.4/bin/node","/Users/gregory.beaver/Documents/GitHub/beacon-labs/node_modules/.bin/cypress","run-ct","--project","integration-tests","--spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx"] +0ms
  cypress:cli NODE_OPTIONS is not set +0ms
  cypress:cli:cli program parsing arguments +2ms
  cypress:cli:cli running Cypress run-ct +0ms
  cypress:cli parsed cli options { project: 'integration-tests', spec: 'packages/SearchInput/src/component_tests/ClearButton.test.tsx' } +97ms
  cypress:cli verifying Cypress app +0ms
  cypress:cli checking environment variables +0ms
  cypress:cli checking if executable exists /Users/gregory.beaver/Library/Caches/Cypress/7.6.0/Cypress.app/Contents/MacOS/Cypress +4ms
  cypress:cli Binary is executable? : true +2ms
  cypress:cli binaryDir is  /Users/gregory.beaver/Library/Caches/Cypress/7.6.0/Cypress.app +0ms
  cypress:cli Reading binary package.json from: /Users/gregory.beaver/Library/Caches/Cypress/7.6.0/Cypress.app/Contents/Resources/app/package.json +0ms
  cypress:cli Found binary version 7.6.0 installed in: /Users/gregory.beaver/Library/Caches/Cypress/7.6.0/Cypress.app +3ms
  cypress:cli { verified: true } +4ms
  cypress:cli is Verified ? true +2ms
  cypress:cli:run processing run options { project: 'integration-tests', spec: 'packages/SearchInput/src/component_tests/ClearButton.test.tsx', testingType: 'component', key: null, reporter: null, reporterOptions: null } +0ms
  cypress:cli:run --key is not set, looking up environment variable CYPRESS_RECORD_KEY +0ms
  cypress:cli:run run to spawn.start args ["--run-project","integration-tests","--spec","packages/SearchInput/src/component_tests/ClearButton.test.tsx","--testing-type","component"] +0ms
  cypress:cli needs to start own Xvfb? false +0ms
  cypress:cli spawning, should retry on display problem? false +0ms
  cypress:cli spawning Cypress with executable: /Users/gregory.beaver/Library/Caches/Cypress/7.6.0/Cypress.app/Contents/MacOS/Cypress +6ms
  cypress:cli spawn args [ '--no-sandbox', '--', '--run-project', 'integration-tests', '--spec', 'packages/SearchInput/src/component_tests/ClearButton.test.tsx', '--testing-type', 'component', '--cwd', '/Users/gregory.beaver/Documents/GitHub/beacon-labs' ] { detached: false, stdio: [ 'inherit', 'inherit', 'pipe' ] } +0ms
  cypress:cli piping child STDERR to process STDERR +6ms

and at the end of the whole thing:

  cypress:server:browsers browser process killed +3ms
  cypress:server:plugins plugin event registered? { event: 'after:run', isRegistered: true } +6s
  cypress:server:plugins execute plugin event 'after:run' Node 'v14.15.1' with args: { startedTestsAt: 2021-08-25T17:18:44.968Z, endedTestsAt: 2021-08-25T17:18:49.460Z, totalDuration: 4492, totalSuites: 1, totalTests: 2, totalFailed: 0, totalPassed: 2, totalPending: 0, totalSkipped: 0, runs: [ { stats: [Object], reporter: 'spec', reporterStats: [Object], hooks: [Array], tests: [Array], error: null, video: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests/cypress/videos/SearchInput/src/component_tests/ClearButton.test.tsx.mp4', spec: [Object], shouldUploadVideo: true } ], browserPath: '', browserName: 'electron', browserVersion: '89.0.4328.0', osName: 'darwin', osVersion: '20.5.0', cypressVersion: '7.6.0', runUrl: undefined, config: { eyesIsDisabled: false, eyesBrowser: '[{"width":1280,"height":720,"name":"chrome"}]', eyesFailCypressOnDiff: true, eyesDisableBrowserFetching: false, eyesLegacyHooks: true, eyesTestConcurrency: 20, eyesPort: 59156, configFile: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests/cypress.json', env: { codeCoverageTasksRegistered: true }, testingType: 'component', version: '7.6.0', baseUrl: 'http://localhost:59165', retries: { runMode: 2, openMode: 0 }, integrationFolder: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/packages', testFiles: '**/component_tests/*.test.tsx', component: { componentFolder: '../packages', testFiles: '**/component_tests/*.test.tsx' }, componentFolder: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/packages', projectRoot: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests', projectName: 'integration-tests', rawJson: { baseUrl: 'http://localhost:6009', retries: [Object], integrationFolder: '../packages', testFiles: '**/component_tests/*.test.tsx', component: [Object], componentFolder: '../packages', envFile: {}, projectRoot: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests', projectName: 'integration-tests' }, morgan: false, isTextTerminal: true, socketId: 'tvz05', report: true, browsers: [ [Object], [Object], [Object] ], animationDistanceThreshold: 5, autoOpen: false, blockHosts: null, chromeWebSecurity: true, clientRoute: '/__/', defaultCommandTimeout: 4000, devServerPublicPathRoute: '/__cypress/src', downloadsFolder: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/int  cypress:cli child event fired { event: 'exit', code: 0, signal: null } +1m
  cypress:cli child event fired { event: 'close', code: 0, signal: null } +1ms
✨  Done in 61.55s.

@cellog
Copy link
Author

cellog commented Aug 25, 2021

oh - it looks like the tests did pass, sorry. I put a forced failure in and got:

  cypress:server:browsers browser process killed +2ms
  cypress:server:plugins plugin event registered? { event: 'after:run', isRegistered: true } +7s
  cypress:server:plugins execute plugin event 'after:run' Node 'v14.15.1' with args: { startedTestsAt: 2021-08-25T17:22:32.883Z, endedTestsAt: 2021-08-25T17:22:50.637Z, totalDuration: 17754, totalSuites: 1, totalTests: 2, totalFailed: 1, totalPassed: 1, totalPending: 0, totalSkipped: 0, runs: [ { stats: [Object], reporter: 'spec', reporterStats: [Object], hooks: [Array], tests: [Array], error: null, video: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests/cypress/videos/SearchInput/src/component_tests/ClearButton.test.tsx.mp4', spec: [Object], shouldUploadVideo: true } ], browserPath: '', browserName: 'electron', browserVersion: '89.0.4328.0', osName: 'darwin', osVersion: '20.5.0', cypressVersion: '7.6.0', runUrl: undefined, config: { eyesIsDisabled: false, eyesBrowser: '[{"width":1280,"height":720,"name":"chrome"}]', eyesFailCypressOnDiff: true, eyesDisableBrowserFetching: false, eyesLegacyHooks: true, eyesTestConcurrency: 20, eyesPort: 60170, configFile: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests/cypress.json', env: { codeCoverageTasksRegistered: true }, testingType: 'component', version: '7.6.0', baseUrl: 'http://localhost:60179', retries: { runMode: 2, openMode: 0 }, integrationFolder: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/packages', testFiles: '**/component_tests/*.test.tsx', component: { componentFolder: '../packages', testFiles: '**/component_tests/*.test.tsx' }, componentFolder: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/packages', projectRoot: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests', projectName: 'integration-tests', rawJson: { baseUrl: 'http://localhost:6009', retries: [Object], integrationFolder: '../packages', testFiles: '**/component_tests/*.test.tsx', component: [Object], componentFolder: '../packages', envFile: {}, projectRoot: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests', projectName: 'integration-tests' }, morgan: false, isTextTerminal: true, socketId: '2dhqi', report: true, browsers: [ [Object], [Object], [Object] ], animationDistanceThreshold: 5, autoOpen: false, blockHosts: null, chromeWebSecurity: true, clientRoute: '/__/', defaultCommandTimeout: 4000, devServerPublicPathRoute: '/__cypress/src', downloadsFolder: '/Users/gregory.beaver/Documents/GitHub/beacon-labs/integration-tests/cypress/downloads', e2e:   cypress:cli child event fired { event: 'exit', code: 0, signal: null } +1m
  cypress:cli child event fired { event: 'close', code: 0, signal: null } +0ms

You can see that after:run is called with test failures, but it's exiting with code: 0

@bahmutov
Copy link
Contributor

I wonder if somewhere in the release and test pipeline a tool like https://github.com/bahmutov/cypress-expect could be used to verify the failing tests and the exit code

@jennifer-shehane
Copy link
Member

Thanks @cellog, yah I can see from the debug logs that it's reporting totalFailures: 1 but firing an exit code of 0. This definitely appears to be a bug.

Is there a way you could reduce your repro to a version we can run and see the issue? I'm running ct tests with a failure, and one with a failure with retries but they are exiting with the correct exit code, so there's likely something specific to your project's setup that's causing the issue we'd love to track down. Thanks!

@cellog
Copy link
Author

cellog commented Aug 25, 2021

ok, I'll try to get something small setup, probably tomorrow since we're close to end of day

@cellog
Copy link
Author

cellog commented Aug 25, 2021

whoa - the plot thickens - I'm seeing test failures suddenly from tests that didn't fail before, like cypress was all "oh crap" and suddenly is returning exit code of 1.

@cellog
Copy link
Author

cellog commented Aug 25, 2021

ohhhh I have a theory!! These tests failed applitools snapshots. Could it be that applitools is overriding the return value with its own thing? Could the bug be in their extension?

@lmiller1990
Copy link
Contributor

lmiller1990 commented Aug 26, 2021

I can reproduce the problem using the module API. I posted it here. It could be the same problem described here.

Edit: I was able to reproduce w/o module API as well, see this run.

@lmiller1990
Copy link
Contributor

lmiller1990 commented Aug 26, 2021

Here is a more minimal reproduction that you can run locally: lmiller1990/cypress-react-template#8. Identical tests, run return exit code 1 (correct for failure), run-ct returns 0 (should be 1).

Near identical reproduction in the cypress repo does not have this problem: https://app.circleci.com/pipelines/github/cypress-io/cypress/23239/workflows/77b84ba0-27c3-4b7e-87f5-b1254839744b/jobs/857552

Something must be different about how we are running Cypress and how it runs when published on npm. I will continue debugging.

@lmiller1990
Copy link
Contributor

lmiller1990 commented Aug 26, 2021

I could not reproduce this problem with 7.6.0 as suggested above, but I can verify there was a regression between 8.2.0 and 8.3.0.

  • 8.2.0: fails, ends process w/ exit code 1: CircleCI
  • 8.3.0: fails but ends process w/ exit code 0: CircleCI

Comparison. v8.2.0...v8.3.0

@lmiller1990
Copy link
Contributor

lmiller1990 commented Aug 26, 2021

Ok, I found the problem.

v8.2.0...v8.3.0#diff-f2f892654ff3b0ae74472e3606a75f32817c5a029dbafd3112c1be0e8ab5f90bR27

Updating electron from v12 to v13 seems to have changed something around how the exit code is returned. I checked out v8.3.0 (tag), reverted the electron version, and it returns the correct exit code.

I wonder why only run-ct is impacted. I'll find out why.

Edit 1: I found that for run it enters here with code=1, but run-ct it is code=0. Getting closer.

Edit 2: As suspected, at this point totalFailed is always 0 for run-ct. This lines up with what is reported in this issue.

Edit 3: This appears to only be a problem with the electron browser. --browser=chrome gives you the correct exit code. So that would be a work-around. cypress@8.2.0 should not have this problem - but seems the OP disagrees. Need to verify if we have 1 problem or more than 1 problem.

@lmiller1990
Copy link
Contributor

lmiller1990 commented Aug 26, 2021

This should fix it: #17894. Also includes detailed breakdown.

I cannot explain how you encountered this on Cypress 7.6, though. I'll need a reproduction for that one. This fixed the bug introduced in Cypress 8.3 where run-ct does not report the correct exit code when running using the bundled electron browser.

@cypress-bot cypress-bot bot added stage: needs review The PR code is done & tested, needs review and removed stage: work in progress labels Aug 26, 2021
@cellog
Copy link
Author

cellog commented Aug 26, 2021

ohhhh I have a theory!! These tests failed applitools snapshots. Could it be that applitools is overriding the return value with its own thing? Could the bug be in their extension?

just to follow up this was PEBKAC - I was looking at our integration tests which run the e2e version of cypress, and my brain said "these are component tests". Applitools extension is not the culprit. Going to work on a simple repro

@cellog
Copy link
Author

cellog commented Aug 26, 2021

@lmiller1990 I would like to try to apply this simple patch locally, and see if it fixes the bug, as this will be a lot less effort than making a repro case. Where in god's name does that file live? I did a full text search of node_modules for .ready( and it just isn't in there as far as I can tell.

@cellog
Copy link
Author

cellog commented Aug 26, 2021

@DamienCassou
Copy link

@cellog I think what you have in node_modules is only the Cypress launcher. Cypress itself is downloaded in your home directory and cached there. Look at ~/.cache/Cypress/8.3.0/Cypress/resources/app/packages/server/lib/modes/run-ct.js/.

@cellog
Copy link
Author

cellog commented Aug 26, 2021

thanks Damien! For future reference the path in Mac OS is ~/Library/Caches/Cypress/8.3.0/Cypress.app/Resources/app<the rest from here is the same> but you helped immensely, I was finally able to verify the fix.

  • I can confirm with 8.3.0 the fix works on my end.
  • I can ALSO confirm that with 7.6.0 the fix works on my end, so thank you! We will definitely upgrade to latest Cypress once this is QAed and released, nothing holding that upgrade back on our end.

@lmiller1990 thank you!

@lmiller1990
Copy link
Contributor

Great to hear this is fixed, we will be doing a patch release soon to get this out.

If anyone is interested, we actually have pretty neat setup to test Cypress in an e2e manner. Here's the new test verifying that this bug does not regress : https://github.com/cypress-io/cypress/pull/17894/files#diff-f01be2f25ace400c3f80f0cd17786e27db5352291a9555ec4f2e5cc206a15330 which would have caught this issue (and will moving forward).

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Aug 27, 2021

Released in 8.3.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v8.3.1, please open a new issue.

@cypress-bot cypress-bot bot removed the stage: needs review The PR code is done & tested, needs review label Aug 27, 2021
@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators Aug 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
6 participants