Cypress generates superfluous screenshots for passing tests too, after afterEach() hook once failed but retried.

[tzolnai]

This issue is related to the newer test retries feature. In general, when afterEach() hook fails once all other test cases of the test file are just skipped. However when we have retries enabled (retries : 1), then cypress tries to run the test case again even if it failed in the afterEach() hook. I think somehow the afterEach() hook's failure state stuck, because after the test passes for the second try and so the test goes on with the remaining test cases, cypress creates a screenshot for all remaining tests even if they are not failing. Note that I'm speaking about the screenshot made for the test failures.

Current behavior

Superfluous screenshots are made every time afterEach() hook is called. These screenshots have the 'failed' string in their name, suggesting that a failure happened, however the tests passed.

Desired behavior

Cypress should not generate failure screenshots, when nothing is failing.

Test code to reproduce

describe('Superfluous screenshots.', {retries : 1}, function() {
	afterEach(function() {
		Cypress.env('SOMEVARIABLE', 'X');
		expect(this.currentTest.state).not.to.be.equal('failed');
	});

	it('Failing test which passes for the second time.', function() {
		expect(Cypress.env('SOMEVARIABLE')).to.be.equal('X');
	});

	it('Passing test 1.', function() {
		expect(true).to.be.true;
	});

	it('Passing test 2.', function() {
		expect(false).to.be.false;
	});
});

I used an environment variable to make the first test case and afterEach() hook to fail for the first time and then they pass for the second time, we run the test.

I see the following output when I run this in headless mode:

  Running:  writer/test_spec.js                                                             (1 of 1)


  Superfluous screenshots.
    (Attempt 1 of 2) Failing test which passes for the second time.
    (Attempt 1 of 2) Failing test which passes for the second time.
    ✓ Failing test which passes for the second time. (178ms)
    ✓ Failing test which passes for the second time. (196ms)
    ✓ Passing test 1. (115ms)
    ✓ Passing test 2. (113ms)


  4 passing (4s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        3                                                                                │
  │ Passing:      3                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  6                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     4 seconds                                                                        │
  │ Spec Ran:     writer/test_spec.js                                                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  /home/zolnai/libreoffice/online/cypress_test/cypress/screenshots/writer/test_spe     (1280x720)
     c.js/Superfluous screenshots -- Failing test which passes for the second time (f               
     ailed).png                                                                                     
  -  /home/zolnai/libreoffice/online/cypress_test/cypress/screenshots/writer/test_spe     (1280x720)
     c.js/Superfluous screenshots -- Failing test which passes for the second time --               
      after each hook (failed).png                                                                  
  -  /home/zolnai/libreoffice/online/cypress_test/cypress/screenshots/writer/test_spe     (1280x720)
     c.js/Superfluous screenshots -- Failing test which passes for the second time --               
      after each hook (failed) (attempt 2).png                                                      
  -  /home/zolnai/libreoffice/online/cypress_test/cypress/screenshots/writer/test_spe     (1280x720)
     c.js/Superfluous screenshots -- Failing test which passes for the second time --               
      after each hook (failed) (attempt 2) (1).png                                                  
  -  /home/zolnai/libreoffice/online/cypress_test/cypress/screenshots/writer/test_spe     (1280x720)
     c.js/Superfluous screenshots -- Passing test 1 -- after each hook (failed).png                 
  -  /home/zolnai/libreoffice/online/cypress_test/cypress/screenshots/writer/test_spe     (1280x720)
     c.js/Superfluous screenshots -- Passing test 2 -- after each hook (failed).png                 


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  writer/test_spec.js                      00:04        3        3        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:04        3        3        -        -        - 

Note that the second and third test cases pass without any failure, but screenshots were made for them too. I checked the screenshots and they do not show any failure, everything is green.

My expectation here is two screenshots: one for the failure in the test case, the second one is for the failure in afterEach hook. However, we get six screenshots here. Four is about the first test case and two for the other two test cases, which have never failed.

Versions

cypress version: 5.5.0
browser: chromium 78.0.3904.108
OS: openSUSE Leap 15.0

[jennifer-shehane]

Yeah, this is definitely unintended.

Reproducible example

describe('Superfluous screenshots.', {retries: 1}, function () {
	afterEach(function () {
		Cypress.env('SOMEVARIABLE', 'X')
		expect(this.currentTest.state).not.to.equal('failed')
	})

	it('Failing test which passes for the second time.', () => {
		expect(Cypress.env('SOMEVARIABLE')).to.equal('X')
	})

	it('Passing test 1.', () => {
		expect(true).to.be.true
	})

	it('Passing test 2.', () => {
		expect(false).to.be.false
	})
})

[cypress-bot]

The code for this is done in cypress-io/cypress#9558, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 6.3.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v6.3.0, please open a new issue.