Cypress does not clear whitelisted cookies between runs during cypress open

[dialex]

When I open Cypress and click to run all tests, nothing happens.

Current behavior:

If I run cypress run then yes, it runs all tests in the command line. But with cypress open it shows the UI and when I click run (all) test(s), no test is executed... it just hangs there

Desired behavior:

Open Cypress, click run all tests and actually run all tests.

Test code to reproduce

I don't have steps to reproduce this issue but I've tried:

• upgrading Cypress 3.7.0 to 4.0.1 ❌
• upgrading Node from 12.* to 13.* ❌
• deleting node_modules ❌
• deleting /Users/Library/Caches/Cypress/ ❌
• running the same test code, the same cypress version on a diff machine and it works ✅

So the problem seems to be my machine somehow. It was working perfectly fine until yesterday, when I tried the retry plugin, I even reported it. Then I could no longer run tests visually. Some misconfiguration? Something corrupted?

How can I reset whatever is causing Cypress to misbehave?

Versions

Cypress 4.0.1 / macOS 10.14.6 / Chrome 80

[kuceb]

@dialex I believe this is a bug with chrome 80. Can you try a previous version of chrome (chromium)? You can download it here https://chromium.cypress.io/

[jennifer-shehane]

I cannot recreate this behavior with our cypress-example-kitchensink repo in Chrome 80 on MacOS.

• Open the DevTools console, are there any errors there?
• Download Chromium 79. Does this still happen there?
• What happens if you click an individual specfile instead of 'Run All Tests' button during cypress open?
• How many tests are supposed to run total? Because we don't really recommend 'Run All Tests' for a large amount of tests due to performance reasons.
• Have you tried reducing the numTestsKeptInMemory? Does this help?

IMO, tracking down the change you made before/after this happened will probably be the clue though. Did you add some new tests or config?

[dialex]

Thanks for taking the time @jennifer-shehane 🙏
Indeed, my colleague also has Chrome 80 on MacOS and doesn't have my problem.

• ❎DevTools console doesn't have any errors
• ❎Cypress runs tests with Chromium 79 and Firefox 72 (but not Chrome 80)
• ❎If I click to run a single test file, I get the same behaviour as with "run all tests"
• ❎We have 38 tests across 15 files -- they used to run fine on my machine and they still run fine on my colleague's machine
• ❎I reduced the numTestsKeptInMemory from 50 (default) to 10 -- same behaviour

More debug:

• ❎Reinstalled Chrome -- same behaviour
• ❎Cloned the repo to a new folder, installed deps, urn all tests -- same behaviour
• ✅Cloned a diff repo that also uses Cypress (but with a lot less plugins)
  • with Cypress 3.6.1 I ran all tests targeting Chrome 80 -- and they executed!
  • then I yarn upgrade Cy to 3.8.3 and ran all tests again -- and they executed!
  • then I manually set Cy to 4.0.1 and ran all tests again -- and they executed!

So I took this working repo as a baseline, and then starting adding my code:

• added tests -- they executed
• added dependencies -- they executed
• added config/index files -- they executed
• added commands.js -- they failed to execute 🚨🚨🚨
• reverted commands.js -- they continue failing 😡
• added commands.js, one by one -- they executed... so the cause is not this file

At this point both folders are identical 🤷‍♂️

• so I go back to the repo/folder with issues, run the tests -- the problem is gone 🤔
• incredulous, I click the restart tests button -- the problem is back 😭W T actual F
• I go to the working repo, run the tests -- they fail, even though no file was changed and they were running before

I can't get my mind around this... it doesn't seem deterministic, there must be something hidden controlling this behaviour. It doesn't make any sense the same come one time runs tests and then it doesn't.

[kuceb]

@dialex is this only a problem when you use the 'run all tests' button?

[dialex]

@bkucera Here's the summary:

• It's fine if I cypress run
• It's fine if I cypress open and run all tests with Electron, Chromium or Firefox
• It's fine if I cypress open and run a specific test with Electron, Chromium or Firefox
• It doesn't work if I cypress open and run all tests with Chrome
• It doesn't work if I cypress open and run a specific test with Chrome

[jennifer-shehane]

After you see the error happening, could try and follow these instructions to Clear App Data to see if it fixes the failures?

Can you provide the code from your commands.js file?

[dialex]

IT WORKS! The solution was to delete the App Data folder.

OMG, thank you so much @jennifer-shehane 🙇

[dialex]

It's happening again 😭 So the delete App Data folder is just a hack that temporarily fixes the issue, but after a few test runs the issue returns and Cypress stops running tests.

Current behaviour:
tests don't run > delete App Data > restart Cy > click to run, tests run > click to run, tests don't run

[jennifer-shehane]

Yeah, this is a pretty weird issue we haven't seen with anyone else, but I suppose it narrows down the issue to something that is being written to the App Data.

Can you make sure to disabled all extensions in Chrome within Cypress? Sometimes extensions can do weird things.

Are there any error in the DevTools console if you open that during the run?

Other than that - it may be something related to the tests you're running. Can you narrow down the test suite to the smallest reproducible set of commands that reproduce the freezing after you clear the App Data?

[dialex]

Can you make sure to disabled all extensions in Chrome within Cypress?

Right, I don't have any extensions installed or enabled in Cypress' Chrome.

Are there any error in the DevTools console if you open that during the run?

Nothing.

Can you narrow down the test suite to the smallest reproducible set (...)

I can try, I'll need time for that. When I have an update, I'll let you know. Thanks

[dialex]

Ok, I was finally able to reproduce this issue! FYI @jennifer-shehane

I'm using this "plugin" code snippet to detect when stop the test suite execution as soon as a test of type #failfast fails.

In a nutshell: 1) afterEach test, if the test status was failed and the test was labeled as #failfast, then a cookie is set; 2) before every test file, Cypress checks that cookie, and if the value is true, it skips to the next.

My hypothesis is: for that mechanism to work, the cookie needs to be whitelisted, so that it persists between tests of the same test run. After we get one failure, the test suite is aborted. When we try to rerun it, the cookie is still there (whitelisted), the before sees its latest value (true) and immediately aborts the test run without running any test.

This is the cause of the issue:
⚠️Cypress does not clear whitelisted cookies between test runs, via UI

Here's why I think this is a bug:

• If I run the same code via CLI, the whitelisted cookie is not persisted between diff test runs. But if I run it via UI, the whitelisted cookie is persisted between test run (ie. closing the test window, clicking "Stop", clicking "Run all tests").
• Worse, if you close the Cypress process and re-open it, the cookie is still there. I think that behaviour (1) is not coherent between UI and CLI, and (2) surpasses the scope of whitelisting a cookie.

I hope this helps finding the faulty code, let me know if you need any extra data 😉

[jennifer-shehane]

I'm not able to recreate the behavior of whitelisted cookies persisting in between Stop and Rerun of tests during cypress open. They will persist and not be reset if you hit the 'rerun' button in the UI, but hitting Stop and rerunning is completely killing the browser and restarting a new one - there's no ways for cookies to persist here.

describe('tests', () => {
  before(() => {
    Cypress.Cookies.defaults({
      whitelist: 'token',
    })
  })

  it('test this', () => {
    Cypress.Cookies.debug(true)
  
    cy.visit('https://example.cypress.io/commands/cookies')
  
    cy.getCookie('token').should('not.exist')
    cy.get('#getCookie .set-a-cookie').click()
    cy.getCookie('token').should('have.property', 'value', '123ABC')
  });

  it('test cookie is still here', () => {
    Cypress.Cookies.debug(true)

    cy.visit('https://example.cypress.io/commands/cookies')

    cy.getCookie('token').should('have.property', 'value', '123ABC')
  });
})

On every 'stop' / 'rerun', the whitelisted cookie does not exist. It definitively does not save whitelisted cookies throughout killing the Cypress process and reopning.

There is some other mechanism going on in your test code / setup that must be setting the cookie in an unexpected way before you're expecting.

If you can provide a way to show this is a bug regarding some setup, please provide an example using the example.cypress.io site like shown above so that we can recreate.

[shwarcu]

Hi @jennifer-shehane I am struggling with the same issue and it's breaking our tests.

We are running Cypress tests against 2 environments. Let's name them dev1.app.com and dev2.app.com.

I have whitelisted 2 cookies by their names.
I run tests on dev1.app.com environment. Everything works fine.
Then I close Cypress completely and run it again with open command but for second environment dev2.app.com.

At the beginning of my test I am calling cy.getCookies();
and as a result, whitelisted cookies with domain dev1.app.com are still there, returned, after restarting Cypress.

The workaround I found so far is to manually delete all cookies that do not come from the current domain. Something like this:

  cy.getCookies().then(function(cookies) {
      const envDomain = Cypress.config('baseUrl').replace('https://', '');
      if (cookie.domain !== envDomain) {
        cy.clearCookie(cookie.name);
      }
    });
  });

[shwarcu]

@jennifer-shehane were you able to check this problem but when cookies belong to different domains?

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.