Cypress restarting during test

[TheOptimist]

Current behavior:

When running a specific test in our Jenkins docker container Cypress restarts a specific test as if it the underlying file has been changed. This observation is based solely on the video, and seeing the test restart with different data generation.

The test eventually fails because a request to the API to set the scenario up returns an error - the amount of time that Cypress spends restarting this test is random and has been as high as 20 minutes. Clearly the error from the API is concerning (it's probably a bug) but it does at least force this test to not take an eternity.

Below archive file contains the full log from running Cypress in Jenkins, the video of the specific test that fails and some elements of the test code in a single file to give some context about the specific actions that are taking place.

cypress-restarting-during-test.zip

Desired behavior:

I'm expecting the test to complete, either in a pass or fail result. The continual restarting is very surprising and unexpected.

Steps to reproduce:

I understand that issues without repro steps are harder to justify spending any time on. I'm unsure how to even reproduce this - I cannot trigger this behaviour locally in either Chrome or Electron.

Versions:

Cypress 3.1.5
Docker Container: Ubuntu 18.04
Electron 59
Chrome 71

[jennifer-shehane]

Hey @TheOptimist, this is no fun 😞

Do you happen to have a baseUrl set? I know that when you do not define a baseUrl, Cypress initially visits a url with a random port, but in order to handle proxies properly, when Cypress gets to a cy.visit(), we then switch the main url to the same domain as the url in cy.visit().

That seems to be what is happening, although in your case, there is something that is causing this to loop forever, which is not what should be happening.

Some of this behavior is explained here: https://on.cypress.io/best-practices#Setting-a-global-baseUrl

We've also seen the 'reloading forever' behavior with cy.go() here: #896 Probably not related, but just wanted to reference similar behavior.

• Is there any sort of re-routing behavior coded into your app upon visiting the login/organizationlogin?
• Are you able to reproduce this locally? In Chrome? In Electron?
• When running locally, can you pop open the devtools and see if anything is strange there?
• Maybe add a debugger to your app code somewhere near the onload of this page and run in Cypress? Anything strange there?

[TheOptimist]

@jennifer-shehane No fun is a very good description.

There is a baseUrl set, but we do change it during the setup phase of the test. It's set to app.domain in the config file, and there is some interaction with that domain to create a tenant. The test then changes the baseUrl to org.domain (org changes from test to test).

  const cypressBaseUrl = Cypress.config().baseUrl;
  if (cypressBaseUrl && cypressBaseUrl.indexOf(newOrganization) === -1) {
    Cypress.config().baseUrl = cypressBaseUrl.replace(currentOrganization, newOrganization);
    currentOrganization = newOrganization;
  }

It's interesting that there is similar behaviour displayed with cy.go() - I'm not specifically using that, but it does at least open up some more avenues for deeper exploration. There is no re-routing logic at the login/organizationlogin url - there is elsewhere, and I'll pay a bit more attention to what happens in that space to see if I can work it out.

Unfortunately I'm completely unable to reproduce locally in either Chrome or Electron, with both run and open - this is the bit that's the most frustrating. I did think about creating a small repo to test whether swapping the baseUrl is causing it...but it all works locally. That said, I might still do that - might be something odd happening in Ubuntu.

[jennifer-shehane]

Can you try removing the baseUrl settings and overrides altogether and just plainly using the full url in cy.visit() to create the same behavior you have? Does this make a difference?

[TheOptimist]

Will try and do this. Somehow I've created a situation where doing the above doesn't seem to be as simple as it should be. I may need to refactor all calls to the API to now take an organization so they can build the appropriate domain (they assume the org domain has already been set). Writing that last statement makes me want to re-evaluate that idea...not sure I like the idea of assumptions in code.

I'm also creating an Ubuntu machine so I can see if I can recreate this problem in a "local" machine. It's Friday...so don't expect any miracles :)

[TheOptimist]

Ok, between various other bits and pieces, I am now calling request and visit with the full URL without changing the baseUrl configuration midway through a test. Same results unfortunately. I've also managed to get a local linux environment built to the same (as far as possible) spec as our Jenkins machines. Cannot reproduce on local environment either :)

My next steps are two-fold - we run Jenkins inside a Docker container, then connect to another container running on different hardware. There may be something strange happening in the network layer between these two machines (that said, our current Selenium based tests continue to operate fine). I'm going to bring the running of the target web server back into the 'localhost' to eliminate that possibility (and hopefully start a conversation about "CI on demand" rather than a bunch of static servers).

Continuing to push against this to see if I can resolve (and then hopefully identify a tiny repo). Really waiting for Cypress 4.0 for lots of reasons, this one included :)

[TheOptimist]

@jennifer-shehane I consider this issue "resolved" at the moment, or rather we have successfully implemented a different test execution strategy that doesn't exhibit the behaviour. We've altered our CI pipelines so the server under test runs in a local docker container on the AWS instance, rather than via ECS on another instance. This seems to have resolved the specific problem of Cypress restarting the test. Without troubleshooting too deeply (read "guessing") it's probably a redirect in our app, coupled with weird network stacks between multiple layers of docker and AWS routing.

[JimLynchCodes]

Hello, in my opinion this is a huge "bug" in Cypress.

I have a Cypress test that immediately when starting saves an empty object to a database. We were noticing that it was creating TWO objects and were completely baffled why thing was happening.

It took us LITERALLY WEEKS to figure out that cypres was restarting the tests after 3 seconds.

Regardless of whether the arbitrary "baseUrl" property is given, this restarting behavior should NEVER be the default behavior because it makes absolutely no sense.

[ItsWasteD]

This still isn't fixed for me. I have my baseUrl set and in some tests after some steps (it) it looks like the spec-file is reloaded. For me it is a problem because in one step a variable with a timestamp is set and after the spec-file is reloaded that timestamp isn't the same anymore. Therefore cy.contains(timestamp) isn't working because the file created with that timestamp in the name isn't found so the test is failing... What can I do? @jennifer-shehane

[kweiberth]

Hi @jennifer-shehane, we were experiencing this issue, and statically setting baseUrl in cypress.json solved it for us. However, we had been dynamically setting baseUrl before each test in our support/index.js file, like this:

Cypress.config('baseUrl', `${getOrigin()}/base-path`);

but this issue was still happening. Setting the baseUrl dynamically allows us to run tests against localhost or stage, configurable via command-line/env vars. Is this is a bug? Why would there be a "false initial" load if we set baseUrl via Cypress.config() but not if we set it statically in cypress.json?

Also, if possible, could you explain a little more about why there is this double load/run of the tests? I understand having to open the browser before knowing where the test will cy.visit(), but why is it running the beforeEach() logic, then starting to cy.visit(), and then restarting everything (beforeEach and test)? This is what is happening to us and it's very similar to what is happening in the screen recording of the OP.

[jennifer-shehane]

What you're all experiencing is this issue - #2777 although another manifestation of it - where you set the baseUrl dynamically.

I agree that this is not ideal and is definitely unintended behavior, although it is how Cypress was designed.

As to why this happens:

Without baseUrl set, Cypress loads main window in localhost + random port
Url address shows localhost:53927/__/#tests/integration/organizations/list_spec.coffee

As soon as it encounters a cy.visit(), Cypress then switches to the url of the main window to the url specified >in your visit. This can result in a ‘flash’ or ‘reload’ when your tests first start.
https://on.cypress.io/best-practices#Setting-a-global-baseUrl

This is currently the best workaround in this situation I believe: #2777 (comment)

[kweiberth]

Ah, okay, I understand now... we're setting baseUrl dynamically, but doing so inside support file, and thus it's executing in the browser environment, after Cypress has already opened up with localhost + random port.

I see in that issue you linked to there is a comment where the person is setting baseUrl in plugins file. Would this be a better place to set it? If we set it here, will this happen before Cypress opens localhost + random port and, thus, avoid this double load? I will also test this, but won't be able to until next week.

[jennifer-shehane]

@kweiberth I believe so. We read in the pluginsFile once before launch.

[kweiberth]

Hi @jennifer-shehane, just wanted to follow up on this to say that setting baseUrl dynamically in the plugins file, e.g.

config.baseUrl = `${getOrigin()}/base-path`;

seems to work well, i.e. there is no "double start" issue bc it seems the base URL is set and recognized by Cypress before it spawns the browser environment.

Thanks for your help and clarification!