Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP requests time out when Cypress is executed within Docker and Jenkins #1039

Closed
ckrohn opened this issue Dec 7, 2017 · 27 comments

Comments

@ckrohn
Copy link

commented Dec 7, 2017

Is this a Feature or Bug?

Bug

Current behavior:

Within our company we're planning to introduce Cypress as our main tool for testing end-to-end processes within our web applications.

First of all: Cypress looks amazing and we really wan't to give it a shot!

Unfortunately, we're having some trouble while automating the tests within our Docker- and Jenkins-based build environment.

Our build infrastructure looks like this:

For our test case, we've got a VM with RHEL 7.4 with kernel 3.10.0 that is basically only running an installation of Docker (version 17.06.2-ce).
On this machine we're currently running Jenkins 2.93 within a Docker container and one single Jenkins slave container based on the official Cypress Docker image on the same VM.

Before I continue: we've tried every single Jenkins version from 2.60 until 2.93 and all of the Cypress base images (including the Chrome integration which works smoothly since 1.1.4).

Every combination shows the same behaviour:

When we're trying to execute our Cypress tests on the before mentioned Jenkins Slave Container, everything works fine until Cypress actually calls the website under test. What happens is, that all the requests run in a timeout.

When I look into the video recorded during the test run within the CI setup, I can see that Cypress calls the correct website but loads indefinitely. I've tried to increase all the configurable timeouts to ridiculous values with no effect.

HTTP requests sent manually from within the Docker container work flawlessly in no time and a similar setup on different local machines works perfectly too.

What might be important: We're working within a secure environment where we have to open every single communication way within the firewall. I suspect, that Cypress might possibly do something when requesting a website that a simple wget or curl might not and that we simply have to configure our firewall appropriately.

The website we're trying to test runs on a different machine in a different network but the communication via 80/TCP and 443/TCP has been opened (as our manual tests from within the container prove).

Is there anything special we have to consider within our firewall or network configuration?

Desired behavior:

Requests are executed correctly and don't run in a timeout.

How to reproduce:

Actually I'm not sure how to reproduce this behaviour because everything works on our local machines.

Test code:

CYPRESS_BINARY_VERSION=/home/person/cypress.zip npm install --no-progress --verbose
DEBUG=cypress:* $(npm bin)/cypress run --env HOST=dev --record --key our-key

Additional Info (images, stack traces, etc)

CypressError: Timed out after waiting '60000ms' for your remote page to load.

Your page did not fire its 'load' event within '60000ms'.

You can try increasing the 'pageLoadTimeout' value in 'cypress.json' to wait longer.

Browsers will not fire the 'load' event until all stylesheets and scripts are done downloading.

When this 'load' event occurs, Cypress will continue running commands.

Because this error occurred during a 'before all' hook we are skipping the remaining tests in the current suite: 'Elastic Search'
    at Object.cypressErr (http://-G4/__cypress/runner/cypress_runner.js:65873:11)
    at Object.throwErr (http://our.test.website/__cypress/runner/cypress_runner.js:65838:18)
    at Object.throwErrByPath (http://our.test.website/__cypress/runner/cypress_runner.js:65865:17)
    at timedOutWaitingForPageLoad (http://our.test.website/__cypress/runner/cypress_runner.js:56915:17)
    at http://our.test.website/__cypress/runner/cypress_runner.js:57516:18
    at tryCatcher (http://our.test.website/__cypress/runner/cypress_runner.js:6268:23)
    at http://our.test.website/__cypress/runner/cypress_runner.js:1585:41
    at tryCatcher (http://our.test.website/__cypress/runner/cypress_runner.js:6268:23)
    at Promise._settlePromiseFromHandler (http://our.test.website/__cypress/runner/cypress_runner.js:4290:31)
    at Promise._settlePromise (http://our.test.website/__cypress/runner/cypress_runner.js:4347:18)
    at Promise._settlePromise0 (http://our.test.website/__cypress/runner/cypress_runner.js:4392:10)
    at Promise._settlePromises (http://our.test.website/__cypress/runner/cypress_runner.js:4467:18)
    at Async._drainQueue (http://our.test.website/__cypress/runner/cypress_runner.js:1200:16)
    at Async._drainQueues (http://our.test.website/__cypress/runner/cypress_runner.js:1210:10)
    at Async.drainQueues (http://our.test.website/__cypress/runner/cypress_runner.js:1084:14)]]></failure>
    </testcase>
    <testcase name="Elastic Search &quot;after all&quot; hook for &quot;with map&quot;" time="60.079" classname="&quot;after all&quot; hook for &quot;with map&quot;">
      <failure><![CDATA[Error: Cypress command timeout of '4000ms' exceeded.

Because this error occurred during a 'after all' hook we are skipping the remaining tests in the current suite: 'Elastic Search'
    at http://our.real.domain/__cypress/runner/cypress_runner.js:64279:25]]></failure>
  • Operating System: RHEL 7.4 (3.10.0)
  • Cypress Version: 1.1.3, 1.1.4
  • Browser Version: Electron, Chrome 62
@brian-mann

This comment has been minimized.

Copy link
Member

commented Dec 7, 2017

Cypress is proxying all of the requests that the browser makes.

By chance does your environment also require a proxy? You mentioned firewall, but is it possible that a proxy is also part of this setup?

Can you see anything via cy.visit? In other words is the page partially loading, but never completing? Or does it just sit at a blank white screen?

Also can you run a single test using cy.request instead of a cy.visit to see if those go out correctly?

They actually use different implementations under the hood and that would be valuable to know.

There was an issue opened a couple days ago specifically here: #1013 (comment)

They refer to requests indefinitely timing out (but I haven't actually looked at this yet) Perhaps this is a related problem.

Whatever it is I'm confident its in the proxy layer and we'll be able to sort it out. Reproducing your environment would be nice, but I'm not sure if its feasible. Perhaps there is some docker configuration you can share that reproduces it.

The only other thing I can offer is direct support from our team itself to work with you 1:1 to get it resolved.

@ckrohn

This comment has been minimized.

Copy link
Author

commented Dec 7, 2017

Thanks for the blazingly fast reply!

I'm pretty sure our environment doesn't require a proxy but I'll talk to our IT guys tomorrow to clarify this.

When I'm using cy.visit, the page just sits at a blank white screen and basically loads as long as the configured pageLoadTimeout.

Funnily enough calling the application via cy.request results in an HTTP 200 within the expected response time, what makes me pretty confident that we'll be able fix this somehow.

If you already have some suggestions based on my previous statement, let me know.

If you need further information about our docker and network configuration anyways, I'll try to put it together tomorrow and attach it to this issue.

PS: The issue has been labeled with CI: travis but I think it should be CI: jenkins :)

@ggauravr

This comment has been minimized.

Copy link

commented Feb 13, 2018

@brian-mann @christopher-krohn any update on this guys? I'm seeing a similar issue, but with Docker and TeamCity though. Sounds like if it's something with the Docker or Cypress config, might fix my issue as well.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 13, 2018

We actually may have discovered something causing network request delays in Chrome. It's the argument --disable-background-networking. This would only affect Chrome, not Electron.

In the newest version of Cypress 2.0.0 (pending release) we've removed this flag - but you can utilize the plugin API to slice it out of the browser args today if you'd like to see what it's like to run without it.

https://docs.cypress.io/api/plugins/browser-launch-api.html

@americos

This comment has been minimized.

Copy link

commented Mar 15, 2018

I'm having this same issue, almost exact same environment... but I'm sure I'm behind a corporate proxy.

Would be great if we could pass a proxy setting on cy.visit() or cy.request()

@andreyselitsky

This comment has been minimized.

Copy link

commented Jul 20, 2018

We are experiencing the same issue and we are on Cypress 3.0.2. Also it occurs sporadically, making our tests flaky. Any ideas what we could do here?

To give a little bit of the context - it is Angular 5 app, there are multiple .js chunks that are being loaded dynamically during the start of anapplication (default ng/webpack setup).

We managed to reproduce this issue locally and we saw very strange picture in Chrome's network tab - it was showing that request is loading for minutes (2 minutes unless ng timeout aborted it). At the same time we checked nginx/node logs and they are showing that file was served in milliseconds.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Jul 20, 2018

@andreyselitsky May be related to this other request timeout at 2 mins issue: #2055 (comment)

@andreyselitsky

This comment has been minimized.

Copy link

commented Jul 23, 2018

@jennifer-shehane it might be somehow related. We will try increasing the pageLoadTimeout to 3 minutes and see if we see anything after 2 minutes. Because right now our tests are just failing with pageLoadTimeout and it is hard to say which request was problematic.

UPDATE: So we've got the following error in cypress console when debug was enabled:

Mon, 23 Jul 2018 10:16:50 GMT cypress:server:proxy request failed { url: '*****.***/main.fefede1e40e5b31fe2f5.js', status: 500, error: 'Error: ESOCKETTIMEDOUT\n    

at ClientRequest.<anonymous>(/Users/*/Library/Caches/Cypress/3.0.2/Cypress.app/Contents/Resources/app/packages/server/node_modules/request/request.js:812:19)\n    
at Object.onceWrapper (events.js:314:30)\n    
at emitNone (events.js:105:13)\n    
at ClientRequest.emit (events.js:207:7)\n    
at TLSSocket.emitTimeout (_http_client.js:722:34)\n    
at Object.onceWrapper (events.js:314:30)\n    
at emitNone (events.js:105:13)\n    
at TLSSocket.emit (events.js:207:7)\n    
at TLSSocket.Socket._onTimeout (net.js:402:8)\n    
at ontimeout (timers.js:469:11)\n    
at tryOnTimeout (timers.js:304:5)\n    
at Timer.listOnTimeout (timers.js:264:5)\n' }

Any ideas?

@mattvb91

This comment has been minimized.

Copy link

commented Aug 27, 2018

👍 for visibility. Same issue here. No proxy required in our env. v3.1.0

@darenyong

This comment has been minimized.

Copy link

commented Sep 5, 2018

I'm also having a similar issue. Same error displayed, the docker image works fine on my mac, but when I run the container in the corporate lab (where Jenkins will ssh into), the page load timeout error occurs when running cypress after the node app has been started.

Note: If I map the container port 80 to host 8080, and hit the http://lab.machine:8080, I can load and view the node app just fine.

@darenyong

This comment has been minimized.

Copy link

commented Sep 5, 2018

I have tracked down my docker cypress proxy issue to my CSS.

To add some detail to my situation, I have defined the HTTP_PROXY, HTTPS_PROXY and NO_PROXY, by passing -e parameter to docker run.

When I shell into the container, and execute the following command, the proxy settings appear to work and the request succeeds.

curl "https://fonts.googleapis.com/css?family=Source%20Sans%20Pro:400,700,400italic,700italic&subset=latin"

When I run Cypress test to cy.visit() my node app, it loads the index.html, but appears to hang inside Electron waiting for a response for loading the google fonts above.

If I remove loading the CSS in my node app, then Cypress works fine, the test runs to completion, AND a video of the test is successfully uploaded to the Cypress Dashboard in the cloud (which indicates Cypress IS using the proxy for the video upload)

However, when running the actual test, Cypress appears to not use the corporate proxy and the request to fonts.googleapis.com will hang.

@JamesRamm

This comment has been minimized.

Copy link

commented Sep 19, 2018

I have the same problem, but simply running docker on localhost.

In fact, I can launch all my services with docker and then run cypress on the host (not in docker) and all my tests will past.

But if I launch cypress in a docker container, I get random timeout errors. This is running docker on localhost.

@andreyselitsky

This comment has been minimized.

Copy link

commented Oct 10, 2018

Folks, we managed to resolve this by replacing fetch with XHR and this problem disappeared #95 (comment)

Though, It might be also fixed by some other changes in application code, we are not sure 100%.

@mattvb91

This comment has been minimized.

Copy link

commented Oct 11, 2018

@andreyselitsky could you expand on that? Using that exact snippet fixed the timeout issue in docker/jenkins? Cheers

@andreyselitsky

This comment has been minimized.

Copy link

commented Oct 11, 2018

@mattvb91 this is my guess, I can't prove it.

But when we switched from the native fetch to XHR we didn't get timeouts inside docker on ci anymore.

We basically used this recipe https://github.com/cypress-io/cypress-example-recipes/blob/master/examples/stubbing-spying__window-fetch/cypress/integration/polyfill-fetch-from-tests-spec.js

@mattvb91

This comment has been minimized.

Copy link

commented Oct 11, 2018

@andreyselitsky thanks will try it out and report back.

@JohnnyQQQQ

This comment has been minimized.

Copy link

commented Dec 13, 2018

We are having the same issue with GitLab-CI running docker and the latest cypress version.

@cookuricoo

This comment has been minimized.

Copy link

commented Dec 25, 2018

Same issue with Jenkins

@jkwohlfahrt

This comment has been minimized.

Copy link

commented Jan 9, 2019

I also am seeing this in my company's Jenkins pipelines instance.

@jkwohlfahrt

This comment has been minimized.

Copy link

commented Jan 9, 2019

Maybe I can provide a little more context as well. I am running our cypress test suite across two environments, qa and production. Our qa test suite runs fine and snappy in Docker and Jenkins pipelines. Our production test suite has been having the timeout issues in the Docker/Jenkins CI setup, but runs consistently without issue running locally in Docker. I bumped our config timeout from the default of 60 seconds to a much longer three minute timeout, and the requests finally did resolve on production, but sometimes took a little over two minutes to resolve. I guess extending the timeout is somewhat of a workaround, but certainly not a great one. I'm working with our in-house devops engineer to see if we can discover anything further. Will report back if I do.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

I know this is difficult to provide a reproducible example of, but there seem to be several issues described within here - some with proxy, some without - some with fetch, some without.

We'll need a reproducible example to narrow this down exactly - something we can run locally and see these exact timeouts. Can anyone provide one?

@mattvb91

This comment has been minimized.

Copy link

commented Jan 30, 2019

Just to add some more info this issue is not restricted to docker/jenkins. Just attempted to install a new gitlab-runner as "shell" executor and have cypress running on the host system without docker and still getting timeouts (albeit less)

@jkwohlfahrt

This comment has been minimized.

Copy link

commented Feb 1, 2019

I was finally able to fix this issue for my test suite. After doing some network tracing on the url I was trying to visit, i noticed the page trying to load a different domain. Because cypress does not allow this, the request times out after two minutes, I'm guessing due to #2055 (comment). I was able to work around this by blacklisting the problematic domain. I'm wondering if this might be the issue for others on this thread.

@mattvb91

This comment has been minimized.

Copy link

commented Feb 8, 2019

@jkwohlfahrt been tracing the network requests too including blacklisting and noticed that even with a blacklist on certain domains and the network tab showing status code 503 for the blacklisted domain it still hangs on "content download". I do think this maybe the right direction. Does it still download the content on your side even tho it has been blacklisted?

screenshot from 2019-02-08 11-37-32

Edit: The request above with "content download time" and status of 503 has the "x-cypress-matched-blacklisted-host" response header in it matching the blacklisted domain so im not sure why it hangs there.

Related to #2714?

Edit 2: After more network tracing I've actually managed to get the timeout on my local machine (without docker) and the timeline for the request that fails looks like this (Failed attempting to load a css file, css is hosted on the same domain the tests are ran on this is not an external domain.):

screenshot from 2019-02-08 14-00-11

@mattvb91

This comment has been minimized.

Copy link

commented Feb 15, 2019

@jennifer-shehane any idea from your side if the comment above could be related to this otherwise I might open up a new issue for it.

@cypress-bot

This comment has been minimized.

Copy link

commented Apr 1, 2019

The code for this is done in cypress-io/cypress#3531, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@cypress-bot

This comment has been minimized.

Copy link

commented May 17, 2019

Released in 3.3.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.