New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testem Chrome Timeouts #1021

Open
patocallaghan opened this Issue Nov 19, 2016 · 107 comments

Comments

Projects
None yet
@patocallaghan
Copy link
Contributor

patocallaghan commented Nov 19, 2016

Hey,

as I mention in #1015 we frequently get testem failures in CI when running against the Chrome (we use Testem through Ember CLI). Our test runs either fail to start (as if testem can't find the browser) or fail to finish (all tests have run and pass, but the browser window sits there and it eventually times out). It usually just tells us testem.js has not loaded.

As the issue is intermittent it can be hard to reproduce. I've tried to recreate it in a repo by simply forking Travis CI's web repo (uses Ember CLI) and adding ember-exam to it (Ember Exam just allows you to easily use Testem's parallel feature). If running a single test run it'd probably take some time to get a timeout, but if you run ember exam --split=9 --parallel you should start getting some timeouts in a few runs.

I've generated some (pretty verbose) logs here.

Generally on startup I've seen the following errors:

not ok 323 Chrome - error
    ---
        message: >
            Error: Browser failed to connect within 30s. testem.js not loaded?
            Stderr:
             2016-11-19 00:42:01.877 Google Chrome[28575:622538] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa45b8022a0>. Break on NSLog to debug.
            2016-11-19 00:42:01.878 Google Chrome[28575:622538] Call stack:
            (
                "+callStackSymbols disabled for performance reasons"
            )


        Log: |
            { type: 'error',
              text: 'Error: Browser failed to connect within 30s. testem.js not loaded?' }
            { type: 'error',
              text: '2016-11-19 00:42:01.877 Google Chrome[28575:622538] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa45b8022a0>. Break on NSLog to debug.\n2016-11-19 00:42:01.878 Google Chrome[28575:622538] Call stack:\n(\n    "+callStackSymbols disabled for performance reasons"\n)\n' }
    ...

On timeouts at the end I've seen

  engine:ws writing "3" +1ms
  engine:polling closing +649ms
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +8ms
  engine:polling transport not writable - buffering orderly close +1ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +654ms
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +78ms
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +5s
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +2s
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
not ok 205 Chrome - error
    ---
        message: >
            Error: Browser disconnected
            Stderr:
             2016-11-19 00:49:10.060 Google Chrome[28965:629220] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa7a07af8a0>. Break on NSLog to debug.
            2016-11-19 00:49:10.061 Google Chrome[28965:629220] Call stack:
            (
                "+callStackSymbols disabled for performance reasons"
            )


        Log: |
            { type: 'error', text: 'Error: Browser disconnected' }
            { type: 'error',
              text: '2016-11-19 00:49:10.060 Google Chrome[28965:629220] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa7a07af8a0>. Break on NSLog to debug.\n2016-11-19 00:49:10.061 Google Chrome[28965:629220] Call stack:\n(\n    "+callStackSymbols disabled for performance reasons"\n)\n' }
    ...
  socket.io:client client close with reason transport close +8s
  socket.io:socket closing socket - reason transport close +0ms

Other things to note:

  • we've been experiencing this for a while now, through multiple versions of Chrome, Node and Testem. Currently running latest Chrome, Testem 1.13 and Node 7.
  • We have a large codebase (~150K LOC) so not sure if that amplifies the issue or not.

/cc @stefanpenner (as we chatted briefly about this)

@johanneswuerbach

This comment has been minimized.

Copy link
Member

johanneswuerbach commented Nov 20, 2016

@patocallaghan first thing after looking into your repo, are you sure each browser instance starts in < 30s? This is the default start timeout and it was causing a bunch of browsers to fail for me.

You can tweak this in your config browser_start_timeout https://github.com/testem/testem/blob/master/docs/config_file.md#config-level-options

Depending on your code base and machine the default might be to small as this timeout is the time between spawn and the browser finished parsing and reconnects to the testem server.

@johanneswuerbach

This comment has been minimized.

Copy link
Member

johanneswuerbach commented Nov 20, 2016

Once testem successfully started, it should always close and I would consider this a bug, but can't really reproduce it yet :-(

@patocallaghan

This comment has been minimized.

Copy link
Contributor Author

patocallaghan commented Nov 24, 2016

@johanneswuerbach tried what you suggested with increasing the browser startup timeout but it didn't work. I'm still getting testem failing to start.

With regards this error here:

not ok 1 Browser "google-chrome --user-data-dir=/tmp/testem-2441 --no-default-browser-check --no-first-run --ignore-certificate-errors --test-type --disable-renderer-backgrounding --disable-background-timer-throttling http://localhost:7357/2441/tests/index.html?hidepassed&filter=team%20observe%20%7C" failed to connect. testem.js not loaded?

where would this be thrown from? I can't find any string in the codebase that matches failed to connect. testem.js not loaded?

@patocallaghan

This comment has been minimized.

Copy link
Contributor Author

patocallaghan commented Nov 24, 2016

@johanneswuerbach so I tried to look a little bit more into it and I saw the below error

not ok 1 Chrome - error
    ---
        message: >
            Error: Browser failed to connect within 60s. testem.js not loaded? HELLO
            Stderr:
             [58095:58095:1124/113207:ERROR:browser_main_loop.cc(261)] Gtk: Locale not supported by C library.
                Using the fallback 'C' locale.
            Xlib:  extension "RANDR" missing on display ":99".
            Xlib:  extension "RANDR" missing on display ":99".
            [58095:58118:1124/113207:ERROR:proxy_service_factory.cc(128)] Cannot use V8 Proxy resolver in single process mode.
            [58095:58118:1124/113207:ERROR:proxy_service_factory.cc(128)] Cannot use V8 Proxy resolver in single process mode.


        Log: |
            { type: 'error',
              text: 'Error: Browser failed to connect within 60s. testem.js not loaded? HELLO' }
            { type: 'error',
              text: '[58095:58095:1124/113207:ERROR:browser_main_loop.cc(261)] Gtk: Locale not supported by C library.\n\tUsing the fallback \'C\' locale.\nXlib:  extension "RANDR"missing on display ":99".\nXlib:  extension "RANDR" missing on display ":99".\n[58095:58118:1124/113207:ERROR:proxy_service_factory.cc(128)] Cannot use V8 Proxy resolver in single process mode.\n[58095:58118:1124/113207:ERROR:proxy_service_factory.cc(128)] Cannot use V8 Proxy resolver in single process mode.\n' }
    ...

I'm not sure about the Chrome error itself but interestingly Testem hung after this was thrown and didn't automatically close. It stayed open indefinitely. I wonder is there some codepath being exercised when this type of error his hit where it doesn't close the process?

@avdv

This comment has been minimized.

Copy link
Contributor

avdv commented Jan 3, 2017

Hi.

We have the same problem. Chrome just open the window and the tab spinner is running, but it seems it cannot load the page from localhost:33062/921/tests/index.html. We're using ember-cli too.

We have set the timeout to 180 seconds, but still no dice. 😭

Firefox on the other hand just works, but Chrome starts only 1/50 times. Chrome only show the Xlib: extension "RANDR" missing on display warning message.

I used x11vnc to have a look at the Xvfb display via SSH, and now chrome immediately displays the "Aw Snap, something went wrong while loading the page" error message. I've tried version 54 and 55.

When I reload the page and open the devtools, I can see that loading the 10.6 MiB resources of the page takes 2.5 minutes to finish (whereas firefox only takes a few seconds). So, after setting the timeout to ridiculous 300 seconds, chrome actually starts to run the tests...

@johanneswuerbach

This comment has been minimized.

Copy link
Member

johanneswuerbach commented Jan 3, 2017

Could you inspect the network traffic, whether some websocket traffic is done during that time? Maybe testem client / server are in an endless loop for some reason.

@cs3b

This comment has been minimized.

Copy link

cs3b commented Mar 1, 2017

I've the same issue with Firefox ... and it only is a problem in Travis CI (not sure why), locally works fine, here are the logs from Travis Ci

$ export DISPLAY=:99.0

$ sh -e /etc/init.d/xvfb start
Starting virtual X frame buffer: Xvfb.

$ sleep 3

$ firefox --version
Mozilla Firefox 53.0a2

$ node_modules/ember-cli/bin/ember test
Could not start watchman
Visit https://ember-cli.com/user-guide/#watchman for more info.
Building
cleaning up
cleaning up...
Built project successfully. Stored in "/home/travis/build/Selleo/selleo-mail-log/tmp/core_object-tests_dist-MidNYxQe.tmp".
not ok 1 Firefox - error
    ---
        message: >
            Error: Browser failed to connect within 300s. testem.js not loaded?
            Stderr: 
             ExceptionHandler::GenerateDump cloned child 3172
            ExceptionHandler::SendContinueSignalToChild sent continue signal to child
            ExceptionHandler::WaitForContinueSignal waiting for continue signal...
            
            
        Log: |
            { type: 'error',
              text: 'Error: Browser failed to connect within 300s. testem.js not loaded?' }
            { type: 'error',
              text: 'ExceptionHandler::GenerateDump cloned child 3172\nExceptionHandler::SendContinueSignalToChild sent continue signal to child\nExceptionHandler::WaitForContinueSignal waiting for continue signal...\n' }
    ...

and here is the travis ci config



{
  "language": "node_js",
  "node_js": "6",
  "env": "CXX=g++-4.8",
  "addons": {
    "apt": {
      "sources": [
        "ubuntu-toolchain-r-test"
      ],
      "packages": [
        "g++-4.8"
      ]
    },
    "firefox": "latest-dev"
  },
  "sudo": false,
  "cache": {
    "directories": [
      "$HOME/.npm",
      "$HOME/.yarn-cache",
      "$HOME/.cache"
    ]
  },
  "before_install": [
    "npm config set spin false",
    "npm install -g yarn"
  ],
  "install": [
    "yarn install",
    "node_modules/bower/bin/bower install"
  ],
  "before_script": [
    "export DISPLAY=:99.0",
    "sh -e /etc/init.d/xvfb start",
    "sleep 3",
    "firefox --version"
  ],
  "script": [
    "node_modules/ember-cli/bin/ember test"
  ],
  "group": "stable",
  "dist": "precise",
  "os": "linux"
}

might be that the problem is within travis ci itself - still hard to debug. So far the only solutions is to go back to PhantomJS.

@YoranBrondsema

This comment has been minimized.

Copy link

YoranBrondsema commented Apr 12, 2017

@patocallaghan Have you had any luck fixing this? Or any new information that came up? We're seeing the same issues (tests won't start about 1/2 of the time, on Chrome) and it's really hampering productivity as you can imagine.

  1. It happens more often that the test do not start all, with the Browser failed to connect within 30s. testem.js not loaded? message
  2. It happens that the tests hang at the end, despite having run them all. But this happens only occasionally so I'd rather focus on 1.

I'm really trying to get this fixed but it's not easy as we use Codeship as CI so we don't control the environment, and it doesn't always happen. Most fun bugs... :( So anything new that you discovered would be welcome!

@ghempton

This comment has been minimized.

Copy link

ghempton commented Apr 12, 2017

This same issue happens for us on CircleCI. About 1/10 builds fail to launch with Browser failed to connect within 30s. testem.js not loaded.

@johanneswuerbach

This comment has been minimized.

Copy link
Member

johanneswuerbach commented Apr 13, 2017

I didn't had any luck tracking this down yet, especially as this mainly happens only on CI and isn't really reliable reproducable.

Hopefully chrome headless https://www.chromestatus.com/features/5678767817097216 will bring together with remote debugging better debugging in those environments

@mani-mishra

This comment has been minimized.

Copy link

mani-mishra commented Apr 19, 2017

We are using ember exam and also seeing similar issues with our Team City build server.
The error message is slightly different though.

[Chrome - error] Error: Browser failed to connect within 30s. testem.js not loaded?
Stderr: 
 [0419/061938.039012:WARNING:audio_manager.cc(321)] Multiple instances of AudioManager detected
[0419/061938.039065:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
[0419/061938.371223:ERROR:socket_posix.cc(126)] bind() returned an error, errno=98: Address already in use
[0419/061938.371271:ERROR:devtools_http_handler.cc(221)] Cannot start http server for devtools. Stop devtools.
@johanneswuerbach

This comment has been minimized.

Copy link
Member

johanneswuerbach commented Apr 19, 2017

It would be great to get the chrome debug console output in those cases, maybe using the remote debugger are somehow manually.

Looks like in some cases the initial connect takes >30s with can have multiple reasons: unparsable js, slow js parsing, failed websocket, testem client bug, etc.

Sadly there isn't much, which can be done before connecting (as there is obviously no connection), but maybe the remote debug protocol could provide some access to the console output

@YoranBrondsema

This comment has been minimized.

Copy link

YoranBrondsema commented Apr 19, 2017

@manimis902 Are you running headless Chrome? In that case, your error might be an issue in Chrome itself (see https://bugs.chromium.org/p/chromium/issues/detail?id=678948).

@mani-mishra

This comment has been minimized.

Copy link

mani-mishra commented Apr 19, 2017

@YoranBrondsema Yes we are using chrome headless. I landed on the chromium issue while debugging , although for time being, we solved the failures by increasing browser_start_timeout to 150s.

FYI, our test suite runs in 24 partitions.

@johanneswuerbach

This comment has been minimized.

Copy link
Member

johanneswuerbach commented Apr 19, 2017

@YoranBrondsema do you also run all 24 partitions in parallel? That might be explaining why spawning the 24 instances is taking >30s.

@YoranBrondsema

This comment has been minimized.

Copy link

YoranBrondsema commented Apr 20, 2017

@johanneswuerbach I run 2 partitions in parallel but they are run on 2 separate VMs on Codeship. So only one instance is started on each VM.

@knownasilya

This comment has been minimized.

Copy link
Contributor

knownasilya commented May 19, 2017

Having a similar issue, if I increase the browser timeout, I get this error.

Error: Browser exited unexpectedlyNon-zero exit code: 1[eval("(function(undefined) {window.jasmine && (JSTestingReport
erSL = (function(undefined) {\n\n return function() {\n return {\n failedCount: jasmine.currentEnv_.currentRu
nner_.results().failedCount\n };\n }\n\n})());\n\nwindow.mocha && (JSTestingReporterSL = (function(undefined) {\n
n function hasArray(obj, nameArray) {\n return obj && obj[nameArray] && obj[nameArray].length > 0;\n }\n\n funct
ion hasSuite(obj) {\n return hasArray(obj, "suites");\n }\n\n function hasSpec(obj){\n return hasArray(obj,
"tests");\n }\n\n function reporterSpec(test) {\n var isPassed = test.state === "passed";\n return {\n
description : test.title,\n durationSec : test.duration/1000,\n passed : isPassed,\n passedCount
: isPassed ? 1 : 0,\n failedCount : isPassed ? 0 : 1,\n totalCount : 1\n };\n }\n\n function emptyRep
orter() {\n return {\n description : "",\n durationSec : 0,\n passed : true,\n totalCoun
t : 0\n };\n }\n\n function reporterObjects(objs, fn) {\n var reporter = emptyReporter();\n reporter.resul
t = [];\n\n for (var i=0; i<objs.length; i++) {\n var obj = fn(objs[i]);\n reporter.durationSec += obj.du
rationSec ? obj.durationSec : 0;\n reporter.totalCount += obj.totalCount;\n reporter.passed = reporter.pass
ed && obj.passed;\n reporter.result.push(obj);\n }\n\n return reporter;\n }\n\n function reporterSpecs(te
sts) {\n return reporterObjects(tests, reporterSpec);\n }\n\n function reporterSuites(suites) {\n return repor
terObjects(suites, reporterSuite);\n }\n\n function reporterSuite(suite) {\n var specs = hasSpec(suite) ? repor
terSpecs(suite.tests) : emptyReporter(),\n suites = hasSuite(suite) ? reporterSuites(suite.suites) : emptyRep
orter();\n\n return {\n description : suite.title,\n specs : specs.result || [],\n suites
: suites.result || [],\n durationSec : specs.durationSec + suites.durationSec,\n totalCount : specs.totalC
ount + suites.totalCount,\n passed : specs.passed && suites.passed\n };\n }\n\n return function() {\n

I'm using ember-cli-sauce.

@derekdowling

This comment has been minimized.

Copy link

derekdowling commented Jun 16, 2017

Seeing similar output to: #1021 (comment) running Chrome 59 headless with the following config:

launch_in_ci: [ 'Chrome' ],
browser_args: {
  Chrome: [ '--headless', '--disable-gpu', '--remote-debugging-port=9222', '--no-sandbox'] 
}
@briangonzalez

This comment has been minimized.

Copy link
Contributor

briangonzalez commented Jun 30, 2017

@derekdowling @knownasilya Any luck on your end? We're seeing this, too.

@knownasilya

This comment has been minimized.

Copy link
Contributor

knownasilya commented Jun 30, 2017

No improvements here, unfortunately.

@lolmaus

This comment has been minimized.

Copy link

lolmaus commented Jul 3, 2017

@vitkhab has discovered that migrating to this Docker container resolves the issue. Notice this item on their methodology:

  1. Traps SIGTERM and relays the signal to the browser and Xvfb so that they will not continue to run once the test runner is finished. This is mostly required to deal with Karma's insistence that the browser process be terminated before it can exit. Without this relaying, Karma would hang indefinitely.
@jonathanong

This comment has been minimized.

Copy link

jonathanong commented Jul 7, 2017

a bigger issue for me is that when i get this error, testem does not exit the process. my CI build ends up hanging. is this a known issue? i don't mind retrying my test if i get this type of error

@bricss

This comment has been minimized.

Copy link

bricss commented Jul 13, 2017

This is how I fixed timeout issue with Chrome 59+:

{
    "framework": "qunit",
    "test_page": "tests/index.html?hidepassed",
    "disable_watching": true,
    "launch_in_ci": [
        "Chrome"
    ],
    "launch_in_dev": [
        "Chrome"
    ],
    "browser_start_timeout": 60,
    "browser_args": {
        "Chrome": [
            "--disable-gpu",
            "--disable-web-security", // optional, since chrome 60+
            "--headless",
            "--incognito", // optional
            "--no-sandbox", // optional, since chrome 60+
            "--remote-debugging-address=0.0.0.0",
            "--remote-debugging-port=9222"
        ]
    }
}
@lolmaus

This comment has been minimized.

Copy link

lolmaus commented Jul 25, 2017

We're seeing Error: Browser disconnected in our Ember tests and the above does not help. 😭 Different issue?

@bricss

This comment has been minimized.

Copy link

bricss commented Jul 26, 2017

@lolmaus Chrome 60 was released today, try it. 🤓

@lolmaus

This comment has been minimized.

Copy link

lolmaus commented Jul 27, 2017

Our erros were caused by our Ember app leaving /tests after an Ember Data upgrade. There's still something fishy, but at least it's not related to Testem.

@ming-codes

This comment has been minimized.

Copy link

ming-codes commented Sep 27, 2018

@step2yeung Going over testem's code, it seems like testem tries to terminate the process immediately after receiving after-tests-complete. This could explain why the socket is disconnected unexpected. Shouldn't testem tries to clean up the socket before terminating the process?

@allthesignals

This comment has been minimized.

Copy link

allthesignals commented Dec 18, 2018

I'm also seeing this error regularly at random times. Sometimes it will switch from that to RuntimeException: Found equal nodes with different coordinates, an error which only appears in Firefox and one which I can't find any information on (I've never heard of it).

EDIT: the error is "Browser timeout exceeded: 10s"

@step2yeung

This comment has been minimized.

Copy link
Contributor

step2yeung commented Dec 18, 2018

@ming-codes When you say "disconnected unexpectedly", can you point to which error in the code? (Want to make sure we are talking about the same one) Would you have any examples of the lack of cleanup causing the error?

But I do agree we might want to do socket cleanup before terminating the process.

@allthesignals this error being..?
Definitely first time seeing RuntimeException: Found equal nodes with different coordinates. Doesn't seem like that is coming from Testem, what is throwing that error?

@allthesignals

This comment has been minimized.

Copy link

allthesignals commented Dec 18, 2018

@step2yeung "Browser timeout exceeded: 10s" - the other one I've learned is unrelated.

@alexscarlett

This comment has been minimized.

Copy link

alexscarlett commented Dec 19, 2018

I'd like to chime in and mention I started seeing the same error that @allthesignals mentioned ("Browser timeout exceeded: 10s") after adding a single window.addEventListener(...); in my app...I've tried to set browser_disconnect_timeout to a large value but still fails. Also, for me the error is thrown only when running the entire test suite.

@stefanpenner

This comment has been minimized.

Copy link
Contributor

stefanpenner commented Dec 19, 2018

Interesting @alexscarlett can you share a reproduction?

@NullVoxPopuli

This comment has been minimized.

Copy link

NullVoxPopuli commented Dec 19, 2018

@stefanpenner my app is a reproduction https://github.com/NullVoxPopuli/emberclear/
but, I'm not sure where this is happening. I have event listeners in multiple places.
The only way I can run my tests at a reasonable pace is in headless

@stefanpenner

This comment has been minimized.

Copy link
Contributor

stefanpenner commented Dec 19, 2018

@step2yeung if you have time, mind investigating ^^ ?

@step2yeung

This comment has been minimized.

Copy link
Contributor

step2yeung commented Dec 20, 2018

@NullVoxPopuli Can you share some reproduction steps to reproduce the error?
Steps i taken so far:

git clone https://github.com/NullVoxPopuli/emberclear.git
cd emberclear/packages/frontend
yarn install
ember build
ember test --path=dist 

And I have been unable to reproduce the timeout (Ran ember test --path=dist for 30 minutes, no failures). Are you able to get a consistent reproduction of the problem?

@step2yeung

This comment has been minimized.

Copy link
Contributor

step2yeung commented Dec 20, 2018

@alexscarlett Same question to you, do you see browser timeout exceeded consistently, at the same test?

One thing that may help with understanding when this error arises: this error is thrown because the socket connection between testem and the browser has gone stale (the heartbeat events that socket.io use to check whether the connection is still alive has a timeout value, and when it does timeout, this is when you get browser timeout exceeded)
From experience, one very basic example is an infinite loop cause by a test, causes the browser's main thread be hogged by the infinite loop, never allowing the socketio code to respond to heartbeat event, causing a browser timeout exceeded.

@alexscarlett

This comment has been minimized.

Copy link

alexscarlett commented Dec 20, 2018

@step2yeung hey there, I do get the browser timeout exceeded error everytime, but it states a different test as the problem every run. seems to be arbitrary

@NullVoxPopuli

This comment has been minimized.

Copy link

NullVoxPopuli commented Dec 20, 2018

@step2yeung I usually just do yarn start:dev and then visit https://localhost:4201/tests
Thanks for looking! 🗡

But also, 30 minutes is weird, the tests should all run in under 3 minutes.
It's not super consistent, but I do occasionally get the message that a test timed out after 1 minute of waiting.

image

@step2yeung

This comment has been minimized.

Copy link
Contributor

step2yeung commented Dec 20, 2018

@NullVoxPopuli Apologies. What I meant was ran the test suite multiple times, within the 30 minutes I've spent, I did not get a browser timeout exceeded. The average time of the test suite for me is ~80-90 seconds.

@NullVoxPopuli

This comment has been minimized.

Copy link

NullVoxPopuli commented Dec 20, 2018

hmm, maybe something is wrong with my computer then :(

@step2yeung

This comment has been minimized.

Copy link
Contributor

step2yeung commented Dec 20, 2018

@NullVoxPopuli Just ran the tests with yarn start:dev and hitting http://localhost:4201/tests.
I'm not sure if this is the issue, but it seems like there is some callback that's not returning (or returns much much later), causing the test to just stay there and wait.

screen shot 2018-12-20 at 9 36 15 am

~a minute later, the test decided that it passed. And this was true for more than 1 test:
screen shot 2018-12-20 at 9 43 41 am

This also leads to some test failing for 60s timeout that is a default for qunit:
screen shot 2018-12-20 at 9 44 49 am

Something there definitely is fishy, since I don't expect the simple tests to have that long of a wait. Although I have yet to see the browser timeout exceeded this might be somewhere you want to explore.

@NullVoxPopuli

This comment has been minimized.

Copy link

NullVoxPopuli commented Dec 20, 2018

is there a way to list everything that ember/qunit is waiting on? like, for every test, log outstanding timers or something? I could track that down, and report back after some investigating

@step2yeung

This comment has been minimized.

Copy link
Contributor

step2yeung commented Dec 20, 2018

@NullVoxPopuli The latest version of ember-qunit (4.2.0) added test-isolation that may help with this: https://github.com/emberjs/ember-qunit/blob/master/TEST_ISOLATION_VALIDATION.md

@thec0keman

This comment has been minimized.

Copy link

thec0keman commented Dec 21, 2018

Is it possible this could be also triggered by memory? We have an Ember test suite with over 3000 tests, and several hundred of them are acceptance tests. We've been seeing this timeout error sporadically when running the whole suite in a single travis instance, but splitting the suite into multiple travis runners seems to have helped.

@snewcomer

This comment has been minimized.

Copy link

snewcomer commented Dec 21, 2018

Recently updated to ember-cli 3.6 and saw this error on our Jenkins setup (node 8.11). Have tried a wide variety of recommendations but no go yet.

Some flgs we pass to our docker instance

--cpuset-cpus 0-11 --memory 165G --memory-reservation 148G --shm-size 512M

screenshot 2018-12-21 at 10 58 20

@brandynbennett

This comment has been minimized.

Copy link

brandynbennett commented Jan 5, 2019

I get Error: Browser timeout exceeded: 10s randomly when using ember exam --split=2 --parallel in CircleCi. When I go back to plain ember test it works consistently.

Just happened to me with plain ember test as well

@brandynbennett

This comment has been minimized.

Copy link

brandynbennett commented Jan 16, 2019

Happened when setting browser time to 500 as well.

@brandynbennett

This comment has been minimized.

Copy link

brandynbennett commented Jan 18, 2019

Was able to get mine fixed by moving to the latest version of ember-gestures (1.1.1) There was a memory leak there that was causing Chrome to run out of memory and thus the socket disconnected.

@snewcomer

This comment has been minimized.

Copy link

snewcomer commented Jan 18, 2019

Yeah I guess it makes sense it is in the dependency graph somewhere. I'm sure it varies for many people. Will keep digging!

@scottkidder

This comment has been minimized.

Copy link

scottkidder commented Feb 5, 2019

I'm currently on ember-gestures 1.1.1 and still seeing these issues. I'm quite certain there is some useful information to be had by using https://github.com/emberjs/ember-qunit/blob/master/docs/TEST_ISOLATION_VALIDATION.md but I haven't had time to dig into it quite yet.

@scottkidder

This comment has been minimized.

Copy link

scottkidder commented Feb 6, 2019

🎉 Using qunit test isolation and the stack traces it provides in the console I believe I have tracked my issues down to some leaky async in ember-cli-loading-slider. 🤞

@lolmaus

This comment has been minimized.

Copy link

lolmaus commented Feb 6, 2019

@scottkidder The file you linked to is unavailable. Can you please update the link?

@patocallaghan

This comment has been minimized.

Copy link
Contributor Author

patocallaghan commented Feb 6, 2019

@bracke

This comment has been minimized.

Copy link

bracke commented Feb 12, 2019

I have the same problem, except it seems to vary between platforms:

On OSX testing works without problem.
On Windows 10 Testem fails as described above.
Note: This is with the same projekt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment