Browser disconnected issue

[valgussev]

What is the Current behavior?

XXX browser disconnected. This problem may appear when a browser hangs or is closed, or due to network issues

What is the Expected behavior?

Do not disconnect or freeze chromium browser

Steps to Reproduce:

The setup we currently have where the problem reproduces is official testcafe/testcafe docker image run on the kubernates pod with the following resources allocated:

resourceLimitCpu: '5000m',
resourceLimitMemory: '5000Mi',

The error XXX browser disconnected. This problem may appear when a browser hangs or is closed, or due to network issues is happening in random places of the testsuite with arbitrary frequency. If to run the same docker image locally on Mac or if to run tests in debian VM everything works just fine and tests never crash.

It reproduces on chromium run with the following flags --no-sandbox --disable-dev-shm-usage and on the latest testcafe version 1.1.1. More frequently if to reduce the test run speed to 0.2 (--speed).

I've taken a look onto chromium logs but there is nothing suspicious except the following errors at the middle of the log file

[0403/084113.645826:ERROR:gpu_process_transport_factory.cc(1016)] Lost UI shared context.
[0403/084113.659220:ERROR:zygote_host_impl_linux.cc(270)] Failed to adjust OOM score of renderer with pid 657: Permission denied (13)
[0403/084113.704354:ERROR:gl_implementation.cc(292)] Failed to load /usr/lib/chromium/swiftshader/libGLESv2.so: Error loading shared library /usr/lib/chromium/swiftshader/libGLESv2.so: No such file or directory
[0403/084113.706164:ERROR:viz_main_impl.cc(201)] Exiting GPU process due to errors during initialization
[0403/084113.707268:ERROR:service_manager_context.cc(250)] Attempting to run unsupported native service: /usr/lib/chromium/content_gpu.service
[0403/084113.754683:ERROR:gl_implementation.cc(292)] Failed to load /usr/lib/chromium/swiftshader/libGLESv2.so: Error loading shared library /usr/lib/chromium/swiftshader/libGLESv2.so: No such file or directory
[0403/084113.756565:ERROR:viz_main_impl.cc(201)] Exiting GPU process due to errors during initialization
[0403/084113.757762:ERROR:service_manager_context.cc(250)] Attempting to run unsupported native service: /usr/lib/chromium/content_gpu.service

It never reproduces if to run tests on BrowserStack browsers, so I believe there could be something wrong with GCP kubernates configuration or a hosting environment. So any suggestions on how to localize and reproduce the problem are very welcome.

Your Environment details:

• testcafe version: 1.1.1
• node.js version: 8.15.0 / 10.15.0
• command-line arguments: CHROME_LOG_FILE=$(pwd)/chrome_debug_$(date +%s).log testcafe 'chromium:headless --no-sandbox --disable-dev-shm-usage --enable-logging --v=1 --disable-gpu --disable-translate --disable-extensions --no-proxy-server --disable-setuid-sandbox --disable-web-security' 'testcafe/'
• browser name and version: Chromium 68/Chromium 72
• platform and version: Linux Alpine / Linux Debian 9

[AlexKamaev]

I can only assume that there is an issue with GCP configuration as you correctly mentioned. Perhaps your pricing plan does not provide enough performance for a browser. I cannot currently determine the real cause of the issue and can only share common recommendations from the documentation: https://devexpress.github.io/testcafe/faq/#not-enough-resources-in-the-container-or-ci-system

[valgussev]

@AlexKamaev thank you for your response. As I mentioned in the issue we have a plenty of resources available and allocated for the pod

resourceLimitCpu: '5000m',
resourceLimitMemory: '5000Mi',

Where tests run fine locally even on a low end macbook pro 2010. So I just don't believe this is a case.

Set the concurrency factor to 1.

Confirm, it is 1

Deploy the application's Web server on a separate machine.

Ok, so I just caught the same problem running tests on the separate VM with the following configuration 2 vCPUs, 7.5 GB memory. So I can confirm it is not specifically related to a docker container setup.

Run tests on a local device outside a container.

Never reproduced this problem locally

A part from those steps any other suggestions to investigate the causes of this issue? May network outage or VRAM effect? Anything you would suggest to take a look, debug or to take logs from?

[miherlosev]

Hi @valerii-cognite
 
Based on the description, it looks like this issue is not related to TestCafe, but to the kubernetes setup/configuration or something else. We are not familiar with Kubernetes pod, so we cannot provide any valid suggestions. If you are able to provide us with a simple example to reproduce this issue, please reopen this issue.

[valgussev]

I've digged into source code. In BrowserConnection I can see a notion of HEARTBEAT_TIMEOUT which if went off the error above is going to be fired. In order to prevent that from happening browser should send a heartbeat to /browser/heartbeat/{id} callback of which will reset the interval. This is set in browser/index.js to happen every HEARTBEAT_INTERVAL. For some reason even though I can see a chromium process up and running and recording logs the heartbeat may be absent for some random intervals or sometimes even exceeding the HEARTBEAT_TIMEOUT. What may cause such a problem?

[valgussev]

I connected to the chromium instance through --remote-debugging-port option and ssh port forwarding. It seems tests are just stuck from time to time, here is an example (be patient it is 29s long)

Here is the test code sample

import { Selector } from 'testcafe';
import uuid from 'uuid/v4';
const salt = uuid();

fixture('Checklists').page('/checklists');

test('Checklists', async t => {
  const createButton = Selector(`[data-test-id="checklist-list-add-button"]`);
  const createConfirmButton = Selector(`[data-test-id="add-custom-list-create-button"]`);
  const createNameInput = Selector(`[data-test-id="add-custom-list-input"]`);
  const newChecklistName = `new checklist ${salt}`;

  console.log('Create new checklist without items');
  await t
    .click(createButton)
    // make sure confirm button is disabled for empty name
    .expect(createConfirmButton.hasAttribute('disabled'))
    .ok()
    .typeText(createNameInput, newChecklistName)
    // make sure confirm button is enabled
    .expect(createConfirmButton.hasAttribute('disabled'))
    .notOk()
    .click(createConfirmButton);
});

Below is the heartbeat breakdown for this test where you can see a gap for heartbeat between 05s and 40s:

[0409/193603.942928:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193603.942981:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193605.931130:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193605.931182:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193640.603728:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193640.603810:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193650.096301:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193650.096357:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193656.527291:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7
[0409/193656.527388:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://172.16.64.26:38209/browser/heartbeat/mvNvxG7

If to load the same page just with chromium without testcafe and run same instructions it works without any freezes. I'm wondering what can block testcafe thread so heavily?