Headless Chrome slows tests by 10x

[fatso83]

What are you trying to achieve?

I am trying to run my tests in headless Chrome using the --headless argument. I expect the tests to run in approximately the same amount of time as the usual runs, as running a clean webdriverio script runs faster headless than in a normal browser window.

What do you get instead?

The tests run about ten times as slow, using 44 seconds instead of 4 seconds.

$ time $(npm bin)/codeceptjs run --verbose > headless-output.txt

real	0m44.973s
user	0m0.992s
sys	0m0.068s

$ nvim codecept.json # removing --headless option

developer at developer-XPS-13-9360 in ~/dev/dx-nuflo/tests/acceptance (acceptance-testing)
$ time $(npm bin)/codeceptjs run --verbose > output.txt

real	0m4.879s
user	0m0.768s
sys	0m0.088s

verbose output from headless run

CodeceptJS v0.6.3
Using test root "/home/developer/dev/dx-nuflo/tests/acceptance"

Top level-routes --
   [1] Starting recording promises
   Emitted | suite.before ([object Object])
   [1] Queued | hook WebDriverIO._beforeSuite()
 a page to add a new encounter should be rendered
 > [Session] Starting singleton browser session
   [2] Starting recording promises
   Emitted | test.before
   [2] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [2] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [2] Queued | return step result
   Emitted | step.before (I see "LOGG INN")
   [2] Queued | see: "LOGG INN"
   Emitted | step.after (I see "LOGG INN")
   [2] Queued | return step result
   [2] Queued | fire test.passed
   [2] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "LOGG INN")
 • I see "LOGG INN"
   Emitted | test.passed ([object Object])
 ✓ OK in 22317ms

   Emitted | test.after
   [2] Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
 fooScen
   [3] Starting recording promises
   Emitted | test.before
   [3] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [3] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [3] Queued | return step result
   Emitted | step.before (I see "FOOLOGG INN")
   [3] Queued | see: "FOOLOGG INN"
   Emitted | step.after (I see "FOOLOGG INN")
   [3] Queued | return step result
   [3] Queued | fire test.passed
   [3] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "FOOLOGG INN")
 • I see "FOOLOGG INN"
   [3] Error | Error
   [3] Starting <teardown> session
   Emitted | test.failed ([object Object])
   [3] <teardown> Queued | hook WebDriverIO._failed()
   [3] <teardown> Queued | () => done(err)
   [3] <teardown> Stopping recording promises
 > Screenshot has been saved to /home/developer/dev/dx-nuflo/tests/acceptance/output/fooScen.failed.png
 ✖ FAILED in 11599ms

   Emitted | test.after
   [3] <teardown> Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
   [4] Starting recording promises
   Emitted | suite.after ([object Object])
   [4] Queued | hook WebDriverIO._afterSuite()

-- FAILURES:

  1) Top level-routes: fooScen:

      expected web page to include "FOOLOGG INN"
      + expected - actual

      -Innlogging
      -Brukernavn
      -Passord
      -LOGG INN
      +FOOLOGG INN
      
  
  Scenario Steps:
  
  - I.see("FOOLOGG INN") at Test.Scenario (top-level-routes_test.js:18:5)
  - I.amOnPage("/encounter/new") at Test.Scenario (top-level-routes_test.js:14:5)
  
  Error
      at Object.<anonymous> (/home/developer/dev/dx-nuflo/node_modules/codeceptjs/lib/helper/WebDriverIO.js:1129:51)
      at Object.exec (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
      at Object.resolve (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:189:29)
      at /home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:538:32
      at _fulfilled (/home/developer/dev/dx-nuflo/node_modules/q/q.js:854:54)
      at self.promiseDispatch.done (/home/developer/dev/dx-nuflo/node_modules/q/q.js:883:30)


  FAIL  | 1 passed, 1 failed   // 34s
   Emitted | global.result ([object Object])


real	0m44.973s
user	0m0.992s
sys	0m0.068s

verbose output from normal (non-headless) run

CodeceptJS v0.6.3
Using test root "/home/developer/dev/dx-nuflo/tests/acceptance"

Top level-routes --
   [1] Starting recording promises
   Emitted | suite.before ([object Object])
   [1] Queued | hook WebDriverIO._beforeSuite()
 a page to add a new encounter should be rendered
 > [Session] Starting singleton browser session
   [2] Starting recording promises
   Emitted | test.before
   [2] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [2] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [2] Queued | return step result
   Emitted | step.before (I see "LOGG INN")
   [2] Queued | see: "LOGG INN"
   Emitted | step.after (I see "LOGG INN")
   [2] Queued | return step result
   [2] Queued | fire test.passed
   [2] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "LOGG INN")
 • I see "LOGG INN"
   Emitted | test.passed ([object Object])
 ✓ OK in 2748ms

   Emitted | test.after
   [2] Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
 fooScen
   [3] Starting recording promises
   Emitted | test.before
   [3] Queued | hook WebDriverIO._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I am on page "/encounter/new")
   [3] Queued | amOnPage: "/encounter/new"
   Emitted | step.after (I am on page "/encounter/new")
   [3] Queued | return step result
   Emitted | step.before (I see "FOOLOGG INN")
   [3] Queued | see: "FOOLOGG INN"
   Emitted | step.after (I see "FOOLOGG INN")
   [3] Queued | return step result
   [3] Queued | fire test.passed
   [3] Queued | finish test
   Emitted | step.start (I am on page "/encounter/new")
 • I am on page "/encounter/new"
   Emitted | step.start (I see "FOOLOGG INN")
 • I see "FOOLOGG INN"
   [3] Error | Error
   [3] Starting <teardown> session
   Emitted | test.failed ([object Object])
   [3] <teardown> Queued | hook WebDriverIO._failed()
   [3] <teardown> Queued | () => done(err)
   [3] <teardown> Stopping recording promises
 > Screenshot has been saved to /home/developer/dev/dx-nuflo/tests/acceptance/output/fooScen.failed.png
 ✖ FAILED in 1299ms

   Emitted | test.after
   [3] <teardown> Queued | hook WebDriverIO._after()
 > [Session] cleaning cookies and localStorage
   [4] Starting recording promises
   Emitted | suite.after ([object Object])
   [4] Queued | hook WebDriverIO._afterSuite()

-- FAILURES:

  1) Top level-routes: fooScen:

      expected web page to include "FOOLOGG INN"
      + expected - actual

      -Innlogging
      -Brukernavn
      -Passord
      -LOGG INN
      +FOOLOGG INN
      
  
  Scenario Steps:
  
  - I.see("FOOLOGG INN") at Test.Scenario (top-level-routes_test.js:18:5)
  - I.amOnPage("/encounter/new") at Test.Scenario (top-level-routes_test.js:14:5)
  
  Error
      at Object.<anonymous> (/home/developer/dev/dx-nuflo/node_modules/codeceptjs/lib/helper/WebDriverIO.js:1129:51)
      at Object.exec (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
      at Object.resolve (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:189:29)
      at /home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:538:32
      at _fulfilled (/home/developer/dev/dx-nuflo/node_modules/q/q.js:854:54)
      at self.promiseDispatch.done (/home/developer/dev/dx-nuflo/node_modules/q/q.js:883:30)


  FAIL  | 1 passed, 1 failed   // 4s
   Emitted | global.result ([object Object])


real	0m4.879s
user	0m0.768s
sys	0m0.088s

Test source code

Feature('Top level-routes');

Scenario('a page to add a new encounter should be rendered', (I) => {
  I.amOnPage('/encounter/new');
  I.see('LOGG INN');
});

// just to see how a failing test looks
Scenario('fooScen', (I) => {
  I.amOnPage('/encounter/new');
  I.see('FOOLOGG INN');
});

Details

• CodeceptJS version: 0.6.3
• NodeJS Version: 8.1.2
• Operating System: Ubuntu 16.04 LTS
• WebDriverIO: 4.8.0
• Configuration file:

This is my configuration:

{
  "tests": "./*_test.js",
  "timeout": 5000,
  "output": "./output",
  "helpers": {
    "WebDriverIO" : {
      "browser": "chrome",
      "desiredCapabilities": {
          "browserName": "chrome",
          "chromeOptions": {
              "args": [
                  "headless",
                  "disable-gpu"
              ]
          }
      },
      "windowSize": "320x568",
      "restart": false,
      "timeouts": {
        "script": 10000,
        "page load": 3000,
        "implicit" : 5000
      },
      "url": "http://localhost:23450"
    }
  },
  "include": {
    "I": "./steps_file.js"
  },
  "bootstrap": false,
  "mocha": {},
  "name": "acceptance tests",
  "windowSize": "320x568"
}

[laurentpayot]

@fatso83 Did you try with node < 8 such as 7.10.0?

[fatso83]

Tried with Node 7.4.0. Still a lot slower.

A test run took 26 seconds with the non-headless browser and 1m5s with --headless. Not 10x slow, but at least 4x slower (probably due to more page loading in these tests, obscuring performance).

[DavertMik]

Thanks, I will investigate this issue

[DavertMik]

I reproduced the issue but I still have no clue what's wrong.
Just changing Chrome from headless to window makes test run much faster. Can this be caused because by disable-gpu flag? I don't know. Probably headless Chrome is not so stable enough

[Setitch]

I see the same slowness with and without --dissable-gpu so thats not the case. When i connect to the headless i see SO MANY ssl warning and so on - i think its just due to that.

[DavertMik]

Thanks. Maybe this could be the case

[mogryo]

Hey. Is there any updates on this issue? We've ran into similar problem on some of our machines. Our tests just timeout because of the enormous time it takes for headless chrome to do what we ask. And if launched not in headless it runs smooth.

We ran into this problem on windows 7. With latest chromedriver 2.33 and latest chrome. But it's very random. On other machines with the same configuration it runs perfectly in headless (well, not exactly perfectly, but only ~1.5x times slower). Perhaps you have an idea what should we check on these machines? to pin point the mystical difference between them.

[emesterhazy]

Also experiencing extremely slow load times when using Chrome in headless mode in Windows 7 with the latest version of Chromedriver, Chrome, and Selenium. Interestingly I'm not having any issues on Windows 10.

[JamesLMilner]

I'm coming at this from a different place, but interestingly there's an open issue on ChromeDriver which sounds suspiciously related: https://bugs.chromium.org/p/chromedriver/issues/detail?id=2054&q=jenkins&colspec=ID%20Status%20Pri%20Owner%20Summary

[laskeym]

I'm in the same boat as @emesterhazy . I currently have Chrome Version 63.0.3239.132 and ChromeDriver 2.35 on Windows 7 and headless hangs up on retrieving the sites. When I come home from work, I'll mess around with this on my Debian 9 and Windows 10 machine and report the results.

Simple test of retrieving the Facebook home page and grabbing login forms:
Windows 7:

• Headless = 111.667s
• Non-Headless = 3.018s

Also, I have Google Chrome Canary downloaded, but I haven't attempted to configure the driver to use it. Not sure if using Canary over standard Chrome would make a difference.

[oligee80]

Try these two ChromeOptions:

options.addArguments("--proxy-server='direct://'");
options.addArguments("--proxy-bypass-list=*");

I dunno in wich forum it was but this did the trick for me