headless mode not working

[to-do42]

For Vaadin24 and the selenium dependencies coming with this version, it's not possible to run a Vaadin E2E Test in headless mode for Chrome or Firefox. I've test the LoginE2ETest mentioned by the Vaadin tutorial and the MainViewIT from the Vaadin Starter Hello World skeleton project:

I want to create some basic headless Login View E2E tests with TestBench for Vaadin24 according the tutorial in https://vaadin.com/docs/latest/tutorial/e2e-testing-with-testbench#test-the-login-view. Nevertheless, the Firefox window is always shown, whether with firefoxOptions.addArguments("-headless"); or without. I use selenium firefox driver 4.14 and latest geckodriver 0.33.
With the headless argument, the window is show but without any html content, only the menu, navigation elements etc. are shown, but no content. Without the headless argument, everything is shown as expected. I've tried the https://github.com/vaadin/testbench-demo as well with same result: if I enable headless mode in AbstractIT.java, in this case the Chrome window is shown, no headless mode is available.

Since the new start.vaadin.com provides the Hello World Project skeleton containing the MainViewIT test, it's easy to reproduce the issue:

1. go to start.vaadin.com and load the pre-defined Hello World Project with VaadinFlow 24, Java17, Maven + Spring Boot (3.1.5)

2. start test via mvn clean integration-test -Pit
   => browser windows (Chrome per default) is opened and you see the view provided by the application, test succeeds ✅

3. edit the MainViewIT.java and switch to headless mode by adding the following in in method public void open as mentioned in the Vaadin docs

ChromeOptions options = new ChromeOptions();
options.addArguments("--headless", "--disable-gpu");
setDriver(new ChromeDriver(options));

4. run the test again

I've got the following

• browser windows is opened
• no content within (as compared to the main view shown in non-headless mode)
• test succeeds

I've tested on MacOS and Windows, same result. Here at least the spec for one of those workstations:
MacStudio with Apple Silicon (Apple M1 Max, MacOS Ventura 13.5.1), Java 17 (Eclipse Temurin 17.0.8 via SDKMAN), Maven 3.9.4

[TatuLund]

Apparently there are some documentation about this here: https://www.selenium.dev/blog/2023/headless-is-going-away/, which works for me.

        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless=new");
        setDriver(TestBench.createDriver(new ChromeDriver(options)));

Alternatively one can use Parameters.setHeadless(true)

The documentation should be updated accordingly here https://vaadin.com/docs/latest/testing/end-to-end/ci-server#making-sure-the-browsers-are-available

[mvysny]

The setDriver(new FirefoxDriver(options)); call is ignored with the newest @BrowserTest annotation - the browser is created differently and this is undocumented at the moment, related to #1730