Serenity cannot close RemoteWebDriver properly in case tests are run in parallel with maven failsafe

[vayaszsolt]

I am using serenity with cucumber. I am running the tests using BrowserStack. I configured the tests to run in parallel using the maven failsafe plugin capabilities. I created 9 separate JUnit Test Runners and I divided the feature files within these runners. I am receiving some driver cannot quit exception when I run on Jenkins:
"2020-06-16 02:24:36.185 WARN 5836 --- [pool-1-thread-3] n.t.core.webdriver.WebDriverFacade : Error while quitting the driver (Session not started or terminated (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds"
I didn't receive this error when I ran the tests locally.

After receiving this error the actual test is set as PASSED but the following test from the same thread will fail with timeout exception in the beginning because the driver wasn't closed/quit properly. I contacted the BrowserStack support team and they said I need to fix the close/quite of the driver properly.

Anybody has a suggestion on how can I fix this issue?

Versions:
Chrome: 83
Chrome Driver: 83.0.4103.39 (Provided by BrowserStack)
Selenium Java: 3.141.59
Serenity-Core: 2.1.4
Serenity Cucumber4: 1.0.21

Full StackTrace when Driver cannot quit properly:
"2020-06-16 02:24:36.185 WARN 5836 --- [pool-1-thread-3] n.t.core.webdriver.WebDriverFacade : Error while quitting the driver (Session not started or terminated (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'OLAWTA-ERPJ1', ip: '10.5.122.251', os.name: 'Windows Server 2008 R2', os.arch: 'amd64', os.version: '6.1', java.version: '1.8.0_101'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 83.0.4103.39 (ccbf011cb2d2b..., userDataDir: C:\Windows\proxy\scoped_dir...}, cssSelectorsEnabled: true, databaseEnabled: false, goog:chromeOptions: {debuggerAddress: localhost:1723}, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, rotatable: false, setWindowRect: true, strictFileInteractability: false, takesHeapSnapshot: true, takesScreenshot: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unexpectedAlertBehaviour: ignore, unhandledPromptBehavior: ignore, version: 83.0.4103.61, webStorageEnabled: true, webauthn:virtualAuthenticators: true, webdriver.remote.sessionid: 370ed68d3b906bc61853e7ffdc4...}
Session ID: 370ed68d3b906bc61853e7ffdc4047a1f97a5067)
2020-06-16 02:24:36.188 ERROR 5836 --- [pool-1-thread-3] n.t.core.steps.ConsoleLoggingListener "

[wakaleo]

Do you have a reason to think it is a Serenity issue? "n.t.core.webdriver.WebDriverFacade : Error while quitting the driver (Session not started or terminated (WARNING: The server did not provide any stacktrace information)" - this message happens when an exception is thrown when the quit() method is called, so Serenity is indeed trying to close the drivers. Maybe dig into the WebDriverFacade class to see what it is doing in your scenario?

[vayaszsolt]

I don't have any detailed stack-trace information to see execrably from which part the error is coming. I identified the error is coming from WebDriverFacade.quit() method. But this method calls multiple logic. Also I remarked a strange fact. The first two tests are failing with timeout on BrowserStack. I don't have the enough resource to make the exact debug. The BrowserStack support team is saying the issue is on my side. You are saying is at the BrowserStack side :)

As we can see the Exception is not propagate into the Log:
public void quit() {
if (proxyInstanciated()) {
try {
getDriverInstance().quit();
webDriverFactory.shutdownFixtureServices();
webDriverFactory.releaseTimoutFor(getDriverInstance());

        } catch (WebDriverException e) {
            LOGGER.warn("Error while quitting the driver (" + e.getMessage() + ")");
        }
        proxiedWebDriver = null;
    }
}

Do you have any dubbing ideas that I can perform to identify the problem?

[wakaleo]

I'm not saying it's necessarily on the BrowerStack side (not enough information to go on for that). The WebDriverFacade quit() method looks like this: if the error is happening at getDriverInstance().quit(), then it might be related to BrowserStack. Or not. I would need to dig into the code with a sample project to find out. If you can do some digging directly in your project, it will be faster/easier to identify what is going on.

    public void quit() {
        if (proxyInstanciated()) {
            try {
                getDriverInstance().quit();
                webDriverFactory.shutdownFixtureServices();
                webDriverFactory.releaseTimoutFor(getDriverInstance());

            } catch (WebDriverException e) {
                LOGGER.warn("Error while quitting the driver (" + e.getMessage() + ")", e);
            }
            proxiedWebDriver = null;
        }
    }

[vayaszsolt]

From which version did you copy the quit method? As you can see I copied from 2.1.4 and the exception is not propagated to the LOGGER.

[vayaszsolt]

I checked out the latest 2.2.9 version and also there the Exception is not propagated to the LOGGER:

public void quit() {
if (proxyInstanciated()) {
try {
getDriverInstance().quit();
webDriverFactory.shutdownFixtureServices();
webDriverFactory.releaseTimoutFor(getDriverInstance());

        } catch (WebDriverException e) {
            LOGGER.warn("Error while quitting the driver (" + e.getMessage() + ")");
        }
        proxiedWebDriver = null;
   }

}

[wakaleo]

It hasn't been released yet.

[vayaszsolt]

Any other idea how can i debug at Serenity level?

[wakaleo]

I'd just put a breakpoint in the code and run it in your IDE.

[vayaszsolt]

As I said the problem is happening when I running in Jenkins against BrowserStack. Should I move the local configuration to run locally against BrowserStack?

[wakaleo]

Yes, try to run locally in the same configuration as is being run on Jenkins. It won't be a Jenkins issue.

[vayaszsolt]

I did the configuration. The provided stacktrace and debugging information didn't tell me so much. The 'quit' command with the sessionId was sent to the RemoteDriver. The RemoteDriver had answered back with the same information that already was showed in the log: "Session not started or terminated (WARNING: The server did not provide any stacktrace information". This started to happen when I configured to run the tests in parallel. Maybe should I configure something else?

[wakaleo]

That is the response that BrowserStack is sending back. The parallel execution might be related (though I do that fairly regularly with BrowserStack). It should close the driver for each thread - keep the breakpoint and run the tests to compare the drivers being closed, to see if the same driver is being closed twice.

[fozersahin]

Hi @vayaszsolt , you can try Browserstack plugin for Jenkins which is documented here : https://www.browserstack.com/docs/automate/selenium/jenkins. It will create unique local connections for each browser session. I'd suggest you to restart your browser for each example with serenity.restart.browser.for.each=example.