[ChromeDriver] Tests failing with invalid session id

[PragTob]

Howdy fellow wallabyers!

We have some tests that are stubbornly flaky on CI and sometimes locally and I can't for the live of me figure out why that might be happening.

I couldn't reproduce locally right now but on CI a colleague had to rerun the tests 5 times for them to pass.

Error

    1) test langing page contains css (PricingEngineWeb.LandingPageTest)
      apps/pricing_engine_web/test/pricing_engine_web/features/landing_page_test.exs:5
      ** (FunctionClauseError) no function clause matching in Wallaby.Experimental.Selenium.WebdriverClient.cast_as_element/2
      The following arguments were given to Wallaby.Experimental.Selenium.WebdriverClient.cast_as_element/2:

          # 1
          %Wallaby.Session{driver: Wallaby.Experimental.Chrome, id: "444f39bc45a713370fb47558d9e539ac", screenshots: [], server: Wallaby.Experimental.Chrome.Chromedriver, session_url: "http://localhost:47863/session/444f39bc45a713370fb47558d9e539ac", url: "http://localhost:47863/session/444f39bc45a713370fb47558d9e539ac"}

          # 2
          {"message", "invalid session id\n  (Driver info: chromedriver=72.0.3626.121,platform=Linux 4.14.133-88.105.amzn1.x86_64 x86_64)"}

      Attempted function clauses (showing 2 out of 2):

          defp cast_as_element(parent, -%{"ELEMENT" => id}-)
          defp cast_as_element(parent, -%{"element-6066-11e4-a52e-4f735466cecf" => id}-)

      code: |> assert_has(css(".login"))
      stacktrace:
        (wallaby) lib/wallaby/experimental/selenium/webdriver_client.ex:566: Wallaby.Experimental.Selenium.WebdriverClient.cast_as_element/2
        (elixir) lib/enum.ex:1340: anonymous fn/3 in Enum.map/2
        (stdlib) maps.erl:232: :maps.fold_1/3
        (elixir) lib/enum.ex:1964: Enum.map/2
        (wallaby) lib/wallaby/experimental/selenium/webdriver_client.ex:43: Wallaby.Experimental.Selenium.WebdriverClient.find_elements/2
        (wallaby) lib/wallaby/driver/log_checker.ex:6: Wallaby.Driver.LogChecker.check_logs!/2
        (wallaby) lib/wallaby/browser.ex:1140: anonymous fn/3 in Wallaby.Browser.execute_query/2
        (wallaby) lib/wallaby/browser.ex:146: Wallaby.Browser.retry/2
        test/pricing_engine_web/features/landing_page_test.exs:8: (test)
 ..........

Test Code

Multiple tests are randomly failing, one of them being our easiest:

  test "langing page contains css", %{session: session} do
    session
    |> visit("/")
    |> assert_has(css(".login"))
    |> assert_has(css(".login__header"))
    |> assert_has(css(".login__main"))
    |> assert_has(Query.text("Welcome to Pricing Hub"))
    |> assert_has(link("test__google-auth-link"))
  end

The others are usually simple CRUD tests (app isn't that complex yet) :) Often times 2 of them are failing. Sometimes it's just one.

There is no database access here. Just good old landing page. The session is also piped through from beginning to end so no issues with stale sessions.

Environment

• Wallaby 0.23.0
• Chromedriver configured to use with chromium, Chromedriver 72 on CI, 76 locally (no idea why it's that old on CI...)
• Elixir 1.9.0/1.9.1
• Erlang 22
• On CI tests are running inside a docker container for testing
• Tests are being executed in parallel/async

The whole thing reminds me a bit of #221

Any ideas/help/input on how to remedy this would be highly appreciated 👌

[michallepicki]

Tests are being executed in parallel/async

How many tests are being run in parallel? (what's the --max-cases/:max_cases ExUnit option?)

[PragTob]

@michallepicki it's not set explicitly and I don't know how many cores the CI has off the top of my head and I don't have SSH access

edit:
lscpu says 4 CPUs so probably max-cases is set to 8. We thought about reducing it as we think it might be an issue with overloading chromedriver but not quite sure

[PragTob]

After changing tests to run with --max-cases 2 on CI we haven't seen the failures since. Will continue to observe. So, it might be that it's some sort of problem with overloading or state interfering. Question is just whether it's in wallaby or chromedriver or what not.

For whoever comes afterwards, a team member also had an issue where this occurred all the time but there the issue was non matching chromedriver/chromium versions. That's decidedly not the case on CI as linux package management enforces matching versions there.

WHile our problem seems solved I think I'd like to keep this open because it might still be a problem that's fixable or at least better highlightable in wallaby.

[PragTob]

For the record, the problem popped up again restricting it to --max-cases 2 fixed it again 🤷‍♂️

I'm unsure what this is. Part of me believes chromedriver just dies.

Which is supported by the error I get sometimes being:

 ** (MatchError) no match of right hand side value: {:error, "unknown error: Chrome failed to start: exited abnormally\n  (chrome not reachable)\n  (The process started from chrome location /usr/lib/chromium/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=72.0.3626.121,platform=Linux 5.0.0-29-generic x86_64)"}
app_1       |      stacktrace:
app_1       |        (pricing_engine_web) test/support/feature_case.ex:35: PricingEngineWeb.FeatureCase.__ex_unit_setup_0/1
app_1       |        (pricing_engine_web) test/support/feature_case.ex:1: PricingEngineWeb.FeatureCase.__ex_unit__/2
app_1       |        test/pricing_engine_web/features/pricing_grid_test.exs:1: PricingEngineWeb.PricingGridTest.__ex_unit__/2

[PJUllrich]

Small note for future reference: Make sure that your chromedriver version is the same on your local machine and on your CI. Pin the version on your CI if necessary. Every now and then my CI broke because chromedriver got a new version, which wasn't installed on my local machine, leading to the problem that the tests ran locally, but broke in the CI with the above mentioned error meassage.

[josephan]

I had the same issue as well, however my tests were passing on CI but not local (macOS).
My CI was running chromedriver 79 and my local was running chromedriver 77.
I fixed it by running brew cask upgrade chromedriver

[sveredyuk]

+1 to @PragTob

I had the same issue on GitlabCI and local machine.
Gitlab CI + Wallaby.Experimental.Chrome

[PJUllrich]

I ran into this issue again locally and would like to recommend to verify that your chromedriver has the same version as your Chrome-Browser, if you have one installed. In my case, Chrome updated itself to Version 81, but my chromedriver was still at 79. That lead to the {"error" ,"invalid session id"} error. After updating chromedriver to 81, my tests ran through successfully again.

[bitboxer]

I also ran into this problem and have no clue on how to make chrome+wallaby work on travis ci. Details here. Would love some help.

[eahanson]

I had the same problem in the past, and my fix was what @PJUllrich recommended: make sure Chrome and Chromedriver are at the same version. Here's a bash script that I run as part of my apps' doctor script:

#!/usr/bin/env bash

chromedriver_path=$(command -v chromedriver)

chrome_path="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
chromedriver_version=$("${chromedriver_path}" --version)
chrome_version=$("${chrome_path}" --version)

chromedriver_major_version=$("${chromedriver_path}" --version | cut -f 2 -d " " | cut -f 1 -d ".")
chrome_major_version=$("${chrome_path}" --version | cut -f 3 -d " " | cut -f 1 -d ".")

if [ "${chromedriver_major_version}" == "${chrome_major_version}" ]; then
  exit 0
else
  echo "Wallaby often fails with 'invalid session id' if Chromedriver and Chrome have different versions."
  echo "Chromedriver version: ${chromedriver_version} (${chromedriver_path})"
  echo "Chrome version      : ${chrome_version} (${chrome_path})"
  exit 1
fi

[lpender]

@eahanson Thanks for the script it worked great for me!

Little update for Linux compatibility in CI:

#!/usr/bin/env bash

chromedriver_path=$(command -v chromedriver)

unameOut="$(uname -s)"
case "${unameOut}" in
    Linux*)     chrome_path="/usr/bin/google-chrome";;
    Darwin*)    chrome_path="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";;
esac
chromedriver_version=$("${chromedriver_path}" --version)
chrome_version=$("${chrome_path}" --version)

chromedriver_major_version=$("${chromedriver_path}" --version | cut -f 2 -d " " | cut -f 1 -d ".")
chrome_major_version=$("${chrome_path}" --version | cut -f 3 -d " " | cut -f 1 -d ".")

if [ "${chromedriver_major_version}" == "${chrome_major_version}" ]; then
  echo "Chromedriver matches chrome version ✓"
  exit 0
else
  echo "Wallaby often fails with 'invalid session id' if Chromedriver and Chrome have different versions."
  echo "Chromedriver version: ${chromedriver_version} (${chromedriver_path})"
  echo "Chrome version      : ${chrome_version} (${chrome_path})"
  exit 1
fi

[btihen]

I have the same problem on MacOS X: 11.2.3 (20D91)
I installed using ASDF: cat .tool-versions
erlang 23.3.1
elixir 1.11.4-otp-23
nodejs 14.15.5
Postgres 13.2

wallaby 0.28

I have the same version of chromedriver as my browser
I couldn't get wallaby to find chromedriver in the path until I installed it locally in my project.

If there is a script to change please tell me which file to change - (I am relatively new to Phoenix/Elixir and completely new to Wallaby)

I followed the setup instructions at: https://www.tddphoenix.com/setting-up/

✔ ~/devel/learning/germanvelasco/tdd_phoenix/chatter [install_wallaby L|✚ 9…9]
19:41 $ mix test
..

1. test user can visit homepage (ChatterWeb.UserVisitsHomepageTest)
   test/features/user_visits_homepage_test.exs:4
   ** (RuntimeError) invalid session id
   code: |> visit("/")
   stacktrace:
   (wallaby 0.28.0) lib/wallaby/httpclient.ex:136: Wallaby.HTTPClient.check_for_response_errors/1
   (wallaby 0.28.0) lib/wallaby/httpclient.ex:56: Wallaby.HTTPClient.make_request/5
   (wallaby 0.28.0) lib/wallaby/webdriver_client.ex:329: Wallaby.WebdriverClient.visit/2
   (wallaby 0.28.0) lib/wallaby/driver/log_checker.ex:6: Wallaby.Driver.LogChecker.check_logs!/2
   (wallaby 0.28.0) lib/wallaby/browser.ex:1057: Wallaby.Browser.visit/2
   test/features/user_visits_homepage_test.exs:6: (test)

..

Finished in 3.0 seconds
5 tests, 1 failure

Randomized with seed 381981
✘-1 ~/devel/learning/germanvelasco/tdd_phoenix/chatter [install_wallaby L|✚ 9…9]
19:41 $ chromedriver
Starting ChromeDriver 89.0.4389.23 (61b08ee2c50024bab004e48d2b1b083cdbdac579-refs/branch-heads/4389@{#294}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.