Headless chrome test fails due to incorrect chromedriver version

[thisisrandy]

EDIT: I fibbed a bit on a couple of points (struckout below). Not that it's relevant, but to be clear, chrome is not a snap on Ubuntu, but rather a Debian package (it's chromium that's a snap). Also, I got to wondering why Ubuntu 20.x wasn't on the latest version of chrome, and it turned out I wasn't actually up-to-date after all (whoops). I'm now on 100.0.4896.60, and everything of course works as expected since the driver and browser major versions match.

So I guess the question is now if wasm-pack wants to support users who, intentionally or otherwise, aren't on the latest version. I would argue that being able to test on arbitrary browser versions, within reason, is a very sensible thing to support.

🐛 Bug description

I'm following the wasm rust tutorial, which instructs me to run a simple test using wasm-pack test --chrome --headless. However, the test fails with:

Error: non-200 response code: 404
{"value":{"error":"invalid session id","message":"invalid session id","stacktrace":"#0 0x5581c6dd1ad3 ... (a long list of addresses) }

Following #611, I reran with RUST_LOG=wasm_bindgen_test_runner, which revealed that the incorrect version of chromedriver had been selected:

[2022-03-31T21:31:30Z DEBUG wasm_bindgen_test_runner::headless] got: {"sessionId":"ccc87ee615a5b257d818da42ca72618e","status":33,"value":{"message":"session not created: This version of ChromeDriver only supports Chrome version 100\nCurrent browser version is 99.0.4844.84 with binary path /usr/bin/google-chrome\n  (Driver info: chromedriver=100.0.4896.60 (6a5d10861ce8de5fce22564658033b43cb7de047-refs/branch-heads/4896@{#875}),platform=Linux 5.4.0-105-generic x86_64)"}}

Following the docs, I downloaded the correct version of chromedriver and attempted to use CHROMEDRIVER=path/to/chromedriver, but it seems to be ignored:

[2022-03-31T21:39:00Z DEBUG wasm_bindgen_test_runner::headless] executing "/home/randy/.cache/.wasm-pack/chromedriver-806033fccbd67392/chromedriver" "--port=44385"

Possibly I misunderstood if that variable is read by wasm-pack, seeing as it's in a section entitled "Appendix: Testing in headless browsers without wasm-pack". Having read test.rs, I now see that wasm-pack writes CHROMEDRIVER in order to pass to cargo test rather than reading from it.

🤔 Expected Behavior

If I understood #611 correctly, the fix (#703) was, per @alexcrichton, "Let's download the most recent stable version." I don't think that's actually the correct behavior, though. Instead, the browser version should be detected and the correct chromedriver downloaded as appropriate. I am using the up-to-date chrome snap on Ubuntu 20.04.4 LTS, which is at version 99.0.4844.84 (also seen in logs above).

👟 Steps to reproduce

Attempt to run wasm-pack test --chrome --headless for any test on a system with chrome 99.0.4844.84 (and probably any pre-100.x.x.x version) and observe that the incorrect chromedriver version is used.

🌍 Your environment

Include the relevant details of your environment.
wasm-pack version: 0.10.2
rustc version: 1.57.0
chrome version: 99.0.4844.84, which as above, is the up-to-date version from the snap on Ubuntu 20.04.4

[thisisrandy]

I read chromedriver.rs, and there's a note about chrome version being difficult to determine on windows, which I guess is why the code isn't already doing what I proposed, i.e. detecting the browser version and acting appropriately. With that in mind, I'd propose that

1. Some mechanism of passing in a chromedriver path should be available as a last resort. Perhaps the CHROMEDRIVER env var could be read in and used if available?
2. As a stretch, implement best effort version detection, i.e. punt on Windows, making that clear in the docs, but try for everyone else.

[chris13524]

Some mechanism of passing in a chromedriver path should be available as a last resort. Perhaps the CHROMEDRIVER env var could be read in and used if available?

I think this is the best approach