Run integration tests in the Github Workflows on all platforms

[pentamassiv]

@BrettMayson had the idea to use a browser to run integration tests for the crate and laid the foundation for them. They run on Linux, but not on the other platforms. He said they succeed when ran locally. It would be nice if we could fix the tests so that we can run integration tests on all platforms automatically to get notified about regressions.

[pentamassiv]

I took Brett's commits and made a few changes. The tries can be found here: https://github.com/enigo-rs/enigo/tree/tests

Here are a few things that I found out/tried out:

• removed the browser-actions/setup-firefox Action since Firefox is preinstalled on all the Github Runners (ubuntu, Windows, MacOS)
• replaced the dependency "websocket" with "tungstenite" since "websocket" is deprecated and they suggested using "tungstenite"
• make the tests time out so they don't manually need to be canceled each time
• reverted all changes to win/win_impl.rs so that only the testing issues are addressed
• removed the "test_browser" feature again. Instead the integration test is ignored unless explicitly requested (the Github Workflow explicitly runs it) (--include-ignored)
• changed the "index.html" file so that when the test is ran locally, it is visible which parts of the test succeeded
• the output of the tests is visible in the Github Workflow output to easier debug the problem (--nocapture)
• refactored and cleaned up the code for it to be easier understandable
• the "webbrowser" crate can only open the default browser (source), so on Windows Edge is launched instead (also visible in the Github Workflow output)
• webbrowser::Browser::Firefox.exists() returns false even if Firefox is installed so it cannot be trusted
• removed the pyvista/setup-headless-display-action Action and replaced it with a local Action. I had a look at what the Action does. It looks like it just installs mesa3D on on Windows and on Linux it installs OpenGL and xvfb. Then it sets the DISPLAY env var and creates a virtual display with xvfb. The mesa version for Windows was very outdated.
• Use ssciwr/setup-mesa-dist-win to install a more recent version on Windows. There are a few deployment-choice but I am not sure which one would be correct
• Installing OpenGL (libgl1-mesa-glx) on Linux is not required to make the tests pass so I am not installing that anymore

All these explorations resulted in https://github.com/enigo-rs/enigo/tree/75bb460e7b6ce2a9a1697925a3566a0b73805e1a
It passes the test on Linux but times out on Windows and macOS.

[pythoneer]

Has someone tried to run the windows test locally running inside a VM and not nativ under windows?

[pentamassiv]

In the Github Workflow on Windows (Windows Server 2022) I saw the following output:

     Running tests\browser.rs (target\debug\deps\browser-7ff64108df6d7934.exe)

running 1 test
Created channel
WebSocket server thread was spawned
TcpListener was created
Firefox is not installed
Try opening test page
[7020:1432:0210/174513.474:ERROR:edge_auth_errors.cc(484)] EDGE_IDENTITY: Get Default OS Account failed: Error: Primary Error: kImplicitSignInFailure, Secondary Error: kAccountProviderFetchError, Platform error: 0, Error string: 

[7020:1432:0210/174515.049:ERROR:fallback_task_provider.cc(124)] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
[7020:1432:0210/174515.049:ERROR:fallback_task_provider.cc(124)] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
New connection was made from 127.0.0.1:62438
Start waiting for messages
Start processing message
Message::Text received
msg: "open:"
Test page was opened
Done with launch function
Browser was launched
Move mouse
Executed Enigo
test browser_events has been running for over 60 seconds
Error: The operation was canceled.

On Mac, there were no errors. The test just timed out :(

[pentamassiv]

There is xquartz for Mac and cygwin has the packages xorg-server xorg-server-extra which contain xvfb so in theory it should be possible to use xvfb on all platforms but I tried that and failed. There just isn't any output that I can use to further debug it and that is nothing I am familiar with. Not sure if that is the correct path to continue trying.

Here is some code I had a look at to try to fix it:
https://github.com/tcltk/tk/blob/main/.github/workflows/mac-build.yml
https://github.com/tcltk/tk/blob/main/.github/workflows/win-build.yml
https://github.com/DrylandEcology/SOILWAT2/blob/a0816cbfe8fbc466e195799cf65087248303aa1b/.github/workflows/main_win.yml

[pentamassiv]

@Megamannen , in another issue you mentioned that you wrote a similar library. Were you able to run some integration tests for it?

[pentamassiv]

Has someone tried to run the windows test locally running inside a VM and not nativ under windows?

Yes, I just installed Windows 10 in Gnome Boxes and it works like a charm. It is a bit weird, because calling the functions to move the mouse work, but I don't see it move (I see new coordinates in the Firefox log when I use the "test page" though). I guess that is because the host controls the mouse and not the VM.

Also I installed spice-guest-tools so that I can look at the Windows VM in fullscreen. That seems to mess with the mouse coordinates as well. The absolute ones are fine but the relative ones don't match the numbers I called the function with. The main_display_size() function returns the old dimensions of the "screen" before I installed spice. Maybe that is why they are weird.

[pentamassiv]

There is also https://github.com/sickcodes/Docker-OSX which apparently is able to use xvfb too

[pentamassiv]

I am trying to run the integration test on macOS. There is a popup that says that the code wants to control the computer using accessibility features. I have to grant access in the "Security & Privacy" preferences, located in System Preferences. That might be a reason why it does not work on macOS in the CI?

[pentamassiv]

On the mac I borrowed, enabling full screen is CMD+Shift+F, not F11. That could be another issue

[pentamassiv]

The blocker for macOS is the missing accessibility permission. An explanation about the relevant DB can be found here: https://www.rainforestqa.com/blog/macos-tcc-db-deep-dive
Unfortunately it is protected by macOS's SIP so the DB is read-only. Therefore it currently is impossible to run the tests on macOS. :(
Github thought about disabling SIP, but then decided against it: actions/runner-images#650
More relevant issues: actions/runner-images#3286

The permissions can be checked by running:
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" 'select * from access'
We would have to do something like this:
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "INSERT INTO access VALUES ('kTCCServicePostEvent'|'/bin/bash'|1|2|4|2|||0|'UNUSED'||0|1674645394);"
but the DB is read only due to the SIP protection.

[pythoneer]

wow that is really unfortunate – do we have an alternative like a different CI we can run against?

[pythoneer]

maybe appveyor has SIP disabled in their images? appveyor/ci#3426

[pentamassiv]

I created an issue and asked, if they could add the required permissions to the Github runner: actions/runner-images#7269

[pythoneer]

Looks like this PR as merged.

[pentamassiv]

I tried adding more tests and to enable the integration tests on all platforms. My recent attempt can be found at https://github.com/pentamassiv/enigo/tree/more_test_cases. I am now at a point where moving the mouse and entering keys works. However the integration tests continue to fail under Windows and macOS. I added an action to take screenshots of the CI machine on all platforms. The screenshots are then uploaded as an artifacts so they can be looked at after the CI ran and we get more visibility into the machine.

One of the reasons it failed on Windows was that Edge was used and it opened the first-start page instead of the test page. Now I am trying to open Firefox manually, but for some reason that also does not work and I don't get why.

Maybe somebody else has an idea?

[pentamassiv]

We are so close to being able to run the integration tests in the CI...

[pentamassiv]

It is possible to run macOS in a VM by running the following two commands:

docker pull sickcodes/docker-osx:auto
docker run -it --net=host --device /dev/kvm -p 50922:10022 -v /tmp/.X11-unix:/tmp/.X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" -e TERMS_OF_USE=i_agree -e GENERATE_UNIQUE=true sickcodes/docker-osx:auto

You probably also have to disable xhost security with:
xhost +
WARNING: Only do that if you know what you are doing. You can revert it with:
xhost -