Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User input simulation produces incorrect mouse movement in scaled/HiDPI modes on macOS #71

Closed
deepfriedmind opened this issue May 5, 2019 · 16 comments

Comments

Projects
None yet
3 participants
@deepfriedmind
Copy link

commented May 5, 2019

System

  • OS name: macOS
  • OS version: 10.14.4
  • Browser name: Chrome
  • Browser version: 74.0.3729.131

Extension

  • Extension version: 0.5.2
  • User input simulation: yes
  • Client app version: 0.2.0
  • Client app installed successfully: yes

Bug description

When I click the "Solve the challenge" button, the mouse cursor moves to the bottom of the screen and clicks. After this, nothing else happens until it throws the error in the console.
I have tried it in incognito mode with no other extensions enabled.
This happens when using either of the default resolutions. window.devicePixelRatio reports 2.
If I use a 3rd party app to set the native resolution (pretty much unusable) the input simulation works.

Logs

Browser:

Uncaught (in promise) TypeError: Cannot read property 'addEventListener' of undefined
    at E (script.js:1)
    at www.google.com/recaptcha/api2/async chrome-extension:/mpbjkejclgfgadiemmefgebjfooflfhl/src/solve/script.js:1
E @ script.js:1

In script.js, it's this event listener:

d.addEventListener("playing", e, {
    capture: !0,
    once: !0
})

Client app:

2019/05/05 17:03:11.007206 Starting client (version: 0.2.0)
2019/05/05 17:03:11.007258 Receiving message
2019/05/05 17:03:11.007375 Processing message
2019/05/05 17:03:11.007387 Command: ping
2019/05/05 17:03:11.007394 Sending response
2019/05/05 17:03:11.007455 Receiving message
2019/05/05 17:03:11.021969 Processing message
2019/05/05 17:03:11.022001 Command: moveMouse
2019/05/05 17:03:11.022010 X: 285, Y: 1397
2019/05/05 17:03:11.411155 Sending response
2019/05/05 17:03:11.411211 Receiving message
2019/05/05 17:03:11.505082 Processing message
2019/05/05 17:03:11.505141 Command: clickMouse
2019/05/05 17:03:11.515950 Sending response
2019/05/05 17:03:11.516047 Receiving message
2019/05/05 17:03:21.521772 Closing client
@dessant

This comment has been minimized.

Copy link
Owner

commented May 5, 2019

I don't have access to macOS, could you test a change?

if (os === 'windows') {

Replace this line with:

  if (['windows', 'macos'].includes(os)) {

Here is a guide for building: https://github.com/dessant/buster/wiki/Building-the-extension-on-Ubuntu

If you don't have your env set up for building, I can share the built package.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 5, 2019

Here's the patched extension: https://www.dropbox.com/s/vozlkbkg3lh8y1c/buster_captcha_solver_for_humans-0.5.2-patch.zip?dl=0

Disable the currently installed extension, unpack the zip into a folder, visit chrome://extensions, turn on developer mode and install the unpacked extension by dropping the folder on the browser window. You'll also have the run the client installer again, because the unpacked extension has a different ID that needs to be whitelisted by the client app.

@deepfriedmind

This comment has been minimized.

Copy link
Author

commented May 5, 2019

I just built the extension with the change and now the mouse cursor moves differently but it's still not correct, i.e. not solving the captcha.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 5, 2019

Does it still move below the target, but closer?

@deepfriedmind

This comment has been minimized.

Copy link
Author

commented May 5, 2019

Yeah, exactly. It actually seems to move slightly differently every time, even hitting the button at times but then failing to complete the challenge.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 5, 2019

Sounds like the change is needed, but not enough. I'll prepare a page for you to test something.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 5, 2019

https://adaptable-trip.glitch.me/

This is the source:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style>
      html {
        height: 100%;
      }
      body {
        margin: 0;
        height: 2000px;
      }
      #notice {
        background-color: #f00;
        width: 100px;
        height: 100px;
        margin: 100px;
      }
    </style>
  </head>
  <body>
    <div id="notice"></div>
    <script>
      window.addEventListener('click', function(e) {
        console.log(window.devicePixelRatio)

        // requires multiplying by scaling type: yes/no
        console.log('click:',
        e.screenX, // os scale: yes, page zoom: no
        e.screenY, // os scale: yes, page zoom: no
        e.clientX, // os scale: yes, page zoom: yes
        e.clientY // os scale: yes, page zoom: yes
        )

        console.log('window:',
        window.screenX, // os scale: yes, page zoom: yes
        window.screenY, // os scale: yes, page zoom: yes
        )

        const {left: x, top: y, width, height} = document.querySelector('#notice').getBoundingClientRect();
        console.log('node:',
        y, // os scale: yes, page zoom: yes
        y, // os scale: yes, page zoom: yes
        width, // os scale: yes, page zoom: yes
        height // os scale: yes, page zoom: yes
        )
      })
    </script>
  </body>
</html>

Perform these steps once with scaling set to 1, then to 2: Maximize the browser window and click the top left corner of the red square. Now move the mouse to the left and click while touching the edge of the screen.

Please open the browser console and share the output.

@deepfriedmind

This comment has been minimized.

Copy link
Author

commented May 5, 2019

Here you go:

2
click: 101 236 101 102
window: 0 23
node: 100 100 100 100
2
click: 0 236 0 102
window: 0 23
node: 100 100 100 100
1
click: 105 240 105 106
window: 0 23
node: 100 100 100 100
1
click: 0 236 0 102
window: 0 23
node: 100 100 100 100
@dessant

This comment has been minimized.

Copy link
Owner

commented May 5, 2019

Thanks a lot, these appear to be expected values. I'm not sure where to go from here, are the clicks off only by a couple of pixels with the patch?

@deepfriedmind

This comment has been minimized.

Copy link
Author

commented May 5, 2019

It usually ends up around here: https://www.dropbox.com/s/hpf84mpmk71pnov/Screenshot%202019-05-05%2022.00.20.png?dl=0

So, yeah, pretty close.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 9, 2019

@deepfriedmind, as a workaround you could substract a couple of pixels from x and y in your local build. Alternatively, you could navigate to the extension button with the keyboard and press Enter to activate keyboard navigation, that way the bug will not be triggered.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 9, 2019

I won't be able to debug this without a device, if anyone would like to work on this issue, please let me know.

@dessant dessant added the help wanted label May 9, 2019

menzow added a commit to menzow/buster that referenced this issue May 14, 2019

@menzow

This comment has been minimized.

Copy link
Contributor

commented May 14, 2019

I've ran into the same issue locally and after some debugging I've found the culprit. In getElementScreenRect the height and width properties are not corrected for higher dpi (2x etc) screens. This results in the button size being returned as 96x96 pixels while on screen the button is only 48x48px.

After dividing the target element width/height by osScale buster is clicking in the correct position again. I'll open up a PR with my changes. Hope this helps.

Edit:
Demonstration of click positions on macos after applying fix:

Click to view

working-buster-fast

@dessant

This comment has been minimized.

Copy link
Owner

commented May 14, 2019

@menzow, that's a great find, thanks for the PR! I'm wondering if we should divide the width and height by scale instead, which also incorporates the zoom level. Could you do a test with the page zoomed to 300% by the browser?

@dessant

This comment has been minimized.

Copy link
Owner

commented May 14, 2019

@menzow, actually dividing by scale would be incorrect, your fix also works great on Windows.

@dessant

This comment has been minimized.

Copy link
Owner

commented May 14, 2019

The fix has been published for Firefox, Chrome and Opera will likely take a couple of days to pass reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.