Buggy cursor graphic when capturing the game RuneScape

[windwakr]

Platform

Operating system and version: Windows 7 64bit
OBS Studio version: 25.0.08

Expected Behavior

I expect the cursor to appear correctly in the recorded video.

Current Behavior

The cursor appears buggy in the recorded video.

Steps to Reproduce

1. Download the client for RuneScape and start it up. The game dynamically downloads its cache, so the download here should be small. https://www.runescape.com/download
2. You do not need to make an account or log in, as the issue is apparent at the log in screen.
3. Click the gear in the top right of the login window and make sure custom cursors are enabled.
4. Capture the game and you will see the buggy cursor in the recorded video.

Additional information

This problem has been around for a long time. You can find videos on YouTube from years ago with this buggy cursor.

The game does have the option to disable custom cursors, but that's less than desirable.

[Fenrirthviti]

Can you confirm if this happens on supported operating system, such as Windows 10?

[WizardCM]

I can confirm this occurs on 25.0.8 on Windows 10 2004.

We don't do anything special with cursors, but we're currently tracking cursor issues in #2604

[windwakr]

I believe I've found the source of this problem.
In plugins/win-capture/cursor-capture.c the functions copy_from_color, apply_mask, and bit_to_alpha do not take into account potential padding on the mask. This cursor is 13x19 pixels, but there's 16x19 pixels of data in the mask. There's padding to bmWidthBytes.

Mask data interpreted as 13x19 vs 16x19. Looks familiar?

[windwakr]

In addition to one or more of the above mentioned functions needing fixing, so does get_bitmap_data. It doesn't allocate enough memory for the whole mask.

Edit: I think for this function you could just do unsigned int size = bmp->bmHeight * bmp->bmWidthBytes; instead of all the calculation it's doing.

[windwakr]

Sorry for being a bit spammy, but just wanted to confirm that the previous posts are correct about the source of the issue. I was able to do some actual testing with OBS.
I created a fork at https://github.com/windwakr/obs-studio/tree/cursorfix which shows two changes I made to make this function correctly. I'm not sure if this is the best way to go about this, though.

[WizardCM]

@windwakr Awesome work! My personal suggestion would be to submit a PR (tweak the commit titles to match the guidelines & clang-format the changes), and that way the fix itself can be discussed properly in a PR.