Corrupt frame data after screen capture (macOS)

[richiemcilroy]

Description
Experiencing a bug with frame data (from screen capture) being corrupt on some Mac's. Seems to work fine on my 16" MacBook Pro, but when testing on 14" MacBook Pro's the frame image looks corrupt.

My thoughts
I believe this possibly stems from the stride calculation needing to be altered. This is because the frame data can be visible if you mess around with the stride settings - it's not always completely corrupt. The pixel data can be shown, it's just a matter of how it is represented. Also, it may be to do with the scaling factor of the device.

Example of a corrupted frame

Device Specs
14" MacBook Pro
M1 Pro Chip
Sonoma 14.4 (also tested on early versions, e.g. 13.1)

Resolutions tested
1800x1169 (also 2x scale)
1512x982 (also 2x scale)
1352x878 (also 2x scale)

Suspect code
The code for this is located at media.rs which belongs to /apps/desktop: https://github.com/CapSoftware/cap/blob/main/apps/desktop/src-tauri/src/media.rs

The line you're looking for is line 356, match capturer.frame() {. In here, we are iterating over each row and extending the slice to send to video_channel_sender. This then feeds into the video ffmpeg command, at line 437.

The raw frame should be packed bgra (ARGB8888). If you search for ARGB8888 in the codebase, you'll find the capturer package, which ultimately creates the core graphics display stream via CGDisplayStreamCreateWithDispatchQueue.

[algora-pbc]

💎 $200 bounty created by CapSoftware
🙋 If you start working on this, comment /attempt #17 along with your implementation plan
👉 To claim this bounty, submit a pull request that includes the text /claim #17 somewhere in its body
📝 Before proceeding, please make sure you can receive payouts in your country
💵 Payment arrives in your account 2-5 days after the bounty is rewarded
💯 You keep 100% of the bounty award
🙏 Thank you for contributing to CapSoftware/cap!

👉 Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @filleduchaos	Mar 24, 2024, 4:43:08 AM	#18

[shamun-khatri]

@richiemcilroy I want to work on this but is this only related to 14" MacBook Pro and are other users also facing the same prob@richil em on any other models?

[richiemcilroy]

Hey! I can't be sure that it's definitely only 14". It seems to be on retina devices, but for some reason not my 16" 🤔 so it's pretty strange. > @richiemcilroy I want to work on this but is this only related to 14" MacBook Pro and are other users also facing the same prob@richil em on any other models?

[jordan-ae]

@richiemcilroy is there anyway to reproduce this?

[richiemcilroy]

@richiemcilroy is there anyway to reproduce this?

Yeah if you have a 13" MacBook Pro you should be able to directly reproduce (or via an emulator maybe?). If you follow the contribution guide found in the readme you should be able to get started.

If you want to skip all the auth process etc, you could just manually run the desktop app in apps/desktop, and comment out the component in /apps/desktop/src/page.tsx

[richiemcilroy]

hey everyone! Updated the contributing guide to show how you can get setup to develop the Cap desktop app locally. Does not require any external .env var dependencies:

https://github.com/CapSoftware/cap/blob/main/CONTRIBUTING.md

[Bhavyajain21]

hey everyone! Updated the contributing guide to show how you can get setup to develop the Cap desktop app locally. Does not require any external .env var dependencies:

https://github.com/CapSoftware/cap/blob/main/CONTRIBUTING.md

export const resend = new Resend(process.env.RESEND_API_KEY);

It still requires the RESEND_API_KEY?

[filleduchaos]

/attempt #17

Instead of the app only calculating the stride by itself (which is prone to error because of padding for alignment at different native resolutions on macOS), I'm switching to letting the underlying OS framework report how many bytes per row each frame contains (and using that in tandem with the calculated stride to derive an image without padding).

Options

• Cancel my attempt

[algora-pbc]

💡 @filleduchaos submitted a pull request that claims the bounty. You can visit your bounty board to reward.

[shamun-khatri]

@filleduchaos Hey How have you done that? mean I am new to open source. And wants to know how you find that solution and write code.

[algora-pbc]

🎉🎈 @filleduchaos has been awarded $200! 🎈🎊

• Share on socials
• Give feedback

[richiemcilroy]

Amazing work @filleduchaos - rewarded!