Make camera overlay window more stable (macOS)

[richiemcilroy]

Issue
It's possible to break the camera window (window turns black) when the app is first launched. Seems to happen if the app has launched in the background, then you open it after the camera preview has already initialised.

You can use this guide to help you get started developing locally as quickly as possible.

[richiemcilroy]

/bounty $25

[algora-pbc]

💎 $100 bounty • Cap

Steps to solve:

1. Start working: Comment /attempt #25 with your implementation plan
2. Submit work: Create a pull request including /claim #25 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to CapSoftware/Cap!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @feliciien	Apr 12, 2024, 6:21:11 PM	WIP
🔴 @Anshgrover23	Jun 27, 2024, 3:29:57 PM	WIP

[feliciien]

/attempt #25

Algora profile	Completed bounties	Tech	Active attempts	Options
@feliciien	5 bounties from 1 project	MDX, Rust, JavaScript & more		Cancel attempt

[richiemcilroy]

Hey @feliciien are you working on this?

[feliciien]

yes Le sam. 13 avr. 2024 à 12:30, Richie McIlroy ***@***.***> a écrit :

…

Hey @feliciien <https://github.com/feliciien> are you working on this? — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3Y7KB5SREEATMZLJBTOSTLY5EQPLAVCNFSM6AAAAABGCIOVTWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJTGYZTGMZZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Anshgrover23]

/attempt #25

Options

• Cancel my attempt

[richiemcilroy]

/bounty $100

[richiemcilroy]

Upped the bounty here, would love to get this one fixed

[Vijaykv5]

Giving it a try @richiemcilroy

/attempt #25

[richiemcilroy]

awesome, thank you @Vijaykv5!

You can refer to this section here in the contributing guide:

https://github.com/CapSoftware/Cap/blob/main/CONTRIBUTING.md#how-do-i-run-the-desktop-app-locally-without-needing-to-use-auth

[algora-pbc]

Here are some steps and pointers to help you get started on resolving this issue:

Steps to Address the Issue

1. Reproduce the Issue:

   • Follow the steps to reproduce the issue: Launch the app in the background and then open it after the camera preview has already initialized.

2. Investigate Initialization Logic:

   • The issue likely stems from the camera initialization logic in the Camera.tsx component. Specifically, the useEffect hook that initializes the camera might not handle the app being in the background properly.

3. Modify Initialization Logic:

   • Ensure that the camera initialization logic checks if the app is in the foreground before attempting to initialize the camera. You can use the document.hidden property to check if the app is in the background.

4. Handle App Visibility Changes:

   • Add an event listener for the visibilitychange event to handle cases where the app moves from the background to the foreground. Reinitialize the camera if necessary.

Relevant Files and Code

• Camera Component: The main logic for the camera overlay is in the Camera.tsx file.

  • Path: /apps/desktop/src/components/windows/Camera.tsx
  • Key section to modify:

    useEffect(() => {
      if (!videoRef.current || !selectedVideoDevice) return;
      const video = videoRef.current;
      const constraints = {
        video: {
          deviceId: selectedVideoDevice.id,
        },
      };
    
      if (typeof navigator === "undefined") return;
    
      const initializeCamera = () => {
        navigator.mediaDevices
          .getUserMedia(constraints)
          .then((stream) => {
            video.srcObject = stream;
            video.play();
            setIsLoading(false);
          })
          .catch((err) => {
            console.error(err);
          });
      };
    
      if (!document.hidden) {
        initializeCamera();
      }
    
      const handleVisibilityChange = () => {
        if (!document.hidden) {
          initializeCamera();
        }
      };
    
      document.addEventListener("visibilitychange", handleVisibilityChange);
    
      return () => {
        document.removeEventListener("visibilitychange", handleVisibilityChange);
        if (video.srcObject) {
          const stream = video.srcObject as MediaStream;
          stream.getTracks().forEach((track) => {
            track.stop();
          });
        }
      };
    }, [selectedVideoDevice]);

• App Initialization: Ensure the app's main page handles the camera window state correctly.

  • Path: /apps/desktop/src/app/page.tsx
  • Ensure that the camera window is properly initialized and managed when the app state changes.

Potential Implications

• Security: Ensure that the camera permissions are handled securely and that the camera is not accessed without user consent.
• Stability: Properly managing the camera initialization based on app visibility will improve the stability of the camera overlay.
• Bugs: Be cautious of potential race conditions or state inconsistencies when handling visibility changes.

Testing

• Test the changes thoroughly by launching the app in various states (foreground, background) and ensuring the camera overlay initializes correctly.
• Verify that the camera permissions are requested and handled appropriately.

[ItsEeleeya]

Is the previous message automated?
I cannot reproduce the issue even when I set the app to be unfocused and minimized on start. It happens at random when I change the input video quickly between 3 devices.

Also, I keep getting the update available message even though my fork is synced with the main branch. (Apparently release is newer than main for now.)

[ItsEeleeya]

Hey @richiemcilroy! Could you please try out this branch and check if it's solved the issue?
If not I'd greatly appreciate it if you could provide a little more info on how to reproduce this.

Also, I came across a few other issues trying this. It seems like switching between cameras doesn't work properly and it gets stuck on the same input the app started with (Same commit, and the latest release).

All Tao windows come into focus by default unless specified. The camera overlay is set to always be on top. After forcing them to go behind everything, the video gets hidden and then starts playing again once the window is visible and this is without any changes required it seems.
Has anyone else reported this same issue?

If the provided fix works, then I'm going to cleanup and re-do some parts of Camera.tsx and MediaDeviceContext.tsx (And probably a few more…)

[Vijaykv5]

Hey @ItsEeleeya
I'm currently working on fixing them and would raise a PR soon

[ItsEeleeya]

Hey @ItsEeleeya I'm currently working on fixing them and would raise a PR soon

Awesome!
It'd be great if you could also fix the device changing issue

its time to go back working on the tauri v2 transition for me 👀

[richiemcilroy]

hey @ItsEeleeya - you want to create your branch as a PR so I can take a look?

[Vijaykv5]

Can we close this issue @richiemcilroy if its fixed!