chrome tabCapture example

[ddenis1994]

hi am tiring to use the tabCapture API without success for a few days now

what I tried
running the tab capture in the content - results in undefined to the API
running the tab capture inside an iframe that the src is HTML that accessible to the extension (iFrame.src = chrome.runtime.getURL("gif.html");) -result in an error when the user clicks on a button to initiate the recorded Error starting tab capture

maybe someone can share code example for using the tab capture API in manifest version 3

thanks

[ddenis1994]

I am trying to capture the tab video for later edit share and upload
in the end, I solved it by creating streamID inside the popup page and pass it to the content script for capturing .

[kwertop]

HI @ddenis1994, @guest271314
Could you please post any sample code for fetching mediaStream in content script. I've managed to get streamId from popup.html but when I'm doing:

navigator.mediaDevices.getUserMedia({ audio: { mandatory: { chromeMediaSource: 'desktop', //also tried by setting this to 'tab' chromeMediaSourceId: streamId } } }, (tabStream) => { // })

in the content script, I'm getting DOMException: Requested device not found error for chromeMediaSource = desktop while DOMException: Error starting tab capture error for chromeMediaSource = tab .

[kwertop]

@guest271314 I'm trying to capture audio in any live web based meeting conference app playing in a chrome tab. I'm able to capture user's audio input but I also need to capture the system audio.

[kwertop]

@guest271314 By system audio, I meant the output audio of the tab. Like a speaker in a zoom meeting.

[kwertop]

I tried using getDisplayMedia(). It throws the following error:
Failed to execute 'getDisplayMedia' on 'MediaDevices': Access to the feature "display-capture" is disallowed by permission policy
desktopCapture permission is present in the manifest.json file.

[kwertop]

There isn't any <audio> element in the page source of a zoom call (not sure about AudioContext). This is what I'm trying:

navigator.mediaDevices.getUserMedia({
  audio: {
    mandatory: {
      chromeMediaSource: 'tab',
      chromeMediaSourceId: streamId // this is passed from popup.html using chrome.runtime.sendmessage
    }
  }
}, (tabStream) => {
    // do something with tabStream
});

[LiuShuaiyi]

Is there anyway to start capturing the current tab directly (video only)? If I understand correctly navigator.mediaDevices.getUserMedia always pops up a window for users to select what to capture. What if I don't want that window? What if I just want to start capture the current active tab right away?

[LiuShuaiyi]

Okay I finally got a working prototype with the same approach as said in OP's comment: #627 (comment).

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

[donething]

@donething How does the user activate the popup? Can you post you complete code here?

I'm sorry, I had deleted those code, I cannot post them now.

I had test #627 (comment) on my brower, It works.

Thank you.

[zhw2590582]

@LiuShuaiyi @ddenis1994 After I use the method you mentioned, I can successfully get the audio, but the page will become silent, I want it to continue to play the sound, do you have a way to do this

[zhw2590582]

@guest271314 @ddenis1994 @donething

test-the-audio.zip
You can download and test it, I don't know where I'm doing wrong, can you help me take a look, thanks a lot:

manifest.json

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "tabCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tabId = tabs[0].id;
    // Get the streamId
    chrome.tabCapture.getMediaStreamId(
      {
        consumerTabId: tabId,
      },
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

  console.log("stream--->", stream);

  // Now I successfully recorded the sound, but the page is also muted
  // -------------------------------------------------------------------
  // Connected to an audio player, but still no sound, why ?????????????
  // -------------------------------------------------------------------

  const audioContext = new AudioContext();
  const mediaStream = audioContext.createMediaStreamSource(stream);
  mediaStream.connect(audioContext.destination);
});

[zhw2590582]

@guest271314

The following error can be ignored

Uncaught (in promise) Error: The message port closed before a response was received.

test-the-audio-v2.zip

I use the code you said, the audio can be successfully recorded, and the page can play the sound normally

But I don't want the browser to evoke the tab selection popup again, I want to record directly on the current page

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "desktopCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tab = tabs[0];
    const tabId = tab.id;

    // Get the streamId from desktopCapture
    chrome.desktopCapture.chooseDesktopMedia(
      ["tab", "audio"],
      tab,
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

content.js

const audioContext = new AudioContext();

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      mandatory: {
        chromeMediaSource: "screen",
        chromeMediaSourceId: streamId,
      },
    },
    audio: {
      mandatory: {
        chromeMediaSource: "desktop",
        chromeMediaSourceId: streamId,
      },
    },
  });

  stream.removeTrack(stream.getVideoTracks()[0]);

  console.log("stream--->", stream);

  // Here you can successfully record the sound, and the page will not mute
  // -------------------------------------------
  // If I don't create the recorder node, the recording is garbled, it's weird
  // -------------------------------------------
 
  const mediaStream = audioContext.createMediaStreamSource(stream);

  const recorder = audioContext.createScriptProcessor(0, 1, 1);
  recorder.onaudioprocess = (event) => {
    const inputData = event.inputBuffer.getChannelData(0);
    console.log("audio--->", inputData);
  };

  mediaStream.connect(recorder);
  recorder.connect(audioContext.destination);
});

[zhw2590582]

@guest271314 ok, thanks a lot for your help

[killergerbah]

Using @zhw2590582 's work as a reference (https://github.com/zhw2590582/chrome-audio-capture) I re-implemented audio capture when I upgraded to MV3 (https://github.com/killergerbah/asbplayer/blob/main/extension/src/services/BackgroundPageAudioRecorder.ts). The trick is in recognizing that you can access the tabCapture API from an inactive tab created from HTML packaged with the extension. The UX is still worse than MV2 since you need the extra tab but with some effort it's possible to keep that tab somewhat "invisible."

[Deewde]

Okay I finally got a working prototype with the same approach as said in OP's comment: #627 (comment).

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

But then the original tab sound is muted while recording and there's no way to unmute it. The sound is present in the recording, though. This approach is unusable for any use case I can think of.

[Deewde]

Apologies if I wasn't clear enough - I meant "the original live sound of the tab being recorded" is being muted. I've posted a detailed description of the bug on the Chromium Extensions group.

[Deewde]

@guest271314 Here you go, the more detailed description on StackOverflow. I only want to use getDisplayMedia/desktopCapture as a last resort because of the intrusive prompts. I've also tried to record the speaker output, but that failed too. I see you've also contributed to that exploration yourself, maybe you can provide more details on how you managed to get it working. Didn't work for me.

[Deewde]

Ok, so not working. I aim to upload this to the store, so any client-side changes are out of the question. How about the bug I posted here, what do you think about that? Can you reproduce it yourself? In essence passing a stream id from an iframe to a content script, then using getUserMedia to get that stream, resulting in a good recording but a muted tab.

[Deewde]

I will provide an example as soon as I'm able to. I plan on launching this on the store, so no client-side modifications, including launching with flags.

…

On Fri, Dec 30, 2022, 16:22 guest271314 ***@***.***> wrote: @Deewde <https://github.com/Deewde> To avoid prompts altogether you can 1. Set the default audio device that you intend to capture; 2. Launch Chrome with --use-fake-ui-for-media-stream flag I am not sure what is causing the muting behaviour you described without a repoducible example. — Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHKFQZQNHN3BHJMGJWERQBTWP3V2RANCNFSM5CAGIHZQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Deewde]

I already posted a how to reproduce on StackOverflow, but you require a full-working minimal example, which I can't provide without putting in some time. I will, though.

…

On Fri, Dec 30, 2022, 16:58 guest271314 ***@***.***> wrote: I will provide an example as soon as I'm able to. You should be able to post the code you have so far that causes the muting behaviour. Note, a user can modify the extension once it is on their machine. — Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHKFQZTUAAEX32SKBG3J56DWP32ANANCNFSM5CAGIHZQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cacosandon]

@mirkonasato you can use this example I created: https://github.com/wireworks-app/chrome-screen-recording

[mhmohebbi]

was anyone able to solve this issue to keep the tab unmuted live while it records tab audio?

[maltoze]

https://github.com/GoogleChrome/chrome-extensions-samples/tree/main/functional-samples/sample.tabcapture-recorder

[KiranNadig62]

How do we get access to Media from the user's microphone using the offscreen API?
I am talking about something like this.

function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((response) => {
        if (response.id) 
            resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission:", error);
        if (error.message === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        }
        reject(error);
      });
  });
}

Currently, this throws a NotAllowedError and we are not sure how to proceed.

[darrenhollick]

How do we get access to Media from the user's microphone using the offscreen API? I am talking about something like this.

function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((response) => {
        if (response.id) 
            resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission:", error);
        if (error.message === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        }
        reject(error);
      });
  });
}

Currently, this throws a NotAllowedError and we are not sure how to proceed.

I am having the exact same issue. I am able to successfully use an offscreen document to capture and record a tab. However, I want to include the users commentary from their microphone while using that tab and I can't figure out how to get a stream of the microphone. If I could get the stream I could mix the tracks with the tracks from the tab capture. However calling the following code in the offscreen document produces a NotAllowedError.

const microphoneMedia = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: false,
});

Is there a way to get a streamId of the microphone to pass to the offscreen document similar to the same way we're getting a stream id for the tab?

[juxnpxblo]

@KiranNadig62 @darrenhollick

How do we get access to Media from the user's microphone using the offscreen API? I am talking about something like this.

function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((response) => {
        if (response.id) 
            resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission:", error);
        if (error.message === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        }
        reject(error);
      });
  });
}

Currently, this throws a NotAllowedError and we are not sure how to proceed.

I am having the exact same issue. I am able to successfully use an offscreen document to capture and record a tab. However, I want to include the users commentary from their microphone while using that tab and I can't figure out how to get a stream of the microphone. If I could get the stream I could mix the tracks with the tracks from the tab capture. However calling the following code in the offscreen document produces a NotAllowedError.

const microphoneMedia = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: false,
});

Is there a way to get a streamId of the microphone to pass to the offscreen document similar to the same way we're getting a stream id for the tab?

you can get microphone stream on offscreen if user had previously consented to microphone permission for your extension

on my demo using an action popup, I setup a popup.html with a script tag sourcing from popup.js, and in popup.js I had a getUserMedia({ audio: true }) call, which triggered the permission prompt my-extension wants to use your microphone on first time

after I consented, I was able to call getUserMedia({ audio: true }) and successfully get a microphne stream; before consenting, I would get a DOMException: Failed due to shutdown

apparently offscreen documents can't invoke permission prompt, so it must be asked somewhere else, be it in an action popup, iframe inside a content script, options page, or whatever context that can invoke permissions prompt on behalf of your extension (note that it must be get on behalf of your extension, if you ask it on a content script for example, you'll get "google.com wants to use your microphone" and that won't give your offscreen the ability to get microphone stream)

[patrickkettner]

@juxnpxblo would you be willing to file a bug on crbug about that? i.e. offscreen document not showing a permission prompt (but working if permission was already given?

[patrickkettner]

@juxnpxblo ping

[aziz-boudi4]

Okay I finally got a working prototype with the same approach as said in OP's comment: #627 (comment).

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

@LiuShuaiyi are you able to record the content of the tab using navigator.mediaDevices.getUserMedia, I'm only able to get the webcam stream and when I use these constraints (below), they don't work as they are not accepted by navigator.mediaDevices.getUserMedia

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      mandatory: {
        chromeMediaSource: "screen",
        chromeMediaSourceId: streamId,
      },
    },
    audio: false
  });

[Sounav-unstudio]

Did anybody manage to get @LiuShuaiyi 's solution working? For me the part where I send the streamID to my content script and try to capture the screen, I get the DOM Exception: Error starting tab capture
Here's my content script code

`if(request.action === "startRecording"){

  console.log('Start recording message received in content.js ', request);
  var streamId = request.streamId;
  console.log('Stream ID ', streamId);
  
  const constraints = {
    audio: {
      mandatory: {
        chromeMediaSource: 'tab',
        chromeMediaSourceId: streamId,
      }
    },
    video: {
      mandatory: {
        chromeMediaSource: 'tab',
        chromeMediaSourceId: streamId,
      }
    }
  };
  console.log('Constraints ', constraints);
  const media = await navigator.mediaDevices.getUserMedia(constraints);

`
The error comes at the last line.
Any help is appreciated!

[aziz-boudi4]

@Sounav-unstudio

Yes I managed to make it work, you have to add as any at the end of the constraints object

 const constraints = {
    audio: {
      mandatory: {
        chromeMediaSource: 'tab',
        chromeMediaSourceId: streamId,
      }
    },
    video: {
      mandatory: {
        chromeMediaSource: 'tab',
        chromeMediaSourceId: streamId,
      }
    }
  } as any

[Sounav-unstudio]

Hey @aziz-boudi4 but I am not using TypeScript. How does this work in JavaScript?

P.S. It didn't work since my code is in JavaScript

[Sounav-unstudio]

The previous issue got fixed. The current issues are the following.

1. Even after getting permission to record the audio of the microphone, the audio in the recorded video is muted. I am getting the stream id and passing it down to my content script. I do not get any errors in my console. Still the recorded video has got no audio whatsoever.

2. The quality of the video is really bad. This is my configuration for the same
   const constraints = { audio: { mandatory: { chromeMediaSource: 'tab', chromeMediaSourceId: streamId, echoCancellation: true, noiseSuppression: true, sampleRate: 44100, }, }, video: { mandatory: { chromeMediaSource: 'tab', chromeMediaSourceId: streamId, maxWidth: window.innerWidth, maxHeight: window.innerHeight, minWidth: window.innerWidth, minHeight: window.innerHeight, minFrameRate: 30, maxFrameRate: 60, }, }, };

   I want the maxWidth and maxHeight properties to be the same as the dimensions of the recording tab. How can I solve both of these issues?