Add captureStream() or Streaming Pipeline to OffscreenCanvas.

[aticzz]

What is the issue with the HTML Standard?

Title:

Request to Add captureStream() or Streaming Pipeline to OffscreenCanvas in Main Thread and Support Streaming from Web Workers via postMessage()

---

Description

In the current implementation of the HTML5 Canvas API, the HTMLCanvasElement provides a method called .captureStream(), which allows the canvas content to be captured and streamed to a MediaStream. This functionality is crucial for use cases such as:

• Video recording
• Live streaming
• Virtual camera feeds
• Real-time broadcasting

However, the OffscreenCanvas API, which enables off-main-thread rendering (e.g., for use in Web Workers), does not support a method like .captureStream() to stream its content. This limitation presents a problem for developers who are using OffscreenCanvas for headless rendering or when offloading canvas processing to Web Workers, as there's no direct way to capture or stream the rendered content.

While it's possible to pass image data from a worker to the main thread using postMessage(), the inability to directly stream from OffscreenCanvas in both the main thread and Web Workers adds unnecessary complexity and reduces performance efficiency.

Problem

• No .captureStream() or streaming method for OffscreenCanvas, meaning developers cannot directly stream rendered content from a worker thread.
• Workarounds required such as rendering to a temporary DOM canvas or using postMessage() to pass image data between the worker and the main thread.
• Inability to leverage media-related APIs like MediaStream within Web Workers, making it more difficult to build real-time streaming and video recording applications.

Feature Request

We propose adding native support for streaming from OffscreenCanvas in both the main thread and from Web Workers via an efficient, standardized pipeline. This could be achieved by:

1. Adding .captureStream() support to OffscreenCanvas in the main thread, just like the existing support in HTMLCanvasElement.

   • This would allow OffscreenCanvas to seamlessly integrate into media workflows without requiring a DOM canvas intermediary.

2. Allowing streaming to be passed from Web Workers via postMessage(), enabling a straightforward mechanism to transmit rendered content from a worker to the main thread for streaming purposes.

Use Cases

• Real-time media applications such as video games or screen recording tools, where performance and low-latency streaming are critical.
• Virtual cameras that need to generate video streams in off-main-thread environments (e.g., Web Workers) for applications like video conferencing.
• Live streaming and broadcasting platforms that require a real-time, headless rendering pipeline for efficient content streaming.
• Media processing tasks where OffscreenCanvas is used to manipulate images or video, but the result needs to be streamed in real-time.

Why This is Important

• Performance Optimization: By allowing OffscreenCanvas to directly stream content to a MediaStream, we can ensure that the system utilizes optimized rendering pipelines and hardware acceleration without requiring unnecessary DOM interaction, improving overall performance.

• Streamlined Developer Experience: Adding streaming support for OffscreenCanvas directly would make it easier for developers to implement real-time media workflows without having to resort to inefficient workarounds (e.g., using a DOM canvas intermediary or complex messaging mechanisms).

• Consistency Across Canvas Types: Developers would be able to use a consistent API for both HTMLCanvasElement and OffscreenCanvas, simplifying development and reducing fragmentation in media-related workflows.

Clarification

While it is technically possible to send image data from a Web Worker to the main thread using postMessage(), this method is inefficient for real-time streaming. The main issue is that Web Workers do not have access to media APIs like MediaStream. Therefore, developers are left with the task of transferring data via messages between threads, which incurs overhead and complicates the process. Additionally, there's no efficient built-in way to stream OffscreenCanvas content directly, either in the main thread or from a worker.

The ideal solution would be to enable both:

• Streaming support directly in the main thread for OffscreenCanvas, just as with HTMLCanvasElement, using methods like .captureStream().
• A mechanism for transferring stream data from Web Workers to the main thread, utilizing the efficient streaming pipeline that works with MediaStream.

Example of Current Workaround

Currently, developers must render OffscreenCanvas to a temporary DOM canvas to use .captureStream(), leading to unnecessary complexity and performance hits:

const offscreenCanvas = new OffscreenCanvas(800, 600);
const tempCanvas = document.createElement('canvas');
const ctx = tempCanvas.getContext('2d');

function render() {
  // Draw on OffscreenCanvas
  ctx.drawImage(offscreenCanvas, 0, 0);

  // Capture stream from temporary DOM canvas
  const stream = tempCanvas.captureStream();
  // Now stream the captured content to a video element or other application
}

[Kaiido]

Note that the current HTMLCanvasElement#captureStream() is not specced here but at https://w3c.github.io/mediacapture-fromelement/#dom-htmlcanvaselement-capturestream

Then, for OffscreenCanvas, per specs¹, you can actually call captureStream() from the placeholder <canvas> (on which you called transferToOffscreen()), or for 100% non-DOM setups, you can feed a VideoTrackGenerator with VideoFrame objects generated from your OffscreenCanvas.

Footnotes

1. Firefox throws when trying to do so, and Safari seems to have a bug where they lose the display of the placeholder <canvas> for the first frame, if still. ↩

[annevk]

Can you please refile this using the correct template: "New feature". And please read some of the recommended background reading as well.