Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merging media fragments from different input containers to a single output container #3

Merged
merged 4 commits into from Aug 28, 2019

Conversation

@guest271314
Copy link
Contributor

commented Jun 26, 2019

No description provided.

guest271314 added 2 commits Jun 26, 2019
@@ -51,6 +51,7 @@ Provide web apps with efficient access to built-in (software and hardware) media
- control over buffer behavior
- spatial and temporal scalability
- Decoded and encoding images
- Merging multiple media fragments (audio and video) originating from different input containers and codecs to a single container having a single codec

This comment has been minimized.

Copy link
@pthatcherg

pthatcherg Aug 15, 2019

Collaborator

Is this something that can be done purely in JS without involved from the browser?

This comment has been minimized.

Copy link
@guest271314

guest271314 Aug 16, 2019

Author Contributor

Web Audio API has a decodeAudioData() and OfflineAudioContext.startRendering() methods. Both methods output an AudioBuffer. The latter method description includes the language

(potentially) faster than real-time. It does not render to the audio hardware, but instead renders as quickly as possible

see WebAudio/web-audio-api#1527, https://discourse.wicg.io/t/offlinemediacontext/3814.

For video, technically any video that can be played at HTML <video> element can be merged into a single track using various browser APIs, including but not limited to canvas.captureStream(), WebRTC RTCSender.replaceTrack(), and navigator.mediaDevices.getDisplayMedia(), all using MediaRecorder, see the branches at https://github.com/guest271314/MediaFragmentRecorder.

If gathering the question correctly, for the requirement of not using browser APIs, specified and shipped as such, see e.g., https://github.com/thenickdude/webm-writer-js, https://github.com/tseylerd/Webm-writer, https://github.com/brion/ogv.js, https://github.com/GoogleChromeLabs/webm-wasm.

W3C MediaSource does have the ability to playback different codecs following the implementation of changeType(). Piping media through MediaRecorder to get WebM files also provides a means to record output of MediaSource SourceBuffer to a single file at Firefox. However, Chromium still has the issue of crashing when captureStream() method of the <video> element where src is set to MediaSource instance is executed w3c/media-source#190, https://bugs.chromium.org/p/chromium/issues/detail?id=943484.

MediaRecorder is not currently specified to continue recording when the src of a <video> changes, e.g., to a different container and codec. MediaRecorder is specified to stop when MediaStreamTrack are added or removed from the MediaStream. One workaround is to utilize WebRTC with replaceTrack() https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-replacetrack, see https://github.com/guest271314/MediaFragmentRecorder/blob/webrtc-replacetrack/MediaFragmentRecorder.html, https://github.com/guest271314/MediaFragmentRecorder/blob/webrtc-replacetrack-canvas-webaudio/MediaFragmentRecorder.html, https://github.com/guest271314/MediaFragmentRecorder/blob/webrtc-replacetrack-htmlmediaelement-capturestream-addtrack/MediaFragmentRecorder.html.

The above approaches all have the restriction of requiring playback of the media, whether a single or multiple container and codec types, meaning at best "real-time", recording. If this repository and specification is intended to be different from the above mentioned browser APIs and workarounds, then it should be possible to input any container or codec that the browsers' internal Media Decoder can parse and Web Media Player can play

(potentially) faster than real-time.

Am still not sure what the expected output of the final version of the code to be specified for web-codecs is? This PR was filed intended to convey the concept described at https://discourse.wicg.io/t/offlinemediacontext/3814 before filed that topic.

This comment has been minimized.

Copy link
@guest271314

guest271314 Aug 16, 2019

Author Contributor

Note also that when the requirement is to merge audio and video tracks, even when the input is from a single container and codec, the track order cannot be omitted from consideration, see w3c/mediacapture-main#611.

This comment has been minimized.

Copy link
@pthatcherg

pthatcherg Aug 16, 2019

Collaborator

Sorry, I missed the part about "one codec" at the end of the line. I thought you mean combining containers without reencoding, but you're talking about reencoding as well. Yes, this API could be used for reencoding. Perhaps we could rewrite this line to make it more clear that the intention is to reencode in order to merge many encoded media streams into one encoded media stream.

This comment has been minimized.

Copy link
@guest271314

guest271314 Aug 16, 2019

Author Contributor

Yes, this API could be used for reencoding. Perhaps we could rewrite this line to make it more clear that the intention is to reencode in order to merge many encoded media streams into one encoded media stream.

Clarity is helpful.

The language in your comment can be used.

Note also #3 (comment) relevant to Explainer

const muxer = ...;  // Writes frames into container
const output = ...;  // Writes container to source (like a file)
// code
.pipeTo(muxer.audio);
// code
.pipeTo(muxer.video);
// code
muxer.readable.pipeInto(output.writable);

there should be options and a method to set the track order at output. Else implementers could set the tracks to any arbitrary order, which would require parsing output in order to determine if audio is at id 0 or 1 when merging output with a different media file where the track order is known and consistent, e.g., https://github.com/guest271314/native-messaging-mkvmerge/blob/master/app/native-messaging-mkvmerge.js#L186 through https://github.com/guest271314/native-messaging-mkvmerge/blob/master/app/native-messaging-mkvmerge.js#L235.

guest271314 added 2 commits Aug 24, 2019

@pthatcherg pthatcherg merged commit 6c2d7cc into WICG:master Aug 28, 2019

@guest271314

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.