Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Add support for MediaRecorder pause/resume
https://bugs.webkit.org/show_bug.cgi?id=217375

Reviewed by Eric Carlson.

LayoutTests/imported/w3c:

* web-platform-tests/mediacapture-record/MediaRecorder-pause-resume-expected.txt:
* web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-expected.txt:
* web-platform-tests/mediacapture-record/idlharness.window-expected.txt:

Source/WebCore:

Implement pause and resume as per spec.
MediaRecorder basically sends pause/resume order to its backend.
The backend then stops observing tracks when paused and resumed observing at resume time.
For video, we make sure to compute the frame timestamp so that the recorded video continues to play without interruption.

Test: http/wpt/mediarecorder/pause-recording.html

* Modules/mediarecorder/MediaRecorder.cpp:
(WebCore::MediaRecorder::pauseRecording):
(WebCore::MediaRecorder::resumeRecording):
* Modules/mediarecorder/MediaRecorder.h:
* Modules/mediarecorder/MediaRecorder.idl:
* platform/mediarecorder/MediaRecorderPrivate.cpp:
(WebCore::MediaRecorderPrivate::pause):
(WebCore::MediaRecorderPrivate::resume):
* platform/mediarecorder/MediaRecorderPrivate.h:
* platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
(WebCore::MediaRecorderPrivateAVFImpl::pauseRecording):
(WebCore::MediaRecorderPrivateAVFImpl::resumeRecording):
* platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
* platform/mediarecorder/MediaRecorderPrivateMock.h:
* platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h:
* platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:
(WebCore::MediaRecorderPrivateWriter::appendVideoSampleBuffer):
(WebCore::MediaRecorderPrivateWriter::pause):
(WebCore::MediaRecorderPrivateWriter::resume):

Source/WebKit:

Add IPC support for sending pause/resume orders.

* GPUProcess/webrtc/RemoteMediaRecorder.cpp:
(WebKit::RemoteMediaRecorder::pause):
(WebKit::RemoteMediaRecorder::resume):
* GPUProcess/webrtc/RemoteMediaRecorder.h:
* GPUProcess/webrtc/RemoteMediaRecorder.messages.in:
* WebProcess/GPU/webrtc/MediaRecorderPrivate.cpp:
(WebKit::MediaRecorderPrivate::pauseRecording):
(WebKit::MediaRecorderPrivate::resumeRecording):
* WebProcess/GPU/webrtc/MediaRecorderPrivate.h:

LayoutTests:

fix-217375

* http/wpt/mediarecorder/pause-recording-expected.txt: Added.
* http/wpt/mediarecorder/pause-recording.html: Added.


Canonical link: https://commits.webkit.org/230199@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@268130 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
youennf committed Oct 7, 2020
1 parent 7237340 commit a182343
Show file tree
Hide file tree
Showing 25 changed files with 302 additions and 34 deletions.
12 changes: 12 additions & 0 deletions LayoutTests/ChangeLog
@@ -1,3 +1,15 @@
2020-10-07 Youenn Fablet <youenn@apple.com>

Add support for MediaRecorder pause/resume
https://bugs.webkit.org/show_bug.cgi?id=217375

Reviewed by Eric Carlson.

fix-217375

* http/wpt/mediarecorder/pause-recording-expected.txt: Added.
* http/wpt/mediarecorder/pause-recording.html: Added.

2020-10-07 Per Arne Vollan <pvollan@apple.com>

[macOS] Remove 'com.apple.cookied' from the WebContent process sandbox.
Expand Down
@@ -0,0 +1,4 @@


PASS Pausing and resuming the recording should impact the video duration

48 changes: 48 additions & 0 deletions LayoutTests/http/wpt/mediarecorder/pause-recording.html
@@ -0,0 +1,48 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>A recorded muted audio track should produce silence</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
</head>
<body>
<video id="video1" controls></video>
<script>
if (window.internals)
window.internals.setUseGPUProcessForWebRTC(false);

function waitFor(duration)
{
return new Promise((resolve) => setTimeout(resolve, duration));
}

promise_test(async (test) => {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true });

const recorder = new MediaRecorder(stream);
const dataPromise = new Promise(resolve => recorder.ondataavailable = (e) => resolve(e.data));

const startPromise = new Promise(resolve => recorder.onstart = resolve);
recorder.start();
await startPromise;

setTimeout(() => recorder.pause(), 50);
setTimeout(() => recorder.resume(), 950);

await waitFor(1000);
recorder.stop();
const blob = await dataPromise;

const url = URL.createObjectURL(blob);
video1.src = url;
await video1.play();

assert_less_than(video1.duration, 0.5);

URL.revokeObjectURL(url);
}, "Pausing and resuming the recording should impact the video duration");

</script>
</body>
</html>
11 changes: 11 additions & 0 deletions LayoutTests/imported/w3c/ChangeLog
@@ -1,3 +1,14 @@
2020-10-07 Youenn Fablet <youenn@apple.com>

Add support for MediaRecorder pause/resume
https://bugs.webkit.org/show_bug.cgi?id=217375

Reviewed by Eric Carlson.

* web-platform-tests/mediacapture-record/MediaRecorder-pause-resume-expected.txt:
* web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-expected.txt:
* web-platform-tests/mediacapture-record/idlharness.window-expected.txt:

2020-10-07 Youenn Fablet <youenn@apple.com>

MediaRecorder::create should not need to create a MediaRecorderPrivate to validate it can record properly
Expand Down
@@ -1,3 +1,3 @@

FAIL MediaRecorder handles pause() and resume() calls appropriately in state and events promise_test: Unhandled rejection with value: object "TypeError: recorder.pause is not a function. (In 'recorder.pause()', 'recorder.pause' is undefined)"
PASS MediaRecorder handles pause() and resume() calls appropriately in state and events

@@ -1,23 +1,21 @@


Harness Error (TIMEOUT), message = null

PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format [passthrough].
PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format [passthrough].
PASS PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format [passthrough].
PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format [passthrough].
PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough].
TIMEOUT PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough]. Test timed out
NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8.
NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8.
NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8.
NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp8.
NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp8.
NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp8.
NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp9.
NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp9.
NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp9.
NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp9.
NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp9.
NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp9.
PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough].
FAIL PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"

Expand Up @@ -13,16 +13,16 @@ PASS MediaRecorder interface: attribute state
PASS MediaRecorder interface: attribute onstart
PASS MediaRecorder interface: attribute onstop
PASS MediaRecorder interface: attribute ondataavailable
FAIL MediaRecorder interface: attribute onpause assert_true: The prototype object must have a property "onpause" expected true got false
FAIL MediaRecorder interface: attribute onresume assert_true: The prototype object must have a property "onresume" expected true got false
PASS MediaRecorder interface: attribute onpause
PASS MediaRecorder interface: attribute onresume
PASS MediaRecorder interface: attribute onerror
FAIL MediaRecorder interface: attribute videoBitsPerSecond assert_true: The prototype object must have a property "videoBitsPerSecond" expected true got false
FAIL MediaRecorder interface: attribute audioBitsPerSecond assert_true: The prototype object must have a property "audioBitsPerSecond" expected true got false
FAIL MediaRecorder interface: attribute audioBitrateMode assert_true: The prototype object must have a property "audioBitrateMode" expected true got false
PASS MediaRecorder interface: operation start(optional unsigned long)
PASS MediaRecorder interface: operation stop()
FAIL MediaRecorder interface: operation pause() assert_own_property: interface prototype object missing non-static operation expected property "pause" missing
FAIL MediaRecorder interface: operation resume() assert_own_property: interface prototype object missing non-static operation expected property "resume" missing
PASS MediaRecorder interface: operation pause()
PASS MediaRecorder interface: operation resume()
PASS MediaRecorder interface: operation requestData()
PASS MediaRecorder interface: operation isTypeSupported(DOMString)
PASS MediaRecorder must be primary interface of [object MediaRecorder]
Expand All @@ -33,17 +33,17 @@ PASS MediaRecorder interface: [object MediaRecorder] must inherit property "stat
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "onstart" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "onstop" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "ondataavailable" with the proper type
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "onpause" with the proper type assert_inherits: property "onpause" not found in prototype chain
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "onresume" with the proper type assert_inherits: property "onresume" not found in prototype chain
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "onpause" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "onresume" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "onerror" with the proper type
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "videoBitsPerSecond" with the proper type assert_inherits: property "videoBitsPerSecond" not found in prototype chain
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "audioBitsPerSecond" with the proper type assert_inherits: property "audioBitsPerSecond" not found in prototype chain
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "audioBitrateMode" with the proper type assert_inherits: property "audioBitrateMode" not found in prototype chain
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "start(optional unsigned long)" with the proper type
PASS MediaRecorder interface: calling start(optional unsigned long) on [object MediaRecorder] with too few arguments must throw TypeError
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "stop()" with the proper type
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "pause()" with the proper type assert_inherits: property "pause" not found in prototype chain
FAIL MediaRecorder interface: [object MediaRecorder] must inherit property "resume()" with the proper type assert_inherits: property "resume" not found in prototype chain
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "pause()" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "resume()" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "requestData()" with the proper type
PASS MediaRecorder interface: [object MediaRecorder] must inherit property "isTypeSupported(DOMString)" with the proper type
PASS MediaRecorder interface: calling isTypeSupported(DOMString) on [object MediaRecorder] with too few arguments must throw TypeError
Expand Down
34 changes: 34 additions & 0 deletions Source/WebCore/ChangeLog
@@ -1,3 +1,37 @@
2020-10-07 Youenn Fablet <youenn@apple.com>

Add support for MediaRecorder pause/resume
https://bugs.webkit.org/show_bug.cgi?id=217375

Reviewed by Eric Carlson.

Implement pause and resume as per spec.
MediaRecorder basically sends pause/resume order to its backend.
The backend then stops observing tracks when paused and resumed observing at resume time.
For video, we make sure to compute the frame timestamp so that the recorded video continues to play without interruption.

Test: http/wpt/mediarecorder/pause-recording.html

* Modules/mediarecorder/MediaRecorder.cpp:
(WebCore::MediaRecorder::pauseRecording):
(WebCore::MediaRecorder::resumeRecording):
* Modules/mediarecorder/MediaRecorder.h:
* Modules/mediarecorder/MediaRecorder.idl:
* platform/mediarecorder/MediaRecorderPrivate.cpp:
(WebCore::MediaRecorderPrivate::pause):
(WebCore::MediaRecorderPrivate::resume):
* platform/mediarecorder/MediaRecorderPrivate.h:
* platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp:
(WebCore::MediaRecorderPrivateAVFImpl::pauseRecording):
(WebCore::MediaRecorderPrivateAVFImpl::resumeRecording):
* platform/mediarecorder/MediaRecorderPrivateAVFImpl.h:
* platform/mediarecorder/MediaRecorderPrivateMock.h:
* platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h:
* platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm:
(WebCore::MediaRecorderPrivateWriter::appendVideoSampleBuffer):
(WebCore::MediaRecorderPrivateWriter::pause):
(WebCore::MediaRecorderPrivateWriter::resume):

2020-10-07 Youenn Fablet <youenn@apple.com>

MediaRecorder::create should not need to create a MediaRecorderPrivate to validate it can record properly
Expand Down
42 changes: 42 additions & 0 deletions Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp
Expand Up @@ -227,6 +227,48 @@ ExceptionOr<void> MediaRecorder::requestData()
return { };
}

ExceptionOr<void> MediaRecorder::pauseRecording()
{
if (state() == RecordingState::Inactive)
return Exception { InvalidStateError, "The MediaRecorder's state cannot be inactive"_s };

if (state() == RecordingState::Paused)
return { };

m_state = RecordingState::Paused;
m_private->pause([this, pendingActivity = makePendingActivity(*this)]() {
if (!m_isActive)
return;
queueTaskKeepingObjectAlive(*this, TaskSource::Networking, [this]() mutable {
if (!m_isActive)
return;
dispatchEvent(Event::create(eventNames().pauseEvent, Event::CanBubble::No, Event::IsCancelable::No));
});
});
return { };
}

ExceptionOr<void> MediaRecorder::resumeRecording()
{
if (state() == RecordingState::Inactive)
return Exception { InvalidStateError, "The MediaRecorder's state cannot be inactive"_s };

if (state() == RecordingState::Recording)
return { };

m_state = RecordingState::Recording;
m_private->resume([this, pendingActivity = makePendingActivity(*this)]() {
if (!m_isActive)
return;
queueTaskKeepingObjectAlive(*this, TaskSource::Networking, [this]() mutable {
if (!m_isActive)
return;
dispatchEvent(Event::create(eventNames().resumeEvent, Event::CanBubble::No, Event::IsCancelable::No));
});
});
return { };
}

void MediaRecorder::fetchData(FetchDataCallback&& callback, TakePrivateRecorder takeRecorder)
{
auto& privateRecorder = *m_private;
Expand Down
2 changes: 2 additions & 0 deletions Source/WebCore/Modules/mediarecorder/MediaRecorder.h
Expand Up @@ -71,6 +71,8 @@ class MediaRecorder final
ExceptionOr<void> startRecording(Optional<unsigned>);
ExceptionOr<void> stopRecording();
ExceptionOr<void> requestData();
ExceptionOr<void> pauseRecording();
ExceptionOr<void> resumeRecording();

MediaStream& stream() { return m_stream.get(); }

Expand Down
10 changes: 4 additions & 6 deletions Source/WebCore/Modules/mediarecorder/MediaRecorder.idl
Expand Up @@ -32,25 +32,23 @@ enum RecordingState { "inactive", "recording", "paused" };
] interface MediaRecorder : EventTarget {
[CallWith=Document] constructor(MediaStream stream, optional MediaRecorderOptions options);

// FIXME: Implement commented out methods/attributes.

readonly attribute MediaStream stream;
readonly attribute DOMString mimeType;
readonly attribute RecordingState state;
attribute EventHandler onstart;
attribute EventHandler onstop;
attribute EventHandler ondataavailable;
// attribute EventHandler onpause;
// attribute EventHandler onresume;
attribute EventHandler onpause;
attribute EventHandler onresume;
attribute EventHandler onerror;
// readonly attribute unsigned long videoBitsPerSecond;
// readonly attribute unsigned long audioBitsPerSecond;
// readonly attribute BitrateMode audioBitrateMode;

[MayThrowException, ImplementedAs=startRecording] undefined start(optional unsigned long timeslice);
[MayThrowException, ImplementedAs=stopRecording] undefined stop();
// undefined pause();
// undefined resume();
[MayThrowException, ImplementedAs=pauseRecording] undefined pause();
[MayThrowException, ImplementedAs=resumeRecording] undefined resume();
[MayThrowException] undefined requestData();

[CallWith=Document] static boolean isTypeSupported(DOMString type);
Expand Down
24 changes: 24 additions & 0 deletions Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.cpp
Expand Up @@ -73,6 +73,30 @@ void MediaRecorderPrivate::stop()
stopRecording();
}

void MediaRecorderPrivate::pause(CompletionHandler<void()>&& completionHandler)
{
ASSERT(!m_pausedAudioSource);
ASSERT(!m_pausedVideoSource);

m_pausedAudioSource = m_audioSource;
m_pausedVideoSource = m_videoSource;

setAudioSource(nullptr);
setVideoSource(nullptr);

pauseRecording(WTFMove(completionHandler));
}

void MediaRecorderPrivate::resume(CompletionHandler<void()>&& completionHandler)
{
ASSERT(m_pausedAudioSource || m_pausedVideoSource);

setAudioSource(WTFMove(m_pausedAudioSource));
setVideoSource(WTFMove(m_pausedVideoSource));

resumeRecording(WTFMove(completionHandler));
}

} // namespace WebCore

#endif // ENABLE(MEDIA_STREAM)
6 changes: 6 additions & 0 deletions Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h
Expand Up @@ -62,6 +62,8 @@ class MediaRecorderPrivate
virtual const String& mimeType() const = 0;

void stop();
void pause(CompletionHandler<void()>&&);
void resume(CompletionHandler<void()>&&);

using StartRecordingCallback = CompletionHandler<void(ExceptionOr<String>&&)>;
virtual void startRecording(StartRecordingCallback&& callback) { callback(String(mimeType())); }
Expand All @@ -80,12 +82,16 @@ class MediaRecorderPrivate

private:
virtual void stopRecording() = 0;
virtual void pauseRecording(CompletionHandler<void()>&&) = 0;
virtual void resumeRecording(CompletionHandler<void()>&&) = 0;

private:
bool m_shouldMuteAudio { false };
bool m_shouldMuteVideo { false };
RefPtr<RealtimeMediaSource> m_audioSource;
RefPtr<RealtimeMediaSource> m_videoSource;
RefPtr<RealtimeMediaSource> m_pausedAudioSource;
RefPtr<RealtimeMediaSource> m_pausedVideoSource;
};

inline void MediaRecorderPrivate::setAudioSource(RefPtr<RealtimeMediaSource>&& audioSource)
Expand Down

0 comments on commit a182343

Please sign in to comment.