Media Recording: Add Background Blur #12067

merapi · 2022-08-08T09:18:02Z

Summary

This version of Background Blur is based on the latest version of the new Trimming flow PR.
The Media Pipe object is very difficult to cooperate and it often throws errors while switching between no effect and blur effect. In this PR I managed to make it work consistently. The recording part is not yet merged into this version (two media recorders one used to get the camera feed, the second for canvas recording when the blur effect is active).

Edges are now smooth so it looks a lot better than the first version:

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

Reviews

Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

This PR addresses an existing issue and I have linked this PR to it in ZenHub
I have tested this code to the best of my abilities
I have verified accessibility to the best of my abilities (docs)
I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
This code is covered by automated tests (unit, integration, and/or e2e) to verify it works as intended (docs)
I have added documentation where necessary
I have added a matching Type: XYZ label to the PR

Fixes #11909
Fixes #12033
Fixes #12245
Fixes #12249
Fixes #12246

googleforcreators-bot · 2022-08-08T09:35:43Z

Plugin builds for 5878aaa are ready 🛎️!

Download development build
Download production build

barklund

This works really well, but it's hard to review with the mixed code from multiple PR's. Let's have @felipebochehin87 quickly QA #11988, so we can merge #12022 and rebase this PR.

packages/story-editor/package.json

package-lock.json

barklund · 2022-08-08T14:26:20Z

#12022 has been merged, so this can be updated to main to better reflect that actual changes.

# Conflicts: # package-lock.json

merapi · 2022-08-18T12:33:20Z

Summary of the latest updates:
You will see two useMediaRecorder instances in the code: mediaRecorder and canvasRecorder.
When all effects are disabled (default state) we will do the recording in the old way (mediaRecorder will be the currentRecorder) - this is more performant way than always redrawing the camera stream onto the canvas and recording the canvas (with and without effects) - so it will be as snappy as possible on slower devices when you don't use any effects.

I also fixed #12033

packages/story-editor/src/components/mediaRecording/settingsModal.js

packages/story-editor/src/app/highlights/quickActions/useQuickActions.js

miina

(Will continue the review on Monday, couldn't go through everything)

packages/story-editor/src/components/mediaRecording/useTrim.js

packages/story-editor/src/components/mediaRecording/settingsModal.js

packages/story-editor/src/components/mediaRecording/provider.js

miina · 2022-08-19T10:27:44Z

packages/story-editor/src/app/highlights/quickActions/useQuickActions.js

@@ -331,6 +340,7 @@ const useQuickActions = () => {
    muteAudio: actions.muteAudio,
    unMuteAudio: actions.unMuteAudio,
    startTrim: actions.startTrim,
+    setVideoEffect: actions.setVideoEffect,
  }));

  const enableMediaRecordingTrimming = useFeature('recordingTrimming');


💡 thought:
Not related to this PR directly but should these actions be disabled during recording? Currently it seems that it's possible to click on it during recording which results in the recording just stopping. Thoughts?

Hm, okay, I'm now trying again and seeing these are disabled generally. Unfortunately I don't know how to reproduce it but somehow I ended up (twice) in a situation where I could click on the Disable Background Blur quick-action while recording -> recording stopped -> when I opened the recorder again, I saw this with neither of the buttons (Pause/Stop) doing anything:

OK, I tried again and the following seems to break it eventually:

Open the recorder

Enable background blur

Start recording

Wait for a couple of seconds -- when testing, the quick-actions became enabled when the blur option was turned on.

Now disable the blur while recording. And now the video stops recording.

Click X to close the recorder tool.

When I repeat the process a few times, the following occurs:

If the video stopped recording because of using quick-actions during recording, and then when opening the recorder tool from the Library again, it starts recording right away when opening.

After a few times of doing this, the video starts recording but doesn't really show the video and the Pause/Stop buttons don't do anything (as above on the screenshot).

I thought it would be just a flag change (incorrect status/inputStatus/isReady), but it turned out that I was trying to fix something that was impossible to fix with the current version of the library. It was not possible to use the custom media recorder more than once per instance. Luckily, when I was about to go crazy I checked the library repo and saw this one-liner 🪄

Note to self: always check if you are using the latest version of the library.

spacedmonkey · 2022-09-13T10:19:05Z

Noticed another weird issues with the settings dialog. See video.

Screen.Recording.2022-09-13.at.11.14.03.mov

merapi · 2022-09-13T10:24:38Z

Noticed another weird issues with the settings dialog. See video.

Screen.Recording.2022-09-13.at.11.14.03.mov

What browser is this? any errors in the console?

spacedmonkey · 2022-09-13T10:31:16Z

What browser is this? any errors in the console?

Chrome 104. Nope.

spacedmonkey · 2022-09-14T15:00:54Z

Seeing slow down of image in safari.
https://user-images.githubusercontent.com/237508/190190745-4cd586d1-1e16-4403-ad02-b3ae0e38bd96.mov

And other issues in firefox.
https://user-images.githubusercontent.com/237508/190190940-63efdbe5-bb73-4582-9cd9-bd60a170b504.mov

merapi · 2022-09-14T15:30:13Z

Seeing slow down of image in safari.

That is expected. It will never match the native blur or get close to it, tho it looks better on my Macbook Air.

And other issues in firefox.

~~That is only happening on 2+ try in the same session, right?~~ Probably fixed? PTAL

spacedmonkey · 2022-09-14T16:17:06Z

Not seeing device list issue anymore.

spacedmonkey · 2022-09-14T16:19:10Z

Safari I am seeing this error when I try active blur background.

spacedmonkey · 2022-09-14T16:23:14Z

Not seeing firefox issues now. 🎉

merapi · 2022-09-14T16:25:38Z

Safari I am seeing this error when I try active blur background.

Yes, it's WASM and some of its memory issues. I added a try/catch there that's the best we can do.

spacedmonkey · 2022-09-14T16:30:24Z

Yes, it's WASM and some of its memory issues. I added a try/catch there that's the best we can do.

More testing, I am seeing the slow version in safari. Blur background works but it is very slow in safari.

Screen.Recording.2022-09-14.at.17.28.05.mov

Morten Barklund and others added 7 commits July 26, 2022 16:59

Editor: Changed video recording trim to happen before insertion

fa03637

Merge branch 'main' into add/11988-recorder-early-trim

c5ee09e

Merge branch 'main' into add/11988-recorder-early-trim

8915c45

Merge branch 'main' into add/11988-recorder-early-trim

ca4da45

Fixed re-trimming recording by caching video strip background

f7c2920

Implemented new trim flow

a412f83

Stable version based on new Trimming

f77f990

merapi added Type: Enhancement New feature or improvement of an existing feature Group: Media Pod: Prometheus labels Aug 8, 2022

merapi self-assigned this Aug 8, 2022

Package lock

184ba71

merapi requested review from swissspidy and spacedmonkey as code owners August 8, 2022 09:21

barklund reviewed Aug 8, 2022

View reviewed changes

packages/story-editor/package.json Show resolved Hide resolved

package-lock.json Outdated Show resolved Hide resolved

swissspidy mentioned this pull request Aug 8, 2022

[WIP] Background separation PoC #12002

Closed

8 tasks

Morten Barklund and others added 5 commits August 9, 2022 13:24

Merge with main

0100efa

Merged playback

32854d0

Dual recorder port

c015e73

Fix image capture

24b8303

Merge branch 'main' into try/11909-background-blur-early-trim-base

c06edd5

# Conflicts: # package-lock.json

merapi requested review from barklund and miina August 18, 2022 12:33

swissspidy reviewed Aug 18, 2022

View reviewed changes

packages/story-editor/src/components/mediaRecording/settingsModal.js Show resolved Hide resolved

swissspidy reviewed Aug 18, 2022

View reviewed changes

packages/story-editor/src/app/highlights/quickActions/useQuickActions.js Outdated Show resolved Hide resolved

miina reviewed Aug 19, 2022

View reviewed changes

swissspidy changed the title ~~Background Blur based on early trim PR~~ Media Recording: Add Background Blur Aug 29, 2022

felipebochehin87 mentioned this pull request Sep 13, 2022

Self Recording blur background causes image to freeze #12246

Closed

Merge branch 'main' into try/11909-background-blur-early-trim-base

b6ac2a5

merapi added 12 commits September 13, 2022 12:52

Test Safari freeze

6ad2dac

Temp Safari fake blur

691c323

Test manual blur in Safari

db3d485

Simpler blur switch

70c0e98

Try faster blur?

e2e11d7

Clean up comments

51ac141

No WASM in tests

05b2ff1

Merge branch 'main' into try/11909-background-blur-early-trim-base

79995b3

Fix random failing test: try more delay

1796235

Try fix for a few tests

a50ffa6

Fix failing test

7a822fc

Fix failing tests

6204583

merapi added 2 commits September 14, 2022 17:39

Fix empty devices list

1cc75fb

Firefox double frame fix

5ba0efc

WASM memory try/catch

5878aaa

merapi merged commit 4992f17 into main Sep 14, 2022

merapi deleted the try/11909-background-blur-early-trim-base branch September 14, 2022 16:28

swissspidy mentioned this pull request Sep 14, 2022

Media Recording: Improve settings dialog #12312

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Media Recording: Add Background Blur #12067

Media Recording: Add Background Blur #12067

merapi commented Aug 8, 2022 •

edited

googleforcreators-bot commented Aug 8, 2022 •

edited

barklund left a comment

barklund commented Aug 8, 2022

merapi commented Aug 18, 2022

miina left a comment

miina Aug 19, 2022

miina Aug 19, 2022

miina Aug 19, 2022

merapi Sep 5, 2022

spacedmonkey commented Sep 13, 2022

merapi commented Sep 13, 2022

spacedmonkey commented Sep 13, 2022

spacedmonkey commented Sep 14, 2022

merapi commented Sep 14, 2022 •

edited

spacedmonkey commented Sep 14, 2022

spacedmonkey commented Sep 14, 2022

spacedmonkey commented Sep 14, 2022

merapi commented Sep 14, 2022

spacedmonkey commented Sep 14, 2022

Media Recording: Add Background Blur #12067

Media Recording: Add Background Blur #12067

Conversation

merapi commented Aug 8, 2022 • edited

Summary

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

Reviews

Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

googleforcreators-bot commented Aug 8, 2022 • edited

barklund left a comment

Choose a reason for hiding this comment

barklund commented Aug 8, 2022

merapi commented Aug 18, 2022

miina left a comment

Choose a reason for hiding this comment

miina Aug 19, 2022

Choose a reason for hiding this comment

miina Aug 19, 2022

Choose a reason for hiding this comment

miina Aug 19, 2022

Choose a reason for hiding this comment

merapi Sep 5, 2022

Choose a reason for hiding this comment

spacedmonkey commented Sep 13, 2022

merapi commented Sep 13, 2022

spacedmonkey commented Sep 13, 2022

spacedmonkey commented Sep 14, 2022

merapi commented Sep 14, 2022 • edited

spacedmonkey commented Sep 14, 2022

spacedmonkey commented Sep 14, 2022

spacedmonkey commented Sep 14, 2022

merapi commented Sep 14, 2022

spacedmonkey commented Sep 14, 2022

merapi commented Aug 8, 2022 •

edited

googleforcreators-bot commented Aug 8, 2022 •

edited

merapi commented Sep 14, 2022 •

edited