Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding trimmer functionality to media recording #11908

Merged
merged 21 commits into from
Jul 25, 2022
Merged

Conversation

barklund
Copy link
Contributor

@barklund barklund commented Jul 12, 2022

Context

This refactors both the existing trimmer and the media recording system to be able to merge the two concepts.

Due to the necessary extra provider, the media recording has been split out from the media recording layer. This will probably cause conflicts with the audio work.

To-do

  • Re-trimming while still in recording mode is a bit broken
  • It is still buggy - sometimes the reference to videoNode is stale and things fail.
  • After trimming, the video still plays in the full duration in the preview, but the trim data has been recorded
  • The preview on-canvas while the video is being trimmed and uploaded is also not trimmed, but shows the full duration
  • The generated thumbnail is taken from the start of the full video, not the start of the trimmed video

User-facing changes

TBD

Testing Instructions

This PR can be tested by following these steps:

  1. Start a media recording
  2. Record yourself
  3. Click the scissors to trim
  4. Play with trimming, cancelling, re-trimming, inserting, etc.

Please note known bugs above.

Checklist

  • This PR addresses an existing issue and I have linked this PR to it in ZenHub
  • I have tested this code to the best of my abilities
  • I have verified accessibility to the best of my abilities (docs)
  • I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
  • This code is covered by automated tests (unit, integration, and/or e2e) to verify it works as intended (docs)
  • I have added documentation where necessary
  • I have added a matching Type: XYZ label to the PR

Fixes #11685
Fixes #11922

@barklund barklund added Type: Enhancement New feature or improvement of an existing feature Pod: Prometheus labels Jul 12, 2022
@barklund barklund requested review from miina and merapi July 12, 2022 14:42
@barklund barklund self-assigned this Jul 12, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Jul 12, 2022

Size Change: +615 B (0%)

Total Size: 2.66 MB

Filename Size Change
assets/js/wp-story-editor.js 328 kB +615 B (0%)
ℹ️ View Unchanged
Filename Size
assets/css/carousel-view-rtl.css 702 B
assets/css/carousel-view.css 701 B
assets/css/web-stories-block-rtl.css 4.52 kB
assets/css/web-stories-block.css 4.56 kB
assets/css/web-stories-embed-rtl.css 318 B
assets/css/web-stories-embed.css 317 B
assets/css/web-stories-list-styles-rtl.css 2.36 kB
assets/css/web-stories-list-styles.css 2.39 kB
assets/css/web-stories-theme-style-twentyeleven-rtl.css 102 B
assets/css/web-stories-theme-style-twentyeleven.css 102 B
assets/css/web-stories-theme-style-twentyfifteen-rtl.css 251 B
assets/css/web-stories-theme-style-twentyfifteen.css 251 B
assets/css/web-stories-theme-style-twentyfourteen-rtl.css 287 B
assets/css/web-stories-theme-style-twentyfourteen.css 287 B
assets/css/web-stories-theme-style-twentyseventeen-rtl.css 288 B
assets/css/web-stories-theme-style-twentyseventeen.css 288 B
assets/css/web-stories-theme-style-twentysixteen-rtl.css 224 B
assets/css/web-stories-theme-style-twentysixteen.css 224 B
assets/css/web-stories-theme-style-twentyten-rtl.css 143 B
assets/css/web-stories-theme-style-twentyten.css 143 B
assets/css/web-stories-theme-style-twentytwelve-rtl.css 256 B
assets/css/web-stories-theme-style-twentytwelve.css 256 B
assets/css/web-stories-theme-style-twentytwenty-rtl.css 86 B
assets/css/web-stories-theme-style-twentytwenty.css 86 B
assets/css/web-stories-theme-style-twentytwentyone-rtl.css 326 B
assets/css/web-stories-theme-style-twentytwentyone.css 326 B
assets/css/web-stories-widget-rtl.css 482 B
assets/css/web-stories-widget.css 482 B
assets/css/wp-dashboard-rtl.css 657 B
assets/css/wp-dashboard.css 659 B
assets/css/wp-story-editor-rtl.css 737 B
assets/css/wp-story-editor.css 738 B
assets/js/1590.js 1.14 MB
assets/js/1814.js 7.46 kB
assets/js/2505.js 34.9 kB
assets/js/3617.js 224 kB
assets/js/4422.js 49.3 kB
assets/js/5980.js 5.48 kB
assets/js/carousel-view.js 3.41 kB
assets/js/chunk-colorthief.js 2.64 kB
assets/js/chunk-ffmpeg.js 5.64 kB
assets/js/chunk-focus-visible.js 1.01 kB
assets/js/chunk-getStoryMarkup.js 5.86 kB
assets/js/chunk-html-to-image.js 4.6 kB
assets/js/chunk-opentype.js 96 B
assets/js/chunk-react-calendar.js 12.4 kB
assets/js/chunk-react-color.js 44.3 kB
assets/js/chunk-resize-observer-polyfill.js 2.57 kB
assets/js/chunk-web-animations-js.js 14.6 kB
assets/js/chunk-web-stories-template-0-metaData.js 546 B
assets/js/chunk-web-stories-template-0.js 10.6 kB
assets/js/chunk-web-stories-template-1-metaData.js 540 B
assets/js/chunk-web-stories-template-1.js 9.01 kB
assets/js/chunk-web-stories-template-10-metaData.js 533 B
assets/js/chunk-web-stories-template-10.js 6.91 kB
assets/js/chunk-web-stories-template-11-metaData.js 540 B
assets/js/chunk-web-stories-template-11.js 8.51 kB
assets/js/chunk-web-stories-template-12-metaData.js 496 B
assets/js/chunk-web-stories-template-12.js 9.48 kB
assets/js/chunk-web-stories-template-13-metaData.js 525 B
assets/js/chunk-web-stories-template-13.js 7.3 kB
assets/js/chunk-web-stories-template-14-metaData.js 582 B
assets/js/chunk-web-stories-template-14.js 7.58 kB
assets/js/chunk-web-stories-template-15-metaData.js 544 B
assets/js/chunk-web-stories-template-15.js 8.21 kB
assets/js/chunk-web-stories-template-16-metaData.js 588 B
assets/js/chunk-web-stories-template-16.js 10.3 kB
assets/js/chunk-web-stories-template-17-metaData.js 539 B
assets/js/chunk-web-stories-template-17.js 8.52 kB
assets/js/chunk-web-stories-template-18-metaData.js 585 B
assets/js/chunk-web-stories-template-18.js 9.05 kB
assets/js/chunk-web-stories-template-19-metaData.js 501 B
assets/js/chunk-web-stories-template-19.js 9.99 kB
assets/js/chunk-web-stories-template-2-metaData.js 586 B
assets/js/chunk-web-stories-template-2.js 9.16 kB
assets/js/chunk-web-stories-template-20-metaData.js 548 B
assets/js/chunk-web-stories-template-20.js 8.59 kB
assets/js/chunk-web-stories-template-21-metaData.js 534 B
assets/js/chunk-web-stories-template-21.js 9.16 kB
assets/js/chunk-web-stories-template-22-metaData.js 525 B
assets/js/chunk-web-stories-template-22.js 7.37 kB
assets/js/chunk-web-stories-template-23-metaData.js 605 B
assets/js/chunk-web-stories-template-23.js 6.99 kB
assets/js/chunk-web-stories-template-24-metaData.js 518 B
assets/js/chunk-web-stories-template-24.js 10.8 kB
assets/js/chunk-web-stories-template-25-metaData.js 544 B
assets/js/chunk-web-stories-template-25.js 7.07 kB
assets/js/chunk-web-stories-template-26-metaData.js 601 B
assets/js/chunk-web-stories-template-26.js 6.85 kB
assets/js/chunk-web-stories-template-27-metaData.js 543 B
assets/js/chunk-web-stories-template-27.js 7.36 kB
assets/js/chunk-web-stories-template-28-metaData.js 532 B
assets/js/chunk-web-stories-template-28.js 8.49 kB
assets/js/chunk-web-stories-template-29-metaData.js 561 B
assets/js/chunk-web-stories-template-29.js 8.49 kB
assets/js/chunk-web-stories-template-3-metaData.js 540 B
assets/js/chunk-web-stories-template-3.js 8.22 kB
assets/js/chunk-web-stories-template-30-metaData.js 576 B
assets/js/chunk-web-stories-template-30.js 7.67 kB
assets/js/chunk-web-stories-template-31-metaData.js 503 B
assets/js/chunk-web-stories-template-31.js 9.61 kB
assets/js/chunk-web-stories-template-32-metaData.js 551 B
assets/js/chunk-web-stories-template-32.js 12.2 kB
assets/js/chunk-web-stories-template-33-metaData.js 492 B
assets/js/chunk-web-stories-template-33.js 8.86 kB
assets/js/chunk-web-stories-template-34-metaData.js 571 B
assets/js/chunk-web-stories-template-34.js 7.57 kB
assets/js/chunk-web-stories-template-35-metaData.js 565 B
assets/js/chunk-web-stories-template-35.js 8.81 kB
assets/js/chunk-web-stories-template-36-metaData.js 576 B
assets/js/chunk-web-stories-template-36.js 11.6 kB
assets/js/chunk-web-stories-template-37-metaData.js 528 B
assets/js/chunk-web-stories-template-37.js 6.47 kB
assets/js/chunk-web-stories-template-38-metaData.js 572 B
assets/js/chunk-web-stories-template-38.js 7.96 kB
assets/js/chunk-web-stories-template-39-metaData.js 589 B
assets/js/chunk-web-stories-template-39.js 7.67 kB
assets/js/chunk-web-stories-template-4-metaData.js 565 B
assets/js/chunk-web-stories-template-4.js 11.5 kB
assets/js/chunk-web-stories-template-40-metaData.js 556 B
assets/js/chunk-web-stories-template-40.js 9.13 kB
assets/js/chunk-web-stories-template-41-metaData.js 572 B
assets/js/chunk-web-stories-template-41.js 7.75 kB
assets/js/chunk-web-stories-template-42-metaData.js 522 B
assets/js/chunk-web-stories-template-42.js 7 kB
assets/js/chunk-web-stories-template-43-metaData.js 558 B
assets/js/chunk-web-stories-template-43.js 8.37 kB
assets/js/chunk-web-stories-template-44-metaData.js 582 B
assets/js/chunk-web-stories-template-44.js 10.1 kB
assets/js/chunk-web-stories-template-45-metaData.js 564 B
assets/js/chunk-web-stories-template-45.js 7.12 kB
assets/js/chunk-web-stories-template-46-metaData.js 531 B
assets/js/chunk-web-stories-template-46.js 5.01 kB
assets/js/chunk-web-stories-template-47-metaData.js 592 B
assets/js/chunk-web-stories-template-47.js 8.46 kB
assets/js/chunk-web-stories-template-48-metaData.js 556 B
assets/js/chunk-web-stories-template-48.js 8.31 kB
assets/js/chunk-web-stories-template-49-metaData.js 518 B
assets/js/chunk-web-stories-template-49.js 9.7 kB
assets/js/chunk-web-stories-template-5-metaData.js 555 B
assets/js/chunk-web-stories-template-5.js 9.38 kB
assets/js/chunk-web-stories-template-50-metaData.js 504 B
assets/js/chunk-web-stories-template-50.js 8.26 kB
assets/js/chunk-web-stories-template-51-metaData.js 527 B
assets/js/chunk-web-stories-template-51.js 9.89 kB
assets/js/chunk-web-stories-template-52-metaData.js 602 B
assets/js/chunk-web-stories-template-52.js 10.1 kB
assets/js/chunk-web-stories-template-53-metaData.js 553 B
assets/js/chunk-web-stories-template-53.js 5.79 kB
assets/js/chunk-web-stories-template-54-metaData.js 547 B
assets/js/chunk-web-stories-template-54.js 7.52 kB
assets/js/chunk-web-stories-template-55-metaData.js 574 B
assets/js/chunk-web-stories-template-55.js 6.56 kB
assets/js/chunk-web-stories-template-56-metaData.js 543 B
assets/js/chunk-web-stories-template-56.js 9.5 kB
assets/js/chunk-web-stories-template-57-metaData.js 528 B
assets/js/chunk-web-stories-template-57.js 14.1 kB
assets/js/chunk-web-stories-template-58-metaData.js 556 B
assets/js/chunk-web-stories-template-58.js 5.61 kB
assets/js/chunk-web-stories-template-59-metaData.js 588 B
assets/js/chunk-web-stories-template-59.js 8.52 kB
assets/js/chunk-web-stories-template-6-metaData.js 569 B
assets/js/chunk-web-stories-template-6.js 7.04 kB
assets/js/chunk-web-stories-template-60-metaData.js 509 B
assets/js/chunk-web-stories-template-60.js 8.89 kB
assets/js/chunk-web-stories-template-7-metaData.js 569 B
assets/js/chunk-web-stories-template-7.js 7.21 kB
assets/js/chunk-web-stories-template-8-metaData.js 569 B
assets/js/chunk-web-stories-template-8.js 8.4 kB
assets/js/chunk-web-stories-template-9-metaData.js 581 B
assets/js/chunk-web-stories-template-9.js 8.49 kB
assets/js/chunk-web-stories-templates.js 443 B
assets/js/chunk-web-stories-textset-0.js 5.08 kB
assets/js/chunk-web-stories-textset-1.js 6.64 kB
assets/js/chunk-web-stories-textset-2.js 7.67 kB
assets/js/chunk-web-stories-textset-3.js 15.1 kB
assets/js/chunk-web-stories-textset-4.js 4.16 kB
assets/js/chunk-web-stories-textset-5.js 5.49 kB
assets/js/chunk-web-stories-textset-6.js 5.3 kB
assets/js/chunk-web-stories-textset-7.js 10.2 kB
assets/js/generateBlurhash.worker.worker.js 1.1 kB
assets/js/imgareaselect.js 3.77 kB
assets/js/lightbox.js 550 B
assets/js/tinymce-button.js 2.84 kB
assets/js/web-stories-activation-notice.js 26.9 kB
assets/js/web-stories-block.js 22.7 kB
assets/js/web-stories-embed.js 20 B
assets/js/web-stories-widget.js 587 B
assets/js/wp-dashboard.js 72.6 kB

compressed-size-action

@googleforcreators-bot
Copy link
Collaborator

googleforcreators-bot commented Jul 12, 2022

Plugin builds for 9c79704 are ready 🛎️!

@merapi
Copy link
Contributor

merapi commented Jul 13, 2022

Might be connected to that stale videoNode, if I trim before the preview playback finishes I get this:

image

image

image

If I wait it seems ok?

Play(/pause) in preview (without touching trimming) doesn't work (works on main).

@barklund barklund changed the title [WIP] Adding trimmer functionality to media recording Adding trimmer functionality to media recording Jul 13, 2022
@barklund
Copy link
Contributor Author

The remaining issues are both very straight-forward to understand and test around, but also pretty annoying to fix. Well, the first one with the preview in the media recorder is fairly easy, I can copy the code from the trimmer to start and stop the video at the right frame offsets, but manipulating the video in the preview on canvas seems a lot harder (and even more code to duplicate), and the frame snapshot too.

I'm considering doing only the first issue in this version and filing the other two tickets as followup. The latter at least seem better suited for WP pod, as this thing is in their domain.

But please, @merapi and @miina, check it out as-is both the result and the code and give me a review. I think this is solid for bug bash at least.

Finally, we could also file a follow-up bug to deal with the duration only coming after the video has played through once making trimming not work until then. We could use the duration (in seconds only though) stored inside the recorder provider until a more precise time is known. We could probably also extract the exact time from the blob somewhere and use that value. Or if we can't do either of those, we could at least disable the trim button until it's available after the first play-through.

</DisplayPageArea>
<Footer captureImage={debouncedCaptureImage} videoRef={videoRef} />
<StyledFooter showOverflow>
{isTrimming ? <VideoTrimmer /> : <Footer />}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should display a loading indicator here until we have the duration when the trim icon is clicked?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we should definitely do something here. I have some ideas for how to improve, but it's a based on technical limitations so hard to solve well in UX.

@miina
Copy link
Contributor

miina commented Jul 14, 2022

manipulating the video in the preview on canvas seems a lot harder

Wondering if it would make sense to upload the video in full length and do the trimming the same way as in case of "normal" video elements, and just remove the video if dismissed / not inserted? Or keeping it as the parent video.

Finally, we could also file a follow-up bug to deal with the duration only coming after the video has played through once making trimming not work until then. We could use the duration (in seconds only though) stored inside the recorder provider until a more precise time is known. We could probably also extract the exact time from the blob somewhere and use that value. Or if we can't do either of those, we could at least disable the trim button until it's available after the first play-through.

Mentioned it in the comment as well but maybe we could just display a loading indicator after clicking on the Trim icon? Otherwise the user might get the impression that trimming is not possible at all, if it's disabled. Thoughts?

I think this is solid for bug bash at least.

SGTM if @merapi is ok with it.

args.resource.lengthFormatted = length
? lengthFormatted
: getVideoLengthDisplay(duration);
args.resource.length = duration;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should round this to a whole number.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is rounded, AFAIK. The original value here comes from the recording counter (which just counts seconds), and when trimming, the value is updated to the ceil() of the trim interval.

Morten Barklund added 6 commits July 15, 2022 13:37
# Conflicts:
#	packages/story-editor/src/app/highlights/quickActions/useQuickActions.js
#	packages/story-editor/src/components/canvas/mediaRecordingLayer.js
#	packages/story-editor/src/components/mediaRecording/footer.js
#	packages/story-editor/src/components/mediaRecording/index.js
@spacedmonkey
Copy link
Contributor

Sharing comment for visibility. #11887 (comment)

@barklund barklund requested a review from miina July 18, 2022 11:48
@swissspidy
Copy link
Collaborator

Sharing comment for visibility. #11887 (comment)

This PR only covers trimming for videos, not for audio files.

In the future we may consider adding an audio trimmer though, but that requires UX and prioritization.

@felipebochehin87
Copy link

Trimming works for gifs and video;
Play/Stop buttons displayed for both videos and gifs;
When I trim/re-trim a video or a gif and preview the story, only the trimmed part of it is reproduced while on preview story screen;
Same as above for publishing story.

https://stories-qa-wordpress-amp.pantheonsite.io/wp-admin/post.php?post=19102&action=edit#page=%25224bf235a6-a904-4a4c-8e36-b5c6a039ecaa%2522

@barklund barklund requested a review from swissspidy July 19, 2022 19:44
@swissspidy

This comment was marked as resolved.

@swissspidy

This comment was marked as resolved.

@spacedmonkey

This comment was marked as resolved.

@spacedmonkey
Copy link
Contributor

When I insert the video into canvas, the time is wrong.

Screenshot 2022-07-21 at 13 56 18

@spacedmonkey
Copy link
Contributor

Screen.Recording.2022-07-21.at.14.00.29.mov

On a 40 second video, it takes nearly 40 seconds for the trim option to show up.

…rs/web-stories-wp into add/11685-trim-recording
Copy link
Contributor

@merapi merapi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

We will deal with the remaining items here #11988

@barklund barklund merged commit be90df1 into main Jul 25, 2022
@barklund barklund deleted the add/11685-trim-recording branch July 25, 2022 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Enhancement New feature or improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Play button is unavailable to users when trimming a gif Media Recording: Add Video Trimming
7 participants