Client-side media: Add animated GIF to video conversion (MP4/WebM)#76946
Open
adamsilverstein wants to merge 12 commits intotrunkfrom
Open
Client-side media: Add animated GIF to video conversion (MP4/WebM)#76946adamsilverstein wants to merge 12 commits intotrunkfrom
adamsilverstein wants to merge 12 commits intotrunkfrom
Conversation
Introduce client-side animated GIF to MP4/WebM conversion using FFmpeg WASM, following the architecture proven in swissspidy/media-experiments. New @wordpress/ffmpeg package: - FFmpeg WASM wrapper with inlined base64 WASM (same approach as @wordpress/vips) - Web Worker architecture via Blob URL pattern - Lazy loaded only when an animated GIF upload is detected Upload pipeline integration: - New TranscodeGif operation type in the upload queue - isAnimatedGif() binary analysis utility for detecting multi-frame GIFs - Concurrency limited to 1 simultaneous video transcode - Graceful fallback when crossOriginIsolated is unavailable Settings: - gifConvert: enable/disable conversion (default: true) - videoOutputFormat: output format preference (default: video/mp4) Closes #76942
Member
Author
WASM Loading Strategy DiscussionThe current implementation inlines the FFmpeg WASM as base64 (matching the Option 1: Inlined base64 (current implementation)The WASM binary is inlined as a base64 data URL at build time, bundled into the worker code string. Pros:
Cons:
Option 2: CDN loading (e.g., jsdelivr, unpkg)Load the WASM from a public CDN at runtime, only when needed. Pros:
Cons:
Option 3: Canonical WordPress plugin with on-demand installationShip FFmpeg WASM as a separate canonical (WordPress.org-hosted) plugin. On first animated GIF upload, upon user confirmation install/activate it in the background, adding FFmpeg capability to the site. Pros:
Cons:
RecommendationFor the initial implementation, the inlined approach (Option 1) is simplest and matches existing patterns. However, given the ~33MB size, Option 3 (canonical plugin) is worth exploring as a follow-up — it keeps everything within the WordPress ecosystem while avoiding the bundle size impact. The upload-media pipeline is already designed for graceful fallback, so the detection/installation flow could be added without changing the core architecture. |
Address CodeRabbit review findings: - Add serialization lock to prevent concurrent operations from corrupting shared FFmpeg MEMFS state - Use unique filenames per operation (keyed by item ID) - Add cancellation checks after async boundaries (lock wait, core init) - Fix output.buffer to use proper slice to avoid including bytes outside the Uint8Array view range
Move the heavy FFmpeg WASM binary (~31MB) out of Gutenberg into a separate canonical WordPress plugin (wp-ffmpeg-wasm). This eliminates the ~33MB base64-inlined WASM from the Gutenberg bundle. Architecture: - The wp-ffmpeg-wasm plugin ships the WASM binary as a static asset and exposes URLs via window.__ffmpegWasmConfig and a REST endpoint - Gutenberg's @wordpress/ffmpeg package becomes a thin wrapper that loads WASM from the plugin's URLs instead of inlining - On first animated GIF upload, Gutenberg silently installs the plugin via the REST API (same pattern as Connectors plugin installation) - Graceful fallback: if plugin can't be installed, GIF uploads as-is Key changes: - Remove @ffmpeg/core dependency from packages/ffmpeg - FFmpeg index.ts accepts coreUrl/wasmUrl params instead of inlining - New ffmpeg-plugin.ts utility handles plugin detection and installation - Remove gutenberg_enqueue_ffmpeg_loader from client-assets.php - Add @wordpress/core-data dependency to upload-media for plugin install The canonical plugin (wp-ffmpeg-wasm) is created separately and will be hosted on WordPress.org.
This reverts commit 7a3db7f.
5 tasks
Member
Author
|
Trying the canonical plugin approach in this PR: #76964 |
Two build issues: - wasmInlinePlugin regex matches paths ending in .wasm, but '@ffmpeg/core/wasm' is a package exports alias that doesn't match. Use the direct path '@ffmpeg/core/dist/umd/ffmpeg-core.wasm' instead. - @ffmpeg/core ESM entry uses import.meta.url which breaks in Blob URL workers. Add wpWorkers resolve mapping to redirect esm/ffmpeg-core.js to umd/ffmpeg-core.js (same pattern as vips).
The @ffmpeg/core package uses an exports map where './wasm' resolves to the .wasm file. The wasmInlinePlugin filter only matched paths ending in '.wasm', missing the '/wasm' alias. Direct paths like '@ffmpeg/core/dist/umd/ffmpeg-core.wasm' were also blocked by esbuild's exports map enforcement. Fix by broadening the wasmInlinePlugin regex to also match '/wasm' suffixes, with a guard that verifies the resolved path actually ends in '.wasm'. This supports both direct .wasm imports (wasm-vips pattern) and package exports aliases (@ffmpeg/core pattern). Also add wpWorkers resolve mapping to redirect the @ffmpeg/core ESM entry (which uses import.meta.url) to the UMD entry.
|
Size Change: +14.1 MB (+180.76%) 🆘 Total Size: 22 MB 📦 View Changed
ℹ️ View Unchanged
|
|
Flaky tests detected in ba936b8. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/25137100523
|
7 tasks
Resolve conflicts in: - package-lock.json (regenerated with npm install) - packages/upload-media/src/store/private-actions.ts (combine HEIC handling with GIF-to-video) - packages/upload-media/src/store/types.ts (adopt new mediaFinalize signature alongside gifConvert settings)
adamsilverstein
requested review from
ajitbohra,
fabiankaegy,
juanmaguitar and
spacedmonkey
as code owners
|
Warning: Type of PR label mismatch To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.
Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task. |
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
adamsilverstein
requested review from
andrewserong and
swissspidy
and removed request for
ajitbohra,
fabiankaegy,
juanmaguitar and
spacedmonkey
adamsilverstein
added
[Status] In Progress
Cover the wrapper at packages/upload-media/src/store/utils/ffmpeg.ts (output file naming for mp4/webm, argument forwarding, module-cache no-ops before load) with a jest moduleNameMapper stub for @wordpress/ffmpeg/worker that matches the existing @wordpress/vips/worker pattern. Extend packages/upload-media/src/store/test/private-actions.js with tests for transcodeGifItem (success dispatches CacheBlobUrl + finishOperation, failure dispatches cancelItem with a GIF_TRANSCODING_ERROR UploadError, early return when item missing) and for the prepareItem animated-GIF branch (mp4 default, webm override, additionalData preservation, gifConvert=false bypass, non-isolated bypass, static-GIF fallthrough). Add selector tests for getActiveVideoProcessingCount and getPendingVideoProcessing covering mixed-queue scenarios.
Member
Author
|
As expected, this PR adds significant weight to the plugin bundle: #76946 (comment) (14 MB currently) |
Member
Author
|
I will work on the backport PR once the Client Side Media feature is re-introduced to core. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds client-side animated GIF to MP4/WebM conversion using FFmpeg WASM during upload, following the architecture proven in swissspidy/media-experiments.
Why: Animated GIFs are 5-10x larger than equivalent MP4/WebM videos. Converting at upload time dramatically reduces file sizes, improves page performance, and uses hardware-accelerated video playback instead of CPU-bound GIF decoding — all transparently to the user.
What's included
New
@wordpress/ffmpegpackage — FFmpeg WASM wrapper mirroring@wordpress/vipsarchitecture:Upload pipeline integration (
@wordpress/upload-media):TranscodeGifoperation type with concurrency limit of 1isAnimatedGif()binary analysis utility (checks GIF magic bytes + frame count)transcodeGifItem()handler following thetranscodeImageItem()patterngifConvert(enable/disable) andvideoOutputFormat(mp4/webm)crossOriginIsolatedcontext (SharedArrayBuffer) — graceful fallbackPHP enqueue — Registers
@wordpress/ffmpeg/loaderin the import mapWhat's NOT included (follow-up PRs)
core/videoCloses #76942
Test plan
isAnimatedGif()— 6 tests covering animated GIFs, static GIFs, non-GIF files, edge casesnpm run buildwith ffmpeg WASM inlining)crossOriginIsolatedis unavailable, eg. in Safari you get existing behavior (no movie conversion)