From 964a3de04c21ef92a03671090bb0ce4cf932e7f1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 1 Dec 2025 06:14:23 +0000 Subject: [PATCH 1/3] Initial plan From afdd64581b190e46bfa27f7213cc1276cd1cc150 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 1 Dec 2025 06:21:44 +0000 Subject: [PATCH 2/3] Add FFmpeg N-API bindings research and related tasks Co-authored-by: subtleGradient <4117+subtleGradient@users.noreply.github.com> --- research/README.md | 6 +- research/ffmpeg-napi.md | 135 ++++++++++ research/nodejs-implementation.md | 11 + research/nodejs-linux-napi-ffmpeg.md | 2 + research/options.md | 94 ++++++- tasks/README.md | 7 +- tasks/evaluate-ffmpeg-napi-bindings.md | 179 +++++++++++++ tasks/napi-poc-addon.md | 7 + tasks/videodecoder-shim-node-av.md | 332 +++++++++++++++++++++++++ 9 files changed, 759 insertions(+), 14 deletions(-) create mode 100644 research/ffmpeg-napi.md create mode 100644 tasks/evaluate-ffmpeg-napi-bindings.md create mode 100644 tasks/videodecoder-shim-node-av.md diff --git a/research/README.md b/research/README.md index 8e490dc..d8882bb 100644 --- a/research/README.md +++ b/research/README.md @@ -24,11 +24,14 @@ Each runtime has unique characteristics and optimal implementation paths: - **[nodejs-linux-napi-ffmpeg.md](./nodejs-linux-napi-ffmpeg.md)** - Detailed technical research for implementing WebCodecs in Node.js on Linux using N-API and FFmpeg. Covers architecture, threading, memory management, codec mapping, and distribution. +- **[ffmpeg-napi.md](./ffmpeg-napi.md)** - **NEW**: Research on existing FFmpeg N-API bindings (node-av, @mmomtchev/ffmpeg, etc.) that can serve as a foundation for WebCodecs implementation, avoiding the need to build raw N-API bindings from scratch. + ## Quick Summary | Runtime | Approach | Key Advantage | Primary Challenge | |---------|----------|---------------|-------------------| | **Node.js** | N-API + FFmpeg | Mature addon ecosystem | Threading/memory complexity | +| **Node.js (alt)** | Existing N-API bindings (node-av) | Skip raw N-API work | Adapter layer needed | | **Deno** | Rust + codec libs | Type-safe, clean architecture | Codec library availability | | **Bun** | WebKit leverage | WebCodecs may already exist | Media backend wiring | @@ -42,7 +45,8 @@ For detailed implementation tasks extracted from research: 1. Start with **[webcodecs-overview.md](./webcodecs-overview.md)** for context on the problem space 2. Review **[options.md](./options.md)** for technical implementation strategies -3. Dive into the runtime-specific document for your target platform +3. **NEW**: Check **[ffmpeg-napi.md](./ffmpeg-napi.md)** for existing N-API bindings that can accelerate development +4. Dive into the runtime-specific document for your target platform ## External Resources diff --git a/research/ffmpeg-napi.md b/research/ffmpeg-napi.md new file mode 100644 index 0000000..1ce42f0 --- /dev/null +++ b/research/ffmpeg-napi.md @@ -0,0 +1,135 @@ +# Existing FFmpeg N-API Bindings Research + +> **Parent document**: [Node.js Implementation Tasks](./nodejs-implementation.md) +> **Related**: [Node.js Linux N-API + FFmpeg Research](./nodejs-linux-napi-ffmpeg.md) | [Implementation Options](./options.md) + +## Summary + +| Aspect | Finding | +|--------|---------| +| **Outcome** | Yes, there are existing N-API FFmpeg bindings you can stand on. | +| **Obstacles** | None of them are WebCodecs-shaped; you still need an adapter layer. | +| **Plan** | Reuse an FFmpeg N-API library as the "engine" and build WebCodecs on top. | + +--- + +## Key Finding + +There are already serious FFmpeg bindings built on Node-API, so you don't have to start from raw C FFmpeg + N-API. This avoids the largest yak: raw FFmpeg + Node-API integration. + +--- + +## 1. Strong Candidates + +### node-av (SeydX) + +- **Repository**: [seydx/node-av](https://github.com/seydx/node-av) +- Native FFmpeg v8 bindings for Node.js. +- Uses a compiled addon (Node-API) with: + - **Low-level API**: `FormatContext`, `CodecContext`, `Frame`, `Packet`, etc., very close to libav* C API. + - **High-level API**: `Demuxer`, `Decoder`, `Encoder`, `Muxer`, plus a "pipeline" API that already models decode/encode chains. +- Supports hardware acceleration (CUDA, VAAPI, auto-detect) and exposes `HardwareContext`. +- Ships prebuilt binaries for all major platforms; on Linux you can just `npm install node-av`. +- Fully typed for TypeScript, modern async/await patterns. + +**This is probably the best "drop-in engine" for a Linux-only WebCodecs implementation right now.** + +--- + +### @mmomtchev/ffmpeg / node-ffmpeg (avcpp + nobind17) + +- **Repository**: [mmomtchev/ffmpeg](https://github.com/mmomtchev/ffmpeg) (library name often referred to as node-ffmpeg). +- Wraps avcpp (a C++ wrapper around FFmpeg's C API) and exposes it to Node via nobind17, which itself sits on Node-API / node-addon-api. +- Provides: + - A fairly complete mapping of FFmpeg (demuxers, decoders, encoders, filters). + - A Node.js Streams API: `Demuxer`, `AudioDecoder`, `VideoDecoder`, `Filter`, `VideoTransform`, `AudioTransform`, `VideoEncoder`, `AudioEncoder`, `Muxer`, etc. +- Designed to allow safe-ish async use and multi-threading, with some explicit caveats around misconfigured streams. + +**Good fit if you like the streams model and don't mind the extra avcpp layer.** + +--- + +### libav (Astronaut Labs) + +- **Package**: `libav` / **Repository**: [astronautlabs/libav](https://github.com/AstronautLabs/libav) +- Node.js bindings to FFmpeg's libav* libraries, adapted to be "comfortable and efficient" in TypeScript. +- Uses Node-API as well, but: + - Marked **"Pre-Alpha Quality; incomplete, broken, heavy development"**. + - Requires system FFmpeg 5.x and dev headers installed on Linux. + +**Probably better as inspiration/reference than a solid base for a WebCodecs project right now.** + +--- + +### ff-helper (napi-rs-based helper) + +- **Package**: `ff-helper` +- A napi-rs binding wrapping a few FFmpeg features, like getting video info and generating screenshots. +- Explicitly **not** a full FFmpeg binding; just a helper. + +**Useful as a tiny example of napi-rs + FFmpeg, but not enough for WebCodecs.** + +--- + +## 2. How This Helps the WebCodecs Plan (Linux-only) + +You can absolutely avoid writing raw C++/Node-API around FFmpeg from scratch: + +1. **Pick one of the full bindings as your "FFmpegService"**: + - If you want modern TS + pipelines + prebuilt binaries → **node-av**. + - If you want streams + avcpp and don't mind extra C++ layer → **node-ffmpeg**. + +2. **Implement WebCodecs classes** (`VideoDecoder`, `VideoEncoder`, `VideoFrame`, etc.) **in TypeScript**, and internally: + - Use node-av's `Decoder` / `Encoder` or node-ffmpeg's `VideoDecoder` / `VideoEncoder` to do the actual work. + - Map WebCodecs `configure()` → underlying codec/stream setup. + - Map `decode()`/`encode()` → feed frames/packets into the N-API binding. + - Use their existing thread/memory/resource management instead of building your own. + +### What You'd Still Need to Build + +You'd still need to: +- Reproduce **WebCodecs semantics** (state machine, reset, close, `isConfigSupported`, error types). +- Possibly wrap/normalize timestamps, formats, and pixel layouts to match spec. + +…but you **avoid the largest yak: raw FFmpeg + Node-API integration**. + +--- + +## 3. Concrete Recommendation + +For a Linux-first WebCodecs-in-Node prototype: + +1. **Start with node-av as the backend**: + - Use its high-level `Decoder`/`Encoder` for quick progress; drop to low-level `CodecContext`/`Frame` when you need more control. + +2. **Wrap it in TS classes that exactly match the WebCodecs IDL**. + +3. **Only reach for your own N-API code** if you discover a semantic gap you cannot bridge cleanly via node-av. + +--- + +## 4. Next Steps + +See extracted tasks for actionable follow-ups: + +- [Evaluate FFmpeg N-API Bindings](../tasks/evaluate-ffmpeg-napi-bindings.md) — Compare node-av vs node-ffmpeg for WebCodecs use +- [VideoDecoder Shim on node-av](../tasks/videodecoder-shim-node-av.md) — Sketch a VideoDecoder shim that sits on top of node-av.Decoder + +--- + +## 5. Library Comparison Matrix + +| Library | API Style | Prebuilt Binaries | HW Accel | TypeScript | Maturity | Best For | +|---------|-----------|-------------------|----------|------------|----------|----------| +| **node-av** | High + Low level | ✅ All platforms | ✅ CUDA/VAAPI | ✅ Full | Production | Quick WebCodecs prototype | +| **@mmomtchev/ffmpeg** | Streams | ✅ | ⚠️ Limited | ✅ | Stable | Stream-based pipelines | +| **libav (Astronaut)** | Low-level | ❌ | ❌ | ✅ | Pre-alpha | Reference/learning | +| **ff-helper** | Utilities | ✅ | ❌ | ✅ | Minimal | Quick video info | + +--- + +## Related Documents + +- [Node.js Linux N-API + FFmpeg Research](./nodejs-linux-napi-ffmpeg.md) — Original from-scratch N-API design +- [Node.js Implementation Tasks](./nodejs-implementation.md) — Higher-level task breakdown +- [Implementation Options](./options.md) — Comparison of implementation approaches diff --git a/research/nodejs-implementation.md b/research/nodejs-implementation.md index 9718707..d57aa1c 100644 --- a/research/nodejs-implementation.md +++ b/research/nodejs-implementation.md @@ -5,6 +5,8 @@ This document outlines the research tasks needed to implement WebCodecs in Node. > **Parent document**: [WebCodecs Overview](./webcodecs-overview.md) > > **Detailed research**: [Node.js Linux N-API + FFmpeg Research](./nodejs-linux-napi-ffmpeg.md) — Deep-dive into architecture, threading, memory management, and FFmpeg integration for Linux. +> +> ⭐ **NEW — Faster path**: [FFmpeg N-API Bindings Research](./ffmpeg-napi.md) — Existing production-ready FFmpeg N-API bindings that can serve as the backend, avoiding raw N-API development. > > **Implementation tasks**: [tasks/](../tasks/) — Extracted actionable tasks with YAML frontmatter for tracking. @@ -14,6 +16,15 @@ This document outlines the research tasks needed to implement WebCodecs in Node. Node.js has the most mature ecosystem for native addons and the strongest community demand (evidenced by the $10k challenge). The implementation will likely involve C++ bindings via N-API with FFmpeg or OS-level codec APIs. +### ⭐ New Finding: Existing N-API Bindings Available + +Recent research discovered that production-ready FFmpeg N-API bindings already exist (see [FFmpeg N-API Bindings Research](./ffmpeg-napi.md)): + +- **node-av**: Full-featured FFmpeg bindings with prebuilts and HW acceleration +- **@mmomtchev/ffmpeg**: Streams-based FFmpeg API + +This means we can **skip raw N-API development** and instead build a WebCodecs-compliant adapter layer on top of these existing libraries. See [Evaluate FFmpeg N-API Bindings](../tasks/evaluate-ffmpeg-napi-bindings.md) and [VideoDecoder Shim on node-av](../tasks/videodecoder-shim-node-av.md) for actionable tasks. + --- ## Research Tasks diff --git a/research/nodejs-linux-napi-ffmpeg.md b/research/nodejs-linux-napi-ffmpeg.md index 76386d1..53889c3 100644 --- a/research/nodejs-linux-napi-ffmpeg.md +++ b/research/nodejs-linux-napi-ffmpeg.md @@ -2,6 +2,8 @@ > **Parent document**: [WebCodecs Overview](./webcodecs-overview.md) > **Related**: [Node.js Implementation Tasks](./nodejs-implementation.md) | [Implementation Options](./options.md) +> +> ⚠️ **Important Update**: See [FFmpeg N-API Bindings Research](./ffmpeg-napi.md) for an **alternative faster path** using existing production-ready FFmpeg N-API bindings (node-av, @mmomtchev/ffmpeg). This document describes the from-scratch approach, which may still be useful if the existing bindings have gaps, but the shim-based approach is now recommended. **Scope** diff --git a/research/options.md b/research/options.md index 3dcb4ec..a3ead34 100644 --- a/research/options.md +++ b/research/options.md @@ -2,7 +2,9 @@ This document explores various approaches to implementing the WebCodecs API in Node.js, based on research of existing implementations and related projects. -> **See also**: [Node.js Linux N-API + FFmpeg Research](./nodejs-linux-napi-ffmpeg.md) for detailed architecture and implementation guidance for the FFmpeg approach on Linux. +> **See also**: +> - [Node.js Linux N-API + FFmpeg Research](./nodejs-linux-napi-ffmpeg.md) for detailed architecture and implementation guidance for the FFmpeg approach on Linux. +> - **NEW**: [FFmpeg N-API Bindings Research](./ffmpeg-napi.md) for existing N-API bindings that can accelerate development. ## Overview @@ -10,9 +12,9 @@ The WebCodecs API provides low-level access to media encoders and decoders. Impl ## Implementation Options -### Option 1: FFmpeg via N-API +### Option 1: FFmpeg via N-API (From Scratch) -**Description**: Create Node.js native bindings to FFmpeg's libavcodec, libavformat, and related libraries. +**Description**: Create Node.js native bindings to FFmpeg's libavcodec, libavformat, and related libraries from scratch. > 📖 **Detailed research available**: [Node.js Linux N-API + FFmpeg Research](./nodejs-linux-napi-ffmpeg.md) covers architecture, threading, memory management, codec mapping, licensing, and distribution for this approach. @@ -43,6 +45,44 @@ The WebCodecs API provides low-level access to media encoders and decoders. Impl --- +### Option 1b: Existing FFmpeg N-API Bindings + WebCodecs Shim ⭐ NEW + +**Description**: Use an existing, production-ready FFmpeg N-API binding (such as node-av) and build a thin WebCodecs-compliant adapter layer on top. + +> 📖 **Detailed research available**: [FFmpeg N-API Bindings Research](./ffmpeg-napi.md) covers available libraries, comparison, and recommended approach. + +**Approach**: +1. Select an existing FFmpeg N-API library (recommended: node-av) +2. Build WebCodecs-compliant TypeScript classes (`VideoDecoder`, `VideoEncoder`, etc.) +3. Map WebCodecs API calls to the underlying library's API +4. Implement WebCodecs state machine and error semantics in the adapter layer + +**Pros**: +- **Avoids the largest yak**: raw FFmpeg + N-API integration already done +- Prebuilt binaries available for all major platforms +- Hardware acceleration support (node-av supports CUDA, VAAPI) +- Full TypeScript support with modern async patterns +- Significant time savings (estimated 2-3 weeks) +- Battle-tested FFmpeg integration + +**Cons**: +- Dependent on third-party library maintenance +- May have semantic gaps requiring workarounds +- Less control over low-level behavior +- Need to map between two different APIs + +**Effort Estimate**: **Low-Medium** (significantly lower than from-scratch) + +**Key Libraries**: +- **node-av (recommended)**: [seydx/node-av](https://github.com/seydx/node-av) — Best combination of features, prebuilds, and HW accel +- **@mmomtchev/ffmpeg**: [mmomtchev/ffmpeg](https://github.com/mmomtchev/ffmpeg) — Streams-based API, good if you prefer that model + +**Recommended Next Steps**: +- [Evaluate FFmpeg N-API Bindings](../tasks/evaluate-ffmpeg-napi-bindings.md) +- [VideoDecoder Shim on node-av](../tasks/videodecoder-shim-node-av.md) + +--- + ### Option 2: WebKit/Chromium Port **Description**: Extract the WebCodecs implementation from WebKit or Chromium and adapt it for standalone use. @@ -188,7 +228,8 @@ The WebCodecs API provides low-level access to media encoders and decoders. Impl | Option | Effort | Performance | Portability | Maintenance | |--------|--------|-------------|-------------|-------------| -| FFmpeg N-API | High | Excellent | Medium | Medium | +| **FFmpeg N-API Bindings + Shim** ⭐ | **Low-Medium** | **Excellent** | **Good** | **Low** | +| FFmpeg N-API (from scratch) | High | Excellent | Medium | Medium | | Browser Port | Very High | Excellent | Low | High | | Bun-Style | Medium | Good | Low | Medium | | WASM | Low | Medium | Excellent | Low | @@ -199,12 +240,38 @@ The WebCodecs API provides low-level access to media encoders and decoders. Impl For the $10k challenge timeline (1 month), we recommend: -### Phase 1: Quick Start with WASM (Week 1-2) +### ⭐ NEW: Fastest Path — Existing N-API Bindings + WebCodecs Shim + +Based on the discovery of existing production-ready FFmpeg N-API bindings (see [FFmpeg N-API Bindings Research](./ffmpeg-napi.md)), the fastest path is now: + +**Week 1: Foundation** +1. Evaluate node-av and @mmomtchev/ffmpeg ([Evaluate FFmpeg N-API Bindings](../tasks/evaluate-ffmpeg-napi-bindings.md)) +2. Select the best candidate (likely node-av) +3. Build VideoDecoder shim ([VideoDecoder Shim on node-av](../tasks/videodecoder-shim-node-av.md)) + +**Week 2: Core Classes** +1. Complete VideoDecoder with state machine, error handling +2. Build VideoEncoder shim +3. Implement VideoFrame and EncodedVideoChunk wrappers + +**Week 3-4: Polish + Testing** +1. Add AudioDecoder, AudioEncoder shims +2. Run WPT test subset +3. Implement isConfigSupported() accurately +4. Documentation and examples + +This approach can save **2-3 weeks** compared to building raw N-API bindings from scratch. + +### Alternative: WASM + Native Hybrid (Original Recommendation) + +If the N-API binding approach hits blockers, fall back to: + +#### Phase 1: Quick Start with WASM (Week 1-2) 1. Use libavjs-webcodecs-polyfill as a starting point 2. Adapt for Node.js environment 3. Get tests passing with basic functionality -### Phase 2: Optimize with Native Bindings (Week 3-4) +#### Phase 2: Optimize with Native Bindings (Week 3-4) 1. Add FFmpeg N-API bindings for performance-critical codecs (VP8, H.264) 2. Use WASM as fallback 3. Implement hardware acceleration detection @@ -221,14 +288,17 @@ For the $10k challenge timeline (1 month), we recommend: 2. **Mediabunny** - High-level video editing on WebCodecs 3. **Effect-TS** - Functional patterns for async operations 4. **libavjs-webcodecs-polyfill** - Pure JS WebCodecs polyfill +5. **node-av** ⭐ NEW - Production-ready FFmpeg N-API bindings with HW accel +6. **@mmomtchev/ffmpeg** ⭐ NEW - FFmpeg bindings with Node.js Streams API ## Conclusion -The most practical approach for rapid development is to start with WebAssembly-based codecs (Option 4) and progressively add native optimizations (Option 1). This provides: +**Updated recommendation**: The most practical approach for rapid development is now to leverage existing FFmpeg N-API bindings (Option 1b) and build a WebCodecs shim layer on top: -1. **Immediate functionality** - Get tests passing quickly -2. **Broad compatibility** - Works on any platform -3. **Performance path** - Clear upgrade route to native -4. **Lower risk** - Incremental development +1. **Fastest time to working code** — Skip raw N-API development +2. **Production-ready FFmpeg integration** — Already battle-tested +3. **Hardware acceleration included** — node-av supports CUDA, VAAPI +4. **Excellent performance** — Native FFmpeg, not WASM +5. **Lower risk** — Smaller codebase to maintain -The WASM approach using libavjs-webcodecs-polyfill can likely get basic functionality working within days, providing a foundation for further optimization. +If the N-API binding approach encounters blockers, the WASM approach (Option 4) remains a viable fallback that can get basic functionality working within days. diff --git a/tasks/README.md b/tasks/README.md index 03ae1aa..dfaef2f 100644 --- a/tasks/README.md +++ b/tasks/README.md @@ -24,10 +24,14 @@ research: ../research/relevant-doc.md | Task | Priority | Effort | Status | |------|----------|--------|--------| +| [Evaluate FFmpeg N-API Bindings](./evaluate-ffmpeg-napi-bindings.md) | critical | medium | todo | +| [VideoDecoder Shim on node-av](./videodecoder-shim-node-av.md) | critical | medium | todo | | [N-API PoC Addon](./napi-poc-addon.md) | critical | large | todo | | [FFmpeg Static Build](./ffmpeg-static-build.md) | critical | medium | todo | | [Codec String Parser](./codec-string-parser.md) | high | small | todo | +> **Note**: The first two tasks (Evaluate FFmpeg N-API Bindings and VideoDecoder Shim) represent an alternative faster path using existing N-API bindings. See [FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md) for details. + ### Week 2: Threading + Memory | Task | Priority | Effort | Status | @@ -45,6 +49,7 @@ research: ../research/relevant-doc.md ## Related Research -- [Node.js Linux N-API + FFmpeg Research](../research/nodejs-linux-napi-ffmpeg.md) — Primary research document +- [FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md) — **NEW**: Existing N-API bindings that can accelerate development +- [Node.js Linux N-API + FFmpeg Research](../research/nodejs-linux-napi-ffmpeg.md) — Primary research document (from-scratch approach) - [Node.js Implementation Overview](../research/nodejs-implementation.md) — Higher-level task breakdown - [Implementation Options](../research/options.md) — Comparison of implementation approaches diff --git a/tasks/evaluate-ffmpeg-napi-bindings.md b/tasks/evaluate-ffmpeg-napi-bindings.md new file mode 100644 index 0000000..adf3ed0 --- /dev/null +++ b/tasks/evaluate-ffmpeg-napi-bindings.md @@ -0,0 +1,179 @@ +--- +title: Evaluate FFmpeg N-API Bindings +status: todo +priority: critical +effort: medium +category: architecture +dependencies: [] +research: ../research/ffmpeg-napi.md +timeline: Week 1 +--- + +# Evaluate FFmpeg N-API Bindings + +Evaluate existing FFmpeg N-API libraries to determine the best foundation for WebCodecs implementation. + +## Objective + +Compare node-av, @mmomtchev/ffmpeg, and other existing FFmpeg Node-API bindings to select the optimal backend for a WebCodecs shim layer. + +## Background + +From [Existing FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md): + +> There are already serious FFmpeg bindings built on Node-API, so you don't have to start from raw C FFmpeg + N-API. This avoids the largest yak: raw FFmpeg + Node-API integration. + +## Candidates to Evaluate + +### 1. node-av (SeydX) + +- **Repository**: [seydx/node-av](https://github.com/seydx/node-av) +- High-level and low-level APIs +- Prebuilt binaries for all platforms +- Hardware acceleration support (CUDA, VAAPI) +- Full TypeScript support + +### 2. @mmomtchev/ffmpeg + +- **Repository**: [mmomtchev/ffmpeg](https://github.com/mmomtchev/ffmpeg) +- Node.js Streams-based API +- Uses avcpp + nobind17 +- Full codec coverage + +### 3. libav (Astronaut Labs) + +- **Repository**: [astronautlabs/libav](https://github.com/AstronautLabs/libav) +- Low-level bindings +- Pre-alpha quality +- Requires system FFmpeg + +## Evaluation Criteria + +| Criterion | Weight | Description | +|-----------|--------|-------------| +| API Compatibility | High | How well does the API map to WebCodecs concepts? | +| Prebuilt Binaries | High | Available for Linux x64/arm64? | +| HW Acceleration | Medium | CUDA/VAAPI support? | +| Maintenance | High | Active development? Recent commits? | +| Documentation | Medium | TypeScript types? Examples? | +| Performance | High | Overhead compared to raw FFmpeg? | +| Error Handling | Medium | Graceful errors that can map to WebCodecs errors? | + +## Tasks + +- [ ] Install node-av and test basic decode/encode +- [ ] Install @mmomtchev/ffmpeg and test basic decode/encode +- [ ] Compare API surface to WebCodecs spec requirements +- [ ] Test hardware acceleration detection +- [ ] Measure decode performance (1080p H.264) +- [ ] Measure encode performance (1080p VP8) +- [ ] Document API mapping gaps +- [ ] Check TypeScript types quality +- [ ] Review error handling patterns +- [ ] Document licensing implications +- [ ] Write recommendation report + +## Test Cases + +### Basic Decode Test + +```typescript +import { Decoder } from 'node-av'; // or equivalent + +async function testDecode() { + // 1. Configure decoder (map to WebCodecs config) + const decoder = new Decoder({ + codec: 'h264', + // ... other options + }); + + // 2. Feed encoded chunk + const result = await decoder.decode(encodedData); + + // 3. Get decoded frame + console.log('Frame:', result.width, result.height); +} +``` + +### Basic Encode Test + +```typescript +import { Encoder } from 'node-av'; // or equivalent + +async function testEncode() { + // 1. Configure encoder + const encoder = new Encoder({ + codec: 'vp8', + width: 1920, + height: 1080, + bitrate: 2_000_000, + }); + + // 2. Feed raw frame + const chunk = await encoder.encode(frameData); + + // 3. Get encoded output + console.log('Chunk size:', chunk.byteLength); +} +``` + +## API Mapping Analysis + +Document how each library's API maps to WebCodecs: + +| WebCodecs | node-av | @mmomtchev/ffmpeg | +|-----------|---------|-------------------| +| `VideoDecoder.configure()` | `new Decoder(config)` | TBD | +| `VideoDecoder.decode(chunk)` | `decoder.decode(data)` | TBD | +| `VideoDecoder.flush()` | `decoder.flush()` | TBD | +| `VideoDecoder.reset()` | TBD | TBD | +| `VideoDecoder.close()` | `decoder.close()` | TBD | +| `VideoEncoder.configure()` | `new Encoder(config)` | TBD | +| `VideoEncoder.encode(frame)` | `encoder.encode(data)` | TBD | +| `VideoFrame` | `Frame` | TBD | +| `EncodedVideoChunk` | `Packet` | TBD | + +## Acceptance Criteria + +1. Both libraries tested with H.264 decode +2. Both libraries tested with VP8/VP9 encode +3. Performance comparison documented +4. API mapping table completed +5. Recommendation made with justification +6. Licensing implications documented +7. Any blockers or gaps identified + +## Deliverables + +- [ ] Test scripts for each library +- [ ] API mapping documentation +- [ ] Performance benchmark results +- [ ] Recommendation report +- [ ] List of gaps/blockers for WebCodecs shim + +## Recommendation Template + +```markdown +## Recommendation: [Library Name] + +### Justification +- ... + +### Trade-offs +- Pros: ... +- Cons: ... + +### Gaps to Address +1. ... +2. ... + +### Estimated Effort +- WebCodecs shim: X weeks +- vs. from-scratch N-API: Y weeks (savings of Z weeks) +``` + +## Related + +- [FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md) +- [VideoDecoder Shim on node-av](./videodecoder-shim-node-av.md) +- [N-API PoC Addon](./napi-poc-addon.md) — Original from-scratch approach diff --git a/tasks/napi-poc-addon.md b/tasks/napi-poc-addon.md index 7a8f0ce..ba336de 100644 --- a/tasks/napi-poc-addon.md +++ b/tasks/napi-poc-addon.md @@ -13,6 +13,13 @@ timeline: Week 1 Build a minimal N-API addon that validates the core architecture patterns for WebCodecs bindings. +> ⚠️ **Alternative Path Available**: Consider using existing FFmpeg N-API bindings (node-av, @mmomtchev/ffmpeg) instead of building from scratch. See: +> - [FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md) +> - [Evaluate FFmpeg N-API Bindings](./evaluate-ffmpeg-napi-bindings.md) +> - [VideoDecoder Shim on node-av](./videodecoder-shim-node-av.md) +> +> This task is still valuable if the existing bindings have gaps, or if you want full control over the native layer. + ## Objective Create a `DummyDecoder` N-API class that demonstrates: diff --git a/tasks/videodecoder-shim-node-av.md b/tasks/videodecoder-shim-node-av.md new file mode 100644 index 0000000..18f70e1 --- /dev/null +++ b/tasks/videodecoder-shim-node-av.md @@ -0,0 +1,332 @@ +--- +title: VideoDecoder Shim on node-av +status: todo +priority: critical +effort: medium +category: implementation +dependencies: + - evaluate-ffmpeg-napi-bindings.md +research: ../research/ffmpeg-napi.md +timeline: Week 1 +--- + +# VideoDecoder Shim on node-av + +Sketch and implement a WebCodecs-compliant VideoDecoder that uses node-av as the backend engine. + +## Objective + +Create a `VideoDecoder` class that: +- Matches the WebCodecs `VideoDecoder` API exactly +- Uses node-av's `Decoder` internally for actual decoding +- Handles WebCodecs state machine semantics +- Properly maps codec strings to FFmpeg codecs + +## Background + +From [Existing FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md#3-concrete-recommendation): + +> For a Linux-first WebCodecs-in-Node prototype: +> 1. Start with node-av as the backend +> 2. Wrap it in TS classes that exactly match the WebCodecs IDL +> 3. Only reach for your own N-API code if you discover a semantic gap + +## Tasks + +- [ ] Study node-av Decoder API and types +- [ ] Study WebCodecs VideoDecoder spec +- [ ] Create `VideoDecoder` class skeleton matching WebCodecs IDL +- [ ] Implement state machine (`unconfigured` → `configured` → `closed`) +- [ ] Implement `configure()` method + - [ ] Parse codec string (reuse codec-string-parser) + - [ ] Map to node-av decoder config + - [ ] Handle hardware acceleration hint +- [ ] Implement `decode()` method + - [ ] Accept `EncodedVideoChunk` + - [ ] Feed to node-av decoder + - [ ] Wrap output as `VideoFrame` +- [ ] Implement `flush()` method +- [ ] Implement `reset()` method +- [ ] Implement `close()` method +- [ ] Implement `output` callback pattern +- [ ] Implement `error` callback pattern +- [ ] Implement `decodeQueueSize` property +- [ ] Implement static `isConfigSupported()` method +- [ ] Write unit tests +- [ ] Write integration tests with real video data + +## WebCodecs VideoDecoder Interface + +Reference from the WebCodecs spec: + +```typescript +interface VideoDecoder { + constructor(init: VideoDecoderInit); + + readonly state: CodecState; + readonly decodeQueueSize: number; + + configure(config: VideoDecoderConfig): void; + decode(chunk: EncodedVideoChunk): void; + flush(): Promise; + reset(): void; + close(): void; + + static isConfigSupported(config: VideoDecoderConfig): Promise; +} + +interface VideoDecoderInit { + output: VideoFrameOutputCallback; + error: WebCodecsErrorCallback; +} + +type CodecState = "unconfigured" | "configured" | "closed"; +``` + +## Implementation Sketch + +### VideoDecoder Class + +```typescript +import { Decoder } from 'node-av'; +import { parseCodecString } from './codec-parser'; +import { VideoFrame } from './VideoFrame'; +import { EncodedVideoChunk } from './EncodedVideoChunk'; + +type CodecState = 'unconfigured' | 'configured' | 'closed'; + +interface VideoDecoderInit { + output: (frame: VideoFrame) => void; + error: (error: DOMException) => void; +} + +export class VideoDecoder { + private _state: CodecState = 'unconfigured'; + private _outputCallback: (frame: VideoFrame) => void; + private _errorCallback: (error: DOMException) => void; + private _decoder: Decoder | null = null; + private _decodeQueueSize = 0; + + constructor(init: VideoDecoderInit) { + this._outputCallback = init.output; + this._errorCallback = init.error; + } + + get state(): CodecState { + return this._state; + } + + get decodeQueueSize(): number { + return this._decodeQueueSize; + } + + configure(config: VideoDecoderConfig): void { + if (this._state === 'closed') { + throw new DOMException('Decoder is closed', 'InvalidStateError'); + } + + // Parse codec string to get FFmpeg codec ID + const parsed = parseCodecString(config.codec); + if (!parsed) { + throw new DOMException(`Unsupported codec: ${config.codec}`, 'NotSupportedError'); + } + + // Create node-av decoder with mapped config + this._decoder = new Decoder({ + codec: parsed.ffmpegCodecId, + // Map hardware acceleration hint + hwAccel: config.hardwareAcceleration === 'prefer-hardware' ? 'auto' : undefined, + }); + + this._state = 'configured'; + } + + decode(chunk: EncodedVideoChunk): void { + if (this._state !== 'configured') { + throw new DOMException('Decoder not configured', 'InvalidStateError'); + } + + this._decodeQueueSize++; + + // Feed chunk to node-av decoder asynchronously + this._decoder!.decode(chunk.data) + .then((result) => { + this._decodeQueueSize--; + + // Wrap result as VideoFrame and deliver via callback + const frame = new VideoFrame(result.data, { + timestamp: chunk.timestamp, + duration: chunk.duration, + format: mapPixelFormat(result.format), + codedWidth: result.width, + codedHeight: result.height, + }); + + this._outputCallback(frame); + }) + .catch((err) => { + this._decodeQueueSize--; + this._errorCallback(new DOMException(err.message, 'OperationError')); + }); + } + + async flush(): Promise { + if (this._state !== 'configured') { + throw new DOMException('Decoder not configured', 'InvalidStateError'); + } + + // Flush node-av decoder + await this._decoder!.flush(); + + // Wait for queue to drain + while (this._decodeQueueSize > 0) { + await new Promise(resolve => setTimeout(resolve, 1)); + } + } + + reset(): void { + if (this._state === 'closed') { + throw new DOMException('Decoder is closed', 'InvalidStateError'); + } + + // Reset internal decoder state + this._decoder?.reset?.(); + this._decodeQueueSize = 0; + this._state = 'unconfigured'; + this._decoder = null; + } + + close(): void { + if (this._state === 'closed') { + return; + } + + this._decoder?.close?.(); + this._decoder = null; + this._decodeQueueSize = 0; + this._state = 'closed'; + } + + static async isConfigSupported(config: VideoDecoderConfig): Promise { + const parsed = parseCodecString(config.codec); + + if (!parsed) { + return { supported: false, config }; + } + + // Check if node-av supports this codec + // This might involve trying to create a decoder + const supported = await checkCodecSupport(parsed.ffmpegCodecId); + + return { + supported, + config: supported ? normalizeConfig(config) : config, + }; + } +} +``` + +### Key Mapping Functions + +```typescript +// Map node-av pixel format to WebCodecs VideoPixelFormat +function mapPixelFormat(format: string): VideoPixelFormat { + const formatMap: Record = { + 'yuv420p': 'I420', + 'nv12': 'NV12', + 'rgba': 'RGBA', + 'bgra': 'BGRA', + // ... more formats + }; + return formatMap[format] ?? 'I420'; +} + +// Normalize config (e.g., align dimensions) +function normalizeConfig(config: VideoDecoderConfig): VideoDecoderConfig { + return { + ...config, + codedWidth: config.codedWidth ? alignTo(config.codedWidth, 2) : undefined, + codedHeight: config.codedHeight ? alignTo(config.codedHeight, 2) : undefined, + }; +} +``` + +## Gaps to Address + +Based on preliminary analysis, these gaps may need bridging: + +| Gap | Severity | Mitigation | +|-----|----------|------------| +| State machine semantics | Medium | Implement in shim layer | +| `decodeQueueSize` tracking | Low | Track in JS wrapper | +| `reset()` behavior | Medium | May need node-av enhancement or workaround | +| Error type mapping | Low | Create DOMException wrapper | +| Timestamp/duration handling | Medium | Pass through carefully | +| Pixel format mapping | Low | Create format mapping table | + +## Test Cases + +### Basic Decode Test + +```typescript +import { VideoDecoder } from './VideoDecoder'; +import { EncodedVideoChunk } from './EncodedVideoChunk'; + +describe('VideoDecoder', () => { + it('decodes H.264 frames', async () => { + const frames: VideoFrame[] = []; + + const decoder = new VideoDecoder({ + output: (frame) => frames.push(frame), + error: (e) => { throw e; }, + }); + + decoder.configure({ codec: 'avc1.42E01E' }); + + const chunk = new EncodedVideoChunk({ + type: 'key', + timestamp: 0, + data: h264KeyFrameData, + }); + + decoder.decode(chunk); + await decoder.flush(); + + expect(frames).toHaveLength(1); + expect(frames[0].codedWidth).toBe(1920); + expect(frames[0].codedHeight).toBe(1080); + + decoder.close(); + }); +}); +``` + +## Acceptance Criteria + +1. `VideoDecoder` class matches WebCodecs interface +2. State machine works correctly (`unconfigured` → `configured` → `closed`) +3. `configure()` accepts standard codec strings +4. `decode()` produces `VideoFrame` objects via callback +5. `flush()` waits for all frames to be delivered +6. `reset()` returns to unconfigured state +7. `close()` releases resources +8. `isConfigSupported()` returns accurate results +9. Error callbacks fire with appropriate `DOMException` types +10. Unit tests pass +11. Integration test with real H.264 data passes + +## Deliverables + +- [ ] `src/VideoDecoder.ts` — VideoDecoder implementation +- [ ] `src/VideoFrame.ts` — VideoFrame class (if not already exists) +- [ ] `src/EncodedVideoChunk.ts` — EncodedVideoChunk class (if not already exists) +- [ ] `tests/VideoDecoder.test.ts` — Unit tests +- [ ] `tests/integration/decode-h264.test.ts` — Integration test +- [ ] Documentation of node-av → WebCodecs mapping + +## Related + +- [Evaluate FFmpeg N-API Bindings](./evaluate-ffmpeg-napi-bindings.md) — Prerequisite evaluation +- [FFmpeg N-API Bindings Research](../research/ffmpeg-napi.md) +- [Codec String Parser](./codec-string-parser.md) — Needed for `configure()` +- [N-API PoC Addon](./napi-poc-addon.md) — Original from-scratch approach (alternative path) From 1a1b5b54983316372c4567098fff2d1fb8afaeca Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 1 Dec 2025 06:23:40 +0000 Subject: [PATCH 3/3] Clarify code snippets as conceptual sketches with helper functions Co-authored-by: subtleGradient <4117+subtleGradient@users.noreply.github.com> --- tasks/videodecoder-shim-node-av.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tasks/videodecoder-shim-node-av.md b/tasks/videodecoder-shim-node-av.md index 18f70e1..253f258 100644 --- a/tasks/videodecoder-shim-node-av.md +++ b/tasks/videodecoder-shim-node-av.md @@ -87,6 +87,8 @@ type CodecState = "unconfigured" | "configured" | "closed"; ### VideoDecoder Class +> **Note**: This is a conceptual sketch showing the API mapping. Helper functions like `mapPixelFormat`, `alignTo`, and `checkCodecSupport` would need to be implemented. + ```typescript import { Decoder } from 'node-av'; import { parseCodecString } from './codec-parser'; @@ -228,6 +230,8 @@ export class VideoDecoder { ### Key Mapping Functions +> **Note**: These are helper functions that would need to be implemented. + ```typescript // Map node-av pixel format to WebCodecs VideoPixelFormat function mapPixelFormat(format: string): VideoPixelFormat { @@ -249,6 +253,18 @@ function normalizeConfig(config: VideoDecoderConfig): VideoDecoderConfig { codedHeight: config.codedHeight ? alignTo(config.codedHeight, 2) : undefined, }; } + +// Helper to align value to a multiple +function alignTo(value: number, alignment: number): number { + return Math.ceil(value / alignment) * alignment; +} + +// Check if codec is supported by node-av +async function checkCodecSupport(codecId: string): Promise { + // TODO: Implementation would query node-av's supported codecs + // or attempt to create a decoder instance + return true; +} ``` ## Gaps to Address