Skip to content

Video Playback

Jump to bottom
Phil Schatzmann edited this page Aug 28, 2026 · 2 revisions

The function to extract audio and video from a container is called demuxing.The audio needs to be fed into an audio decoder and the video into a video decoder.

Demuxers

Here are the supported Demuxers:

Audio Decoders

Details can be found in the generic Audio Codec documentation. I recommend to use a MultiDecoder so that you can support multiple formats.

Video Decoders

The Video Decoders provided by this library have a common API. In order to use the class below, you need to install the corresponsing codec library.

Format Class Library
MJPEG MJPEGDecoder TinyJPEG
MPEG-2 MPGDecoder TinyMPGDecoder
H.264 H264Decoder TinyH264
H.264 H264DecoderESP32S3 codec-h264-ESP32S3
- MultiVideoDecoder add one ore multiple decoders
- MultiVideoDecoderFull MJPEG, MPEG-3 and H.264 support

The Video Player

In order to simplfy the playback of videos we can use the VideoPlayer class. It is the video counterpart of AudioPlayer — it wraps the whole demux → decode → sync → output pipeline behind one object driven by a single copy() call per loop().

Pipeline

Stream (source) → copy() → MultiVideoDemuxer
                              ├─ video → PacedVideoOutput (buffer+pace) → MultiVideoDecoder → VideoOutput
                              └─ audio → [AudioTimeSourceStream, optional] → MultiDecoder → audio output

Key building blocks (all members, all start empty)

  • default_demuxer — a MultiVideoDemuxer, populated via addDemuxer().
  • default_video_decoder — a MultiVideoDecoder, populated via addVideoDecoder().
  • default_audio_decoder — a MultiDecoder, populated via addAudioDecoder().
  • video_sync — a PacedVideoOutput sitting between the demuxer's video output and the actual decoder, buffering/scheduling frames.
  • audio_clock / audio_out — optional AudioTimeSourceStream + EncodedAudioStream, wired only if you call setAudioOutput().

Nothing is pre-registered — this class pulls in zero container/codec libraries by itself. You register exactly what your content needs:

DemuxerAVI aviDemuxer;
VideoPlayer player(aviDemuxer, tftOutput, audioOut);
player.addVideoDecoder(h264Decoder);
player.addVideoDecoder(mjpegDecoder);
player.addAudioDecoder(mp3Decoder, "audio/mpeg");
player.begin(file);

Then in loop(): if (player.copy() == 0) { file.close(); ... }.

Lifecycle

  • Constructors: video-only, or video+audio (with AudioOutput&, Print&, or AudioStream& — picks the most specific overload so audio-info notifications reach the real output).
  • begin(Stream&): wires everything up — video decoder's output/output-source, video_sync as the demuxer's video sink, optionally the audio chain (audio clock if setUseAudioClock(true)) — then calls default_demuxer.begin().
  • copy(): reads one buffer's worth (default 1024 bytes, setBufferSize()) from the source, keeps video_sync's fps synced with the demuxer's parsed rate, writes it into the demuxer (retrying on partial accepts). Returns bytes copied, 0 when source is exhausted — same convention as CodecCopy.
  • copyAll(): blocking loop calling copy() until it returns 0.
  • end() tears the pipeline down (stops video_sync's background render task, decoders, demuxer); setActive(false) just pauses copy()/copyAll() without tearing down.

Audio clock caveat

setUseAudioClock() defaults to false — video is paced against wall-clock time. Only opt in if there's a real, continuously-delivering audio track; wiring the clock against silent/absent audio stalls video forever (this bit decode-mp4.ino historically, per the class comment).

VideoPlayerFull (VideoPlayerFull.h)

A subclass that pre-registers every video codec (H264/MJPEG/MPEG-1) and the common audio ones (MP3/AAC/MP2) in its constructor — the "just point it at a file" convenience, at the cost of pulling in all those codec libraries unconditionally. Still requires you to pass a demuxer (or MultiVideoDemuxerFull for "any container format" too). WAV is deliberately excluded.

One gap worth noting: it doesn't cover the seek-backed/spooled MP4 strategies (decode-mp4-file.ino/decode-mp4-spooled.ino) — those drive their DemuxerMP4 directly instead of through copy().

Clone this wiki locally