-
-
Notifications
You must be signed in to change notification settings - Fork 372
Video
This library supports the recording and playback of video which contains both video frames and audio. Different container formats are used to store video on audio together in the same file.
The simplest and therefore most recommended container format is AVI (Audio Video Interleave). It is simple to both encode and decode and you can easily mix audio and video together. It supports both different audio and video encodings.
Sometimes you can also find the MPG (MPEG digital video and audio). As it names states the video is encoded as MPEG-1 (ISO/IEC 11172 part 2) video and the audio as MPEG-1 Layer II ("MP2") audio but mp3 audio usually works also just fine.
On the desktop you can also find MP4 (formally MPEG-4 Part 14). If you are careful and know what you are doing you can also use it on a Microcontroller. The core issue is that you need to read all the metadata first in order to be able to decode the video and audio and this requires a lot of RAM, or you use some direct access using Files or store the metadata temporarily in some files.
Here is an overview of the most frequent audio formats by container format with the most frequently used variant in bold:
| Format | AVI | MPG | MP4 | Description |
|---|---|---|---|---|
| PCM | X | X | uncompressed, high quality, large files | |
| MP3 | X | (X) | very common, lossy compression | |
| AAC | X | X | X | efficient lossy compression |
| ADPCM | X | older compressed audio format | ||
| MP2 | X | MPEG-1 Layer II audio | ||
| Opus | (X) | Codec developed Internet Engineering Task Force (IETF) |
PCM files are uncompressed and not only use a lot of space, but also require a lot of bandwidth, so they should be avoided on microcontrollers: However if you prefer to use it anyhow, I suggest to go for 8 bit mono only.
Here is the corresponding overview for the supported video encoding formats what we can use on Microcontrollers:
| Video Codec | AVI | MPG | MP4 | Description |
|---|---|---|---|---|
| MJPEG | X | X | Motion JEPG (Most efficient format on Microcontrollers | |
| MPEG-2 | X | DVD/broadcast video | ||
| H.264 | (X) | (X) | X | Very common modern video codec |
The function to extract audio and video from a container is called demuxing.The audio needs to be fed into an audio decoder and the video into a video decoder.
Here are the supported Demuxers:
- DemuxerAVI
- DemuxerMPG
- DemuxerMP4
- MultiVideoDemuxer add Demuxers above
- MultiVideoDemuxerFull AVI, MPG and MP4 support
Details can be found in the generic Audio Codec documentation. I recommend to use a MultiDecoder so that you can support multiple formats.
The Video Decoders provided by this library have a common API. In order to use the class below, you need to install the corresponsing codec library.
| Format | Class | Library |
|---|---|---|
| MJPEG | MJPEGDecoder | TinyJPEG |
| MPEG-2 | MPGDecoder | TinyMPGDecoder |
| H.264 | H264Decoder | TinyH264 |
| H.264 | H264DecoderESP32S3 | codec-h264-ESP32S3 |
- | MultiVideoDecoder | add one ore multiple decoders
- | MultiVideoDecoderFull | MJPEG, MPEG-3 and H.264 support