Repository navigation
Streaming Video and Audio via RTSP
This chapter shows how to stream live camera video together with sound from a microphone via RTSP, to VLC, ffplay or any other RTSP client. It is based on the rtspserver-video-ext.ino example and is the RTSP counterpart of Streaming Video and Audio via a Webserver, which uses the same pipeline. For the basics of the RTSP server see the RTSP Server tutorial.
Contents
- RTSP or HTTP?
- One Track for Video and Audio
- The Pipeline
- RTSPMediaSourceMTS
- The Example
- Watching the Stream
- What We Measured
- Troubleshooting
Both examples - webserver-video-ext for HTTP and rtspserver-video-ext for RTSP - carry the same MPEG transport stream with H.264 video and AAC audio:
| Feature | HTTP | RTSP |
|---|---|---|
| Players | VLC, ffplay, mpv, web browser (with mpegts.js) | VLC, ffplay, mpv, IP camera viewers - not in browsers |
| Clients | any number at the same time | one at a time |
| Transport | TCP | RTP over TCP or UDP |
| Sketch | web server in loop(), shared mutex |
RTSP server runs in its own tasks: loop() is empty |
Use RTSP if your viewer or recorder (e.g. an NVR) expects an RTSP camera; use HTTP if you want to watch in a browser or with several viewers.
The RTSP server carries exactly one track per session: there is no multi-track SDP (see RTSP Server - Combined audio + video). So video and audio are first muxed into an MPEG transport stream by MuxerMTS, and this stream is sent as one RTP track (MP2T, RFC 2250, payload type 33 - see RTSPFormatMTS). The player demuxes it into video and audio and keeps them in sync with the timestamps in the stream.
camera -> CameraFrameSourceTinyCamera -> H264EncoderESP32P4 --\
> MuxerMTS -> RTSPMediaSourceMTS -> RTSPMediaStreamer -> RTSP client
mic -> I2SCodecStream -> AACEncoderVO -------/
The camera, encoder and muxer part is the same as in the webserver example, including VideoMuxerWithTasks, which runs one task for video and one for audio. Only the output is different: the muxer writes to an RTSPMediaSourceMTS, which the RTSP streamer reads from.
For a transport stream from a file, RTSPMediaSource is good enough (see the RTSP Server tutorial). For a live stream that is written by the muxer tasks while the RTSP server task reads it, use RTSPMediaSourceMTS (AudioTools/Communication/RTSP/RTSPMediaSourceMTS.h, included by RTSP.h). It is a Print that the muxer writes to, and an IMediaSource for the streamer:
-
Whole TS packets: the data is kept as complete 188 byte TS packets in a fixed size, lock-free single producer/single consumer ring buffer (
RingBufferSPSC), and every RTP packet contains whole TS packets (up to 7: 1316 bytes, below the usual MTU), as RFC 2250 requires. The writes must not come from several tasks at the same time - VideoMuxerWithTasks makes sure of this with its mutex. - Adaptive send rate: it is a packetized source, so the streamer sends all queued packets at each timer tick (100 times per second by default) instead of a fixed amount: the send rate follows the bitrate of the stream.
- No old data: while no client is streaming, the data is dropped, so a new client doesn't get old data. It starts decoding at the next keyframe, since MuxerMTS repeats the stream description before each one.
-
Video is dropped before audio: if the network is slower than the stream, the buffer fills up. At 3/4 full, the video is dropped from the start of the next video frame on, until the buffer is half empty again and the next keyframe starts. So the picture freezes for a moment - which a viewer hardly notices - while the sound continues. Audio is only dropped if the buffer is completely full. Audio and video are recognized by the stream id of their PES packets, so this works with any transport stream.
droppedVideoPackets()anddroppedPackets()report how much was dropped.
The constructor takes the buffer size in TS packets (default 128, about 24 KB - about 5 seconds of 16 kHz AAC audio alone) and the send rate.
rtspserver-video-ext.ino was tested on a Waveshare ESP32-P4-WIFI6 with its OV5647 MIPI-CSI camera. Compared to the webserver example only the output changes:
// RTSP server: sends the MPEG transport stream as one RTP track
RTSPMediaSourceMTS source;
RTSPMediaStreamer<RTSPPlatformWiFi> streamer(source);
RTSPServer<RTSPPlatformWiFi> rtsp(streamer); // port 8554
// Muxer
MuxerMTS mts(source);
VideoMuxerWithTasks video(mts);In setup() the microphone, the camera and the H.264 encoder are set up exactly as in the webserver example, and mts.setRealTime(true) makes the timestamps follow the capture time. Then the server is started and the tasks:
// connects to WiFi (with power save switched off) and starts the server
rtsp.begin(ssid, password);
video.setVideoSource(camera);
video.setVideoEncoder(encoder);
video.setAudioSource(mic);
video.setAudioEncoder(aac);
video.setTaskParameters(16 * 1024, 2); // stack in bytes on ESP32
video.begin();The RTSP server handles the client in its own tasks, and nothing else touches the muxer, so - unlike the webserver example - no shared mutex is needed and loop() is just delay(1000).
Build with esp32:esp32:esp32p4:ChipVariant=postv3,PSRAM=enabled,PartitionScheme=huge_app.
Open rtsp://<ip-address>:8554/ and request TCP as transport, with a buffer of a few seconds:
vlc --network-caching=5000 rtsp://192.168.1.33:8554/
ffplay -rtsp_transport tcp rtsp://192.168.1.33:8554/
In VLC select "RTP over RTSP (TCP)" in Preferences > Input/Codecs and set "Network caching" there as well. Why both matter:
- TCP instead of UDP: most players use UDP by default. Over WiFi, packets get lost, and over UDP they are gone: the picture breaks up and the sound has gaps. TCP retransmits them.
- A buffer of a few seconds: even with a good signal, a WiFi connection sometimes doesn't transmit anything for 1 to 4 seconds. Over TCP no data is lost then - it just arrives late. A player with a small buffer (VLC's default is 1 second, ffplay hardly buffers at all) runs empty and the sound breaks up. A buffer of about 5 seconds bridges this, at the cost of a corresponding delay.
On the Waveshare ESP32-P4-WIFI6 at 320x240 / 8 frames per second with 16 kHz mono AAC:
- UDP: 1173 of the RTP packets were lost within 90 seconds.
- TCP: 4 minutes at a steady 8 frames per second without decoding errors (apart from the frames before the first keyframe); the occasional send stalls of up to 3 seconds were bridged by dropping video frames.
- Audio over TCP (3 minutes, monitored per AAC frame): no audio frame lost, but at 5 moments the audio arrived 2 to 4 seconds late. A ping sent at the same time didn't get through either: these stalls are in the WiFi connection, not in the sketch.
- Memory: the free heap stayed constant over several sessions, also after clients that disconnected without closing the session.
-
The stream breaks up after some time, with send stalls of many seconds: check the firmware of the WiFi co-processor. On the ESP32-P4 the WiFi is provided by an ESP32-C6 via ESP-Hosted, and its firmware must match the version of the ESP32 core: if the log shows
Response not received for [0x15e](Req_GetCoprocessorFwVersion)andSlave firmware version: 0.0.0, it is outdated. Update it with the core'sESP_HostedOTAexample (File > Examples > ESP_HostedOTA); if downloading over WiFi fails, you can also send the firmware file (the update URL is printed in the log) to the board yourself - over USB or from the SD card - and write it withhostedBeginUpdate(),hostedWriteUpdate(),hostedEndUpdate()andhostedActivateUpdate(). In our tests the long stalls of up to 20 seconds became rare after this update - the short ones of 1 to 4 seconds remained (see below). - The picture breaks up and the sound has gaps: the player uses UDP - request TCP (see above).
- The sound breaks up for a moment from time to time: the player's buffer is too small for the stalls of the WiFi connection - increase it (see above).
-
RTSPMediaSourceMTS: network too slow - dropping video: expected during a WiFi stall; the video continues at the next keyframe. Ifbuffer full ... packets droppedfollows, the stall was longer than the buffer: also audio was lost. -
RTP Stream can't keep up: a send call took longer than the timer period - another sign of a WiFi stall. - A second client is not served: the RTSP server serves one client at a time. If a player was closed without ending the session, the server notices the closed connection and accepts the next client.
- It doesn't play in a browser: browsers can't play RTSP - use the webserver example for a browser.