Skip to content

Streaming Video and Audio via RTSP

pschatzmann edited this page Oct 22, 2026 · 3 revisions

This chapter shows how to stream live camera video together with sound from a microphone via RTSP, to VLC, ffplay or any other RTSP client. It is based on the rtspserver-video-ext.ino example and is the RTSP counterpart of Streaming Video and Audio via a Webserver, which uses the same pipeline. For the basics of the RTSP server see the RTSP Server tutorial.

Contents

RTSP or HTTP?

Both examples - webserver-video-ext for HTTP and rtspserver-video-ext for RTSP - carry the same MPEG transport stream with H.264 video and AAC audio:

Feature HTTP RTSP
Players VLC, ffplay, mpv, web browser (with mpegts.js) VLC, ffplay, mpv, IP camera viewers - not in browsers
Clients any number at the same time one at a time
Transport TCP RTP over TCP or UDP
Sketch web server in loop(), shared mutex RTSP server runs in its own tasks: loop() is empty

Use RTSP if your viewer or recorder (e.g. an NVR) expects an RTSP camera; use HTTP if you want to watch in a browser or with several viewers.

One Track for Video and Audio

The RTSP server carries exactly one track per session: there is no multi-track SDP (see RTSP Server - Combined audio + video). So video and audio are first muxed into an MPEG transport stream by MuxerMTS, and this stream is sent as one RTP track (MP2T, RFC 2250, payload type 33 - see RTSPFormatMTS). The player demuxes it into video and audio and keeps them in sync with the timestamps in the stream.

The Pipeline

camera -> CameraFrameSourceTinyCamera -> H264EncoderESP32P4 --\
                                                                > MuxerMTS -> RTSPMediaSourceMTS -> RTSPMediaStreamer -> RTSP client
mic    -> I2SCodecStream              -> AACEncoderVO -------/

The camera, encoder and muxer part is the same as in the webserver example, including VideoMuxerWithTasks, which runs one task for video and one for audio. Only the output is different: the muxer writes to an RTSPMediaSourceMTS, which the RTSP streamer reads from.

RTSPMediaSourceMTS

For a transport stream from a file, RTSPMediaSource is good enough (see the RTSP Server tutorial). For a live stream that is written by the muxer tasks while the RTSP server task reads it, use RTSPMediaSourceMTS (AudioTools/Communication/RTSP/RTSPMediaSourceMTS.h, included by RTSP.h). It is a Print that the muxer writes to, and an IMediaSource for the streamer:

  • Whole TS packets: the data is kept as complete 188 byte TS packets in a fixed size, lock-free single producer/single consumer ring buffer (RingBufferSPSC), and every RTP packet contains whole TS packets (up to 7: 1316 bytes, below the usual MTU), as RFC 2250 requires. The writes must not come from several tasks at the same time - VideoMuxerWithTasks makes sure of this with its mutex.
  • Adaptive send rate: it is a packetized source, so the streamer sends all queued packets at each timer tick (100 times per second by default) instead of a fixed amount: the send rate follows the bitrate of the stream.
  • No old data: while no client is streaming, the data is dropped, so a new client doesn't get old data. It starts decoding at the next keyframe, since MuxerMTS repeats the stream description before each one.
  • Video is dropped before audio: if the network is slower than the stream, the buffer fills up. At 3/4 full, the video is dropped from the start of the next video frame on, until the buffer is half empty again and the next keyframe starts. So the picture freezes for a moment - which a viewer hardly notices - while the sound continues. Audio is only dropped if the buffer is completely full. Audio and video are recognized by the stream id of their PES packets, so this works with any transport stream. droppedVideoPackets() and droppedPackets() report how much was dropped.

The constructor takes the buffer size in TS packets (default 128, about 24 KB - about 5 seconds of 16 kHz AAC audio alone) and the send rate.

The Example

rtspserver-video-ext.ino was tested on a Waveshare ESP32-P4-WIFI6 with its OV5647 MIPI-CSI camera. Compared to the webserver example only the output changes:

// RTSP server: sends the MPEG transport stream as one RTP track
RTSPMediaSourceMTS source;
RTSPMediaStreamer<RTSPPlatformWiFi> streamer(source);
RTSPServer<RTSPPlatformWiFi> rtsp(streamer);  // port 8554

// Muxer
MuxerMTS mts(source);
VideoMuxerWithTasks video(mts);

In setup() the microphone, the camera and the H.264 encoder are set up exactly as in the webserver example, and mts.setRealTime(true) makes the timestamps follow the capture time. Then the server is started and the tasks:

// connects to WiFi (with power save switched off) and starts the server
rtsp.begin(ssid, password);

video.setVideoSource(camera);
video.setVideoEncoder(encoder);
video.setAudioSource(mic);
video.setAudioEncoder(aac);
video.setTaskParameters(16 * 1024, 2);  // stack in bytes on ESP32
video.begin();

The RTSP server handles the client in its own tasks, and nothing else touches the muxer, so - unlike the webserver example - no shared mutex is needed and loop() is just delay(1000).

Build with esp32:esp32:esp32p4:ChipVariant=postv3,PSRAM=enabled,PartitionScheme=huge_app.

Watching the Stream

Open rtsp://<ip-address>:8554/ and request TCP as transport, with a buffer of a few seconds:

vlc --network-caching=5000 rtsp://192.168.1.33:8554/
ffplay -rtsp_transport tcp rtsp://192.168.1.33:8554/

In VLC select "RTP over RTSP (TCP)" in Preferences > Input/Codecs and set "Network caching" there as well. Why both matter:

  • TCP instead of UDP: most players use UDP by default. Over WiFi, packets get lost, and over UDP they are gone: the picture breaks up and the sound has gaps. TCP retransmits them.
  • A buffer of a few seconds: even with a good signal, a WiFi connection sometimes doesn't transmit anything for 1 to 4 seconds. Over TCP no data is lost then - it just arrives late. A player with a small buffer (VLC's default is 1 second, ffplay hardly buffers at all) runs empty and the sound breaks up. A buffer of about 5 seconds bridges this, at the cost of a corresponding delay.

What We Measured

On the Waveshare ESP32-P4-WIFI6 at 320x240 / 8 frames per second with 16 kHz mono AAC:

  • UDP: 1173 of the RTP packets were lost within 90 seconds.
  • TCP: 4 minutes at a steady 8 frames per second without decoding errors (apart from the frames before the first keyframe); the occasional send stalls of up to 3 seconds were bridged by dropping video frames.
  • Audio over TCP (3 minutes, monitored per AAC frame): no audio frame lost, but at 5 moments the audio arrived 2 to 4 seconds late. A ping sent at the same time didn't get through either: these stalls are in the WiFi connection, not in the sketch.
  • Memory: the free heap stayed constant over several sessions, also after clients that disconnected without closing the session.

Troubleshooting

  • The stream breaks up after some time, with send stalls of many seconds: check the firmware of the WiFi co-processor. On the ESP32-P4 the WiFi is provided by an ESP32-C6 via ESP-Hosted, and its firmware must match the version of the ESP32 core: if the log shows Response not received for [0x15e](Req_GetCoprocessorFwVersion) and Slave firmware version: 0.0.0, it is outdated. Update it with the core's ESP_HostedOTA example (File > Examples > ESP_HostedOTA); if downloading over WiFi fails, you can also send the firmware file (the update URL is printed in the log) to the board yourself - over USB or from the SD card - and write it with hostedBeginUpdate(), hostedWriteUpdate(), hostedEndUpdate() and hostedActivateUpdate(). In our tests the long stalls of up to 20 seconds became rare after this update - the short ones of 1 to 4 seconds remained (see below).
  • The picture breaks up and the sound has gaps: the player uses UDP - request TCP (see above).
  • The sound breaks up for a moment from time to time: the player's buffer is too small for the stalls of the WiFi connection - increase it (see above).
  • RTSPMediaSourceMTS: network too slow - dropping video: expected during a WiFi stall; the video continues at the next keyframe. If buffer full ... packets dropped follows, the stall was longer than the buffer: also audio was lost.
  • RTP Stream can't keep up: a send call took longer than the timer period - another sign of a WiFi stall.
  • A second client is not served: the RTSP server serves one client at a time. If a player was closed without ending the session, the server notices the closed connection and accepts the next client.
  • It doesn't play in a browser: browsers can't play RTSP - use the webserver example for a browser.

Clone this wiki locally