+
+ +
+

RTSP APIs

+
+

RTSP Client

+

The RtspClient class provides an interface to an RTSP server. It is used to +send RTSP requests and receive RTSP responses. It also provides an interface +to the RTP and RTCP sessions that are created as a result of the RTSP +interactions.

+

The RtspClient currently only supports MJPEG streams, since the ESP32 does +not have a hardware decoder for H.264 or H.265.

+

Additionally the client currently only supports UDP transport for RTP and RTCP +packets. TCP transport is not supported.

+

The user can register a callback function to be notified when new, complete JPEG +frames are received. The callback function is called with a pointer to the JPEG +frame.

+
+
+

RTSP Server

+

The RtspServer class provides an implementation of an RTSP server. It is used +to receive RTSP requests and send RTSP responses. It is designed to allow the +user to send JPEG frames to the server, which will then send them to the client +over RTP/UDP.

+

The server currently only supports MJPEG streams, since the ESP32 does not have +a hardware encoder for H.264 or H.265.

+

Additionally, the server currently only supports UDP transport for RTP and RTCP +packets. TCP transport is not supported.

+
+
+

API Reference

+
+

Header File

+ +
+
+

Classes

+
+
+class espp::RtspClient
+

A class for interacting with an RTSP server using RTP and RTCP over UDP

+

This class is used to connect to an RTSP server and receive JPEG frames over RTP. It uses the TCP socket to send RTSP requests and receive RTSP responses. It uses the UDP socket to receive RTP and RTCP packets.

+

The RTSP client is designed to be used with the RTSP server in the [camera-streamer]https://github.com/esp-cpp/camera-streamer) project, but it should work with any RTSP server that sends JPEG frames over RTP.

+
+

Example

+

  espp::RtspClient rtsp_client({
+      .server_address = ip_address, // string of the form {}.{}.{}.{}
+        .rtsp_port = CONFIG_RTSP_SERVER_PORT,
+        .path = "/mjpeg/1",
+        .on_jpeg_frame = [](std::unique_ptr<espp::JpegFrame> jpeg_frame) {
+          fmt::print("Got JPEG frame of size {}x{}\n", jpeg_frame->get_width(), jpeg_frame->get_height());
+        },
+        .log_level = espp::Logger::Verbosity::ERROR,
+        });
+
+  std::error_code ec;
+
+  do {
+    // clear the error code
+    ec.clear();
+    rtsp_client.connect(ec);
+    if (ec) {
+      logger.error("Error connecting to server: {}", ec.message());
+      logger.info("Retrying in 1s...");
+      std::this_thread::sleep_for(1s);
+    }
+  } while (ec);
+
+  rtsp_client.describe(ec);
+  if (ec) {
+    logger.error("Error describing server: {}", ec.message());
+  }
+
+  rtsp_client.setup(ec);
+  if (ec) {
+    logger.error("Error setting up server: {}", ec.message());
+  }
+
+  rtsp_client.play(ec);
+  if (ec) {
+    logger.error("Error playing server: {}", ec.message());
+  }
+
+
+

+
+
+

Public Types

+
+
+using jpeg_frame_callback_t = std::function<void(std::unique_ptr<JpegFrame> jpeg_frame)>
+

Function type for the callback to call when a JPEG frame is received.

+
+ +
+
+

Public Functions

+
+
+inline explicit RtspClient(const Config &config)
+

Constructor

+
+
Parameters
+

config – The configuration for the RTSP client

+
+
+
+ +
+
+inline ~RtspClient()
+

Destructor Disconnects from the RTSP server

+
+ +
+
+inline std::string send_request(const std::string &method, const std::string &path, const std::unordered_map<std::string, std::string> &extra_headers, std::error_code &ec)
+

Send an RTSP request to the server

+
+

Note

+

This is a blocking call

+
+
+

Note

+

This will parse the response and set the session ID if it is present in the response. If the response is not a 200 OK, then an error code will be set and the response will be returned. If the response is a 200 OK, then the response will be returned and the error code will be set to success.

+
+
+
Parameters
+
    +

  • method – The method to use for connecting. Options are “OPTIONS”, “DESCRIBE”, “SETUP”, “PLAY”, and “TEARDOWN”

    • +

  • path – The path to the RTSP stream on the server.

    • +

  • extra_headers – Any extra headers to send with the request. These will be added to the request after the CSeq and Session headers. The key is the header name and the value is the header value. For example, {“Accept”: “application/sdp”} will add “Accept: application/sdp” to the request. The “User-Agent” header will be added automatically. The “CSeq” and “Session” headers will be added automatically. The “Accept” header will be added automatically. The “Transport” header will be added automatically for the “SETUP” method. Defaults to an empty map.

    • +

  • ec – The error code to set if an error occurs

    • +
+
+
Returns
+

The response from the server

+
+
+
+ +
+
+inline void connect(std::error_code &ec)
+

Connect to the RTSP server Connects to the RTSP server and sends the OPTIONS request.

+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+inline void disconnect(std::error_code &ec)
+

Disconnect from the RTSP server Disconnects from the RTSP server and sends the TEARDOWN request.

+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+inline void describe(std::error_code &ec)
+

Describe the RTSP stream Sends the DESCRIBE request to the RTSP server and parses the response.

+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+inline void setup(std::error_code &ec)
+

Setup the RTSP stream

+
+

Note

+

Starts the RTP and RTCP threads. Sends the SETUP request to the RTSP server and parses the response.

+
+
+

Note

+

The default ports are 5000 and 5001 for RTP and RTCP respectively.

+
+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+inline void setup(size_t rtp_port, size_t rtcp_port, std::error_code &ec)
+

Setup the RTSP stream Sends the SETUP request to the RTSP server and parses the response.

+
+

Note

+

Starts the RTP and RTCP threads.

+
+
+
Parameters
+
    +

  • rtp_port – The RTP client port

    • +

  • rtcp_port – The RTCP client port

    • +

  • ec – The error code to set if an error occurs

    • +
+
+
+
+ +
+
+inline void play(std::error_code &ec)
+

Play the RTSP stream Sends the PLAY request to the RTSP server and parses the response.

+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+inline void pause(std::error_code &ec)
+

Pause the RTSP stream Sends the PAUSE request to the RTSP server and parses the response.

+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+inline void teardown(std::error_code &ec)
+

Teardown the RTSP stream Sends the TEARDOWN request to the RTSP server and parses the response.

+
+
Parameters
+

ec – The error code to set if an error occurs

+
+
+
+ +
+
+
+struct Config
+

Configuration for the RTSP client.

+
+

Public Members

+
+
+std::string server_address
+

The server IP Address to connect to.

+
+ +
+
+int rtsp_port = {8554}
+

The port of the RTSP server.

+
+ +
+
+std::string path = {"/mjpeg/1"}
+

The path to the RTSP stream on the server. Will be appended to the server address and port to form the full path of the form “rtsp://<server_address>:<rtsp_port><path>”.

+
+ +
+
+jpeg_frame_callback_t on_jpeg_frame
+

The callback to call when a JPEG frame is received.

+
+ +
+
+espp::Logger::Verbosity log_level = espp::Logger::Verbosity::INFO
+

The verbosity of the logger.

+
+ +
+
+ +
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class espp::RtspServer
+

Class for streaming MJPEG data from a camera using RTSP + RTP Starts a TCP socket to listen for RTSP connections, and then spawns off a new RTSP session for each connection.

+

See also

+

RtspSession

+
+

+
+

example

+

  const int server_port = CONFIG_RTSP_SERVER_PORT;
+  const std::string server_uri = fmt::format("rtsp://{}:{}/mjpeg/1", ip_address, server_port);
+
+  logger.info("Starting RTSP Server on port {}", server_port);
+  logger.info("RTSP URI: {}", server_uri);
+
+  espp::RtspServer rtsp_server({
+      .server_address = ip_address,
+      .port = server_port,
+      .path = "/mjpeg/1",
+      .log_level = espp::Logger::Verbosity::INFO,
+    });
+  rtsp_server.start();
+
+  espp::JpegFrame jpeg_frame(jpeg_data, sizeof(jpeg_data));
+
+  logger.info("Parsed JPEG image, num bytes: {}", jpeg_frame.get_data().size());
+  logger.info("Created frame of size {}x{}", jpeg_frame.get_width(), jpeg_frame.get_height());
+  rtsp_server.send_frame(jpeg_frame);
+
+
+

+
+
+

Note

+

This class does not currently send RTCP packets

+
+
+

Public Functions

+
+
+inline explicit RtspServer(const Config &config)
+

Construct an RTSP server.

+
+
Parameters
+

config – The configuration for the RTSP server

+
+
+
+ +
+
+inline ~RtspServer()
+

Destroy the RTSP server.

+
+ +
+
+inline void set_session_log_level(Logger::Verbosity log_level)
+

Sets the log level for the RTSP sessions created by this server.

+
+

Note

+

This does not affect the log level of the RTSP server itself

+
+
+

Note

+

This does not change the log level of any sessions that have already been created

+
+
+
Parameters
+

log_level – The log level to set

+
+
+
+ +
+
+inline bool start()
+

Start the RTSP server Starts the accept task, session task, and binds the RTSP socket.

+
+
Returns
+

True if the server was started successfully, false otherwise

+
+
+
+ +
+
+inline void stop()
+

Stop the FTP server Stops the accept task, session task, and closes the RTSP socket.

+
+ +
+
+inline void send_frame(const JpegFrame &frame)
+

Send a frame over the RTSP connection Converts the full JPEG frame into a series of simplified RTP/JPEG packets and stores it to be sent over the RTP socket, but does not actually send it.

+
+

Note

+

Overwrites any existing frame that has not been sent

+
+
+
Parameters
+

frame – The frame to send

+
+
+
+ +
+
+
+struct Config
+

Configuration for the RTSP server.

+
+

Public Members

+
+
+std::string server_address
+

The ip address of the server.

+
+ +
+
+int port
+

The port to listen on.

+
+ +
+
+std::string path
+

The path to the RTSP stream.

+
+ +
+
+size_t max_data_size = 1000
+

The maximum size of RTP packet data for the MJPEG stream. Frames will be broken up into multiple packets if they are larger than this. It seems that 1500 works well for sending, but is too large for the esp32 (camera-display) to receive properly.

+
+ +
+
+Logger::Verbosity log_level = Logger::Verbosity::WARN
+

The log level for the RTSP server.

+
+ +
+
+ +
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class espp::RtspSession
+

Class that reepresents an RTSP session, which is uniquely identified by a session id and sends frame data over RTP and RTCP to the client

+
+

Public Functions

+
+
+inline explicit RtspSession(std::unique_ptr<TcpSocket> control_socket, const Config &config)
+

Construct a new RtspSession object.

+
+
Parameters
+
    +

  • control_socket – The control socket of the session

    • +

  • config – The configuration of the session

    • +
+
+
+
+ +
+
+inline uint32_t get_session_id() const
+

Get the session id.

+
+
Returns
+

The session id

+
+
+
+ +
+
+inline bool is_closed() const
+

Check if the session is closed.

+
+
Returns
+

True if the session is closed, false otherwise

+
+
+
+ +
+
+inline bool is_connected() const
+

Get whether the session is connected

+
+
Returns
+

True if the session is connected, false otherwise

+
+
+
+ +
+
+inline bool is_active() const
+

Get whether the session is active

+
+
Returns
+

True if the session is active, false otherwise

+
+
+
+ +
+
+inline void play()
+

Mark the session as active This will cause the server to start sending frames to the client

+
+ +
+
+inline void pause()
+

Pause the session This will cause the server to stop sending frames to the client

+
+

Note

+

This does not stop the session, it just pauses it

+
+
+

Note

+

This is useful for when the client is buffering

+
+
+ +
+
+inline void teardown()
+

Teardown the session This will cause the server to stop sending frames to the client and close the connection

+
+ +
+
+inline bool send_rtp_packet(const RtpPacket &packet)
+

Send an RTP packet to the client

+
+
Parameters
+

packet – The RTP packet to send

+
+
Returns
+

True if the packet was sent successfully, false otherwise

+
+
+
+ +
+
+inline bool send_rtcp_packet(const RtcpPacket &packet)
+

Send an RTCP packet to the client

+
+
Parameters
+

packet – The RTCP packet to send

+
+
Returns
+

True if the packet was sent successfully, false otherwise

+
+
+
+ +
+
+
+struct Config
+

Configuration for the RTSP session.

+
+

Public Members

+
+
+std::string server_address
+

The address of the server.

+
+ +
+
+std::string rtsp_path
+

The RTSP path of the session.

+
+ +
+
+Logger::Verbosity log_level = Logger::Verbosity::WARN
+

The log level of the session.

+
+ +
+
+ +
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class espp::RtpPacket
+

RtpPacket is a class to parse RTP packet.

+

Subclassed by espp::RtpJpegPacket

+
+

Public Functions

+
+
+inline RtpPacket()
+

Construct an empty RtpPacket. The packet_ vector is empty and the header fields are set to 0.

+
+ +
+
+inline explicit RtpPacket(size_t payload_size)
+

Construct an RtpPacket with a payload of size payload_size.

+
+ +
+
+inline explicit RtpPacket(std::string_view data)
+

Construct an RtpPacket from a string_view. Store the string_view in the packet_ vector and parses the header.

+
+
Parameters
+

data – The string_view to parse.

+
+
+
+ +
+
+inline int get_version() const
+

Getters for the RTP header fields.

+
+ +
+
+inline void set_version(int version)
+

Setters for the RTP header fields.

+
+ +
+
+inline void serialize()
+

Serialize the RTP header.

+
+

Note

+

This method should be called after modifying the RTP header fields.

+
+
+

Note

+

This method does not serialize the payload. To set the payload, use set_payload(). To get the payload, use get_payload().

+
+
+ +
+
+inline std::string_view get_data() const
+

Get a string_view of the whole packet.

+
+

Note

+

The string_view is valid as long as the packet_ vector is not modified.

+
+
+

Note

+

If you manually build the packet_ vector, you should make sure that you call serialize() before calling this method.

+
+
+
Returns
+

A string_view of the whole packet.

+
+
+
+ +
+
+inline size_t get_rtp_header_size() const
+

Get the size of the RTP header.

+
+
Returns
+

The size of the RTP header.

+
+
+
+ +
+
+inline std::string_view get_rpt_header() const
+

Get a string_view of the RTP header.

+
+
Returns
+

A string_view of the RTP header.

+
+
+
+ +
+
+inline std::vector<char> &get_packet()
+

Get a reference to the packet_ vector.

+
+
Returns
+

A reference to the packet_ vector.

+
+
+
+ +
+
+inline std::string_view get_payload() const
+

Get a string_view of the payload.

+
+
Returns
+

A string_view of the payload.

+
+
+
+ +
+
+inline void set_payload(std::string_view payload)
+

Set the payload.

+
+
Parameters
+

payload – The payload to set.

+
+
+
+ +
+
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class espp::RtpJpegPacket : public espp::RtpPacket
+

RTP packet for JPEG video. The RTP payload for JPEG is defined in RFC 2435.

+
+

Public Functions

+
+
+inline explicit RtpJpegPacket(std::string_view data)
+

Construct an RTP packet from a buffer.

+
+
Parameters
+

data – The buffer containing the RTP packet.

+
+
+
+ +
+
+inline explicit RtpJpegPacket(const int type_specific, const int frag_type, const int q, const int width, const int height, std::string_view q0, std::string_view q1, std::string_view scan_data)
+

Construct an RTP packet from fields

+

This will construct a packet with quantization tables, so it can only be used for the first packet in a frame.

+
+
Parameters
+
    +

  • type_specific – The type-specific field.

    • +

  • frag_type – The fragment type field.

    • +

  • q – The q field.

    • +

  • width – The width field.

    • +

  • height – The height field.

    • +

  • q0 – The first quantization table.

    • +

  • q1 – The second quantization table.

    • +

  • scan_data – The scan data.

    • +
+
+
+
+ +
+
+inline explicit RtpJpegPacket(const int type_specific, const int offset, const int frag_type, const int q, const int width, const int height, std::string_view scan_data)
+

Construct an RTP packet from fields

+

This will construct a packet without quantization tables, so it cannot be used for the first packet in a frame.

+
+
Parameters
+
    +

  • type_specific – The type-specific field.

    • +

  • offset – The offset field.

    • +

  • frag_type – The fragment type field.

    • +

  • q – The q field.

    • +

  • width – The width field.

    • +

  • height – The height field.

    • +

  • scan_data – The scan data.

    • +
+
+
+
+ +
+
+inline int get_type_specific() const
+

Get the type-specific field.

+
+
Returns
+

The type-specific field.

+
+
+
+ +
+
+inline int get_offset() const
+

Get the offset field.

+
+
Returns
+

The offset field.

+
+
+
+ +
+
+inline int get_q() const
+

Get the fragment type field.

+
+
Returns
+

The fragment type field.

+
+
+
+ +
+
+inline int get_width() const
+

Get the fragment type field.

+
+
Returns
+

The fragment type field.

+
+
+
+ +
+
+inline int get_height() const
+

Get the fragment type field.

+
+
Returns
+

The fragment type field.

+
+
+
+ +
+
+inline std::string_view get_mjpeg_header()
+

Get the mjepg header.

+
+
Returns
+

The mjepg header.

+
+
+
+ +
+
+inline bool has_q_tables() const
+

Get whether the packet contains quantization tables.

+
+

Note

+

The quantization tables are optional. If they are present, the number of quantization tables is always 2.

+
+
+

Note

+

This check is based on the value of the q field. If the q field is 128-256, the packet contains quantization tables.

+
+
+
Returns
+

Whether the packet contains quantization tables.

+
+
+
+ +
+
+inline int get_num_q_tables() const
+

Get the number of quantization tables.

+
+

Note

+

The quantization tables are optional. If they are present, the number of quantization tables is always 2.

+
+
+

Note

+

Only the first packet in a frame contains quantization tables.

+
+
+
Returns
+

The number of quantization tables.

+
+
+
+ +
+
+inline std::string_view get_q_table(int index) const
+

Get the quantization table at the specified index.

+
+
Parameters
+

index – The index of the quantization table.

+
+
Returns
+

The quantization table at the specified index.

+
+
+
+ +
+
+inline std::string_view get_jpeg_data() const
+

Get the JPEG data. The jpeg data is the payload minus the mjpeg header and quantization tables.

+
+
Returns
+

The JPEG data.

+
+
+
+ +
+
+inline int get_version() const
+

Getters for the RTP header fields.

+
+ +
+
+inline void set_version(int version)
+

Setters for the RTP header fields.

+
+ +
+
+inline void serialize()
+

Serialize the RTP header.

+
+

Note

+

This method should be called after modifying the RTP header fields.

+
+
+

Note

+

This method does not serialize the payload. To set the payload, use set_payload(). To get the payload, use get_payload().

+
+
+ +
+
+inline std::string_view get_data() const
+

Get a string_view of the whole packet.

+
+

Note

+

The string_view is valid as long as the packet_ vector is not modified.

+
+
+

Note

+

If you manually build the packet_ vector, you should make sure that you call serialize() before calling this method.

+
+
+
Returns
+

A string_view of the whole packet.

+
+
+
+ +
+
+inline size_t get_rtp_header_size() const
+

Get the size of the RTP header.

+
+
Returns
+

The size of the RTP header.

+
+
+
+ +
+
+inline std::string_view get_rpt_header() const
+

Get a string_view of the RTP header.

+
+
Returns
+

A string_view of the RTP header.

+
+
+
+ +
+
+inline std::vector<char> &get_packet()
+

Get a reference to the packet_ vector.

+
+
Returns
+

A reference to the packet_ vector.

+
+
+
+ +
+
+inline std::string_view get_payload() const
+

Get a string_view of the payload.

+
+
Returns
+

A string_view of the payload.

+
+
+
+ +
+
+inline void set_payload(std::string_view payload)
+

Set the payload.

+
+
Parameters
+

payload – The payload to set.

+
+
+
+ +
+
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class RtcpPacket
+

A class to represent a RTCP packet.

+

This class is used to represent a RTCP packet. It is used as a base class for all RTCP packet types.

+
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class espp::JpegHeader
+

A class to generate a JPEG header for a given image size and quantization tables. The header is generated once and then cached for future use. The header is generated according to the JPEG standard and is compatible with the ESP32 camera driver.

+
+

Public Functions

+
+
+inline explicit JpegHeader(int width, int height, std::string_view q0_table, std::string_view q1_table)
+

Create a JPEG header for a given image size and quantization tables.

+
+
Parameters
+
    +

  • width – The image width in pixels.

    • +

  • height – The image height in pixels.

    • +

  • q0_table – The quantization table for the Y channel.

    • +

  • q1_table – The quantization table for the Cb and Cr channels.

    • +
+
+
+
+ +
+
+inline explicit JpegHeader(std::string_view data)
+

Create a JPEG header from a given JPEG header data.

+
+ +
+
+inline int get_width() const
+

Get the image width.

+
+
Returns
+

The image width in pixels.

+
+
+
+ +
+
+inline int get_height() const
+

Get the image height.

+
+
Returns
+

The image height in pixels.

+
+
+
+ +
+
+inline std::string_view get_data() const
+

Get the JPEG header data.

+
+
Returns
+

The JPEG header data.

+
+
+
+ +
+
+inline std::string_view get_quantization_table(int index) const
+

Get the Quantization table at the index.

+
+
Parameters
+

index – The index of the quantization table.

+
+
Returns
+

The quantization table.

+
+
+
+ +
+
+ +
+
+

Header File

+ +
+
+

Classes

+
+
+class espp::JpegFrame
+

A class that represents a complete JPEG frame.

+

This class is used to collect the JPEG scans that are received in RTP packets and to serialize them into a complete JPEG frame.

+
+

Public Functions

+
+
+inline explicit JpegFrame(const RtpJpegPacket &packet)
+

Construct a JpegFrame from a RtpJpegPacket.

+

This constructor will parse the header of the packet and add the JPEG data to the frame.

+
+
Parameters
+

packet – The packet to parse.

+
+
+
+ +
+
+inline explicit JpegFrame(const char *data, size_t size)
+

Construct a JpegFrame from buffer of jpeg data

+
+
Parameters
+
    +

  • data – The buffer containing the jpeg data.

    • +

  • size – The size of the buffer.

    • +
+
+
+
+ +
+
+inline const JpegHeader &get_header() const
+

Get a reference to the header.

+
+
Returns
+

A reference to the header.

+
+
+
+ +
+
+inline int get_width() const
+

Get the width of the frame.

+
+
Returns
+

The width of the frame.

+
+
+
+ +
+
+inline int get_height() const
+

Get the height of the frame.

+
+
Returns
+

The height of the frame.

+
+
+
+ +
+
+inline bool is_complete() const
+

Check if the frame is complete.

+
+
Returns
+

True if the frame is complete, false otherwise.

+
+
+
+ +
+
+inline void append(const RtpJpegPacket &packet)
+

Append a RtpJpegPacket to the frame. This will add the JPEG data to the frame.

+
+
Parameters
+

packet – The packet containing the scan to append.

+
+
+
+ +
+
+inline void add_scan(const RtpJpegPacket &packet)
+

Append a JPEG scan to the frame. This will add the JPEG data to the frame.

+
+

Note

+

If the packet contains the EOI marker, the frame will be finalized, and no further scans can be added.

+
+
+
Parameters
+

packet – The packet containing the scan to append.

+
+
+
+ +
+
+inline std::string_view get_data() const
+

Get the serialized data. This will return the serialized data.

+
+
Returns
+

The serialized data.

+
+
+
+ +
+
+inline std::string_view get_scan_data() const
+

Get the scan data. This will return the scan data.

+
+
Returns
+

The scan data.

+
+
+
+ +
+
+ +
+
+
+ + +
+