diff --git a/broadcasting/03_RTMP_SystemArchitecture.md b/broadcasting/03_RTMP_Architecture.md similarity index 100% rename from broadcasting/03_RTMP_SystemArchitecture.md rename to broadcasting/03_RTMP_Architecture.md diff --git a/broadcasting/04_RTMP_RunningTheDemo.md b/broadcasting/04_RTMP_Running_The_Demo.md similarity index 100% rename from broadcasting/04_RTMP_RunningTheDemo.md rename to broadcasting/04_RTMP_Running_The_Demo.md diff --git a/broadcasting/08_RTSP_Architecture.md b/broadcasting/08_RTSP_Architecture.md index d4fdaa2..165e899 100644 --- a/broadcasting/08_RTSP_Architecture.md +++ b/broadcasting/08_RTSP_Architecture.md @@ -1,17 +1,15 @@ # Architecture Now let's discuss how the architecture of our solution will look like. -It will be a little different from the RTMP to HLS architecture. -The main component will be the pipeline, which will ingest RTP stream and convert it to HLS. Beyond that we will also need a Connection Manager, which will be responsible for establishing an RTSP connection with the server. +It will be a little different from the RTMP to HLS architecture. In most cases communication with a RTSP server is split into two phases: -![image](assets/rtsp_architecture.drawio.png) +- Negotiation of the stream parameters over RTSP. +- Receiving RTP stream(s) that the client and server have agreed upon. -When initializing, the pipeline will start a Connection Manager which starts an RTSP connection with the server. Once the connection is fully established, the pipeline will be notified. +Both of these phases are handled by RTSP Source. Let's take a closer look how each of them folds out: -Let's take a closer look on each of those components: - -## Connection Manager -The role of the connection manager is to initialize RTSP session and start playing the stream. +## Establishing the connection +When establishing a connection the source will act as a connection manager, initializing the RTSP session and starting the stream playback. It communicates with the server using the [RTSP requests](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works/#RTSP_requests). In fact, we won't need many requests to start playing the stream - take a look at the desired message flow: ![image](assets/connection_manager.drawio.png) @@ -20,18 +18,21 @@ First we want to get the details of the video we will be playing, by sending the Then we call the `SETUP` method, defining the transport protocol (RTP) and client port used for receiving the stream. Now we can start the stream using `PLAY` method. -## Pipeline +## Receiving the stream -The pipeline consists of a couple elements, each of them performing a specific media processing task. You can definitely notice some similarities to the pipeline described in the [RTMP architecture](02_RTMP_Introduction.md). However, we will only be processing video so only the video processing elements will be necessary. +The source is a bin containing a few elements, each of them performing a specific media processing task. You can definitely notice some similarities to the pipeline described in the [RTMP architecture](03_RTMP_Architecture.md). However, we will only be processing video so only the video processing elements will be necessary. ![image](assets/rtsp_pipeline.drawio.png) -We have already used the, `H264 Parser`, `MP4 H264 Payloader`, `CMAF Muxer` and `HLS Sink` elements in the RTMP pipeline, take a look at the [RTMP to HLS architecture](03_RTMP_SystemArchitecture.md) chapter for details of the purpose of those elements. +We have already used the `H264 Parser` and `HLS Sink Bin` elements in the RTMP pipeline, take a look at the [RTMP to HLS architecture](03_RTMP_Architecture.md) chapter for details of the purpose of those elements. Let us describe briefly what is the purpose of the other components: ### UDP Source This element is quite simple - it receives UDP packets from the network and sends their payloads to the next element. -### RTP SessionBin -RTP SessionBin is a Membrane's Bin, which is a Membrane's container used for creating reusable groups of elements. In our case the Bin handles the RTP session with the server, which has been set up by the Connection Manager. +### RTP Demuxer +This element is responsible for getting media packets out of the RTP packets they were transported in and routing them according to their [SSRC](https://datatracker.ietf.org/doc/html/rfc3550#section-3). In our case we only receive a single video stream, so only one output will be used. + +### RTP H264 Depayloader +When transporting H264 streams over RTP they need to be split into chunks and have some additional metadata included. This element's role is to unpack the RTP packets it receives from the Demuxer into a pure H264 stream that can be processed further. diff --git a/broadcasting/09_RTSP_RunningDemo.md b/broadcasting/09_RTSP_Running_The_Demo.md similarity index 51% rename from broadcasting/09_RTSP_RunningDemo.md rename to broadcasting/09_RTSP_Running_The_Demo.md index 2b60920..6f5c617 100644 --- a/broadcasting/09_RTSP_RunningDemo.md +++ b/broadcasting/09_RTSP_Running_The_Demo.md @@ -1,53 +1,43 @@ In the tutorial we won't explain how to implement the solution from the ground up - instead, we will run the existing code from [Membrane demos](https://github.com/membraneframework/membrane_demo). To run the RTSP to HLS converter first clone the demos repo: -```console +```bash git clone https://github.com/membraneframework/membrane_demo.git ``` -```console +```bash cd membrane_demo/rtsp_to_hls ``` Install the dependencies -```console +```bash mix deps.get ``` -Make sure you have those libraries installed as well: -- gcc -- libc-dev -- ffmpeg - -On ubuntu: -```console -apt-get install gcc libc-dev ffmpeg -``` - Take a look inside the `lib/application.ex` file. It's responsible for starting the pipeline. We need to give a few arguments to the pipeline: ```elixir -@rtsp_stream_url "rtsp://rtsp.membrane.work:554/testsrc.264" -@output_path "hls_output" -@rtp_port 20000 +rtsp_stream_url = "rtsp://localhost:30001" +output_path = "hls_output" +rtp_port = 20000 ``` -The `@output_path` attribute defines the storage directory for hls files and the `@rtp_port` defines on which port we will be expecting the rtp stream, once the RTSP connection is established. +The `output_path` attribute defines the storage directory for hls files and the `rtp_port` defines on which port we will be expecting the rtp stream, once the RTSP connection is established. -The `@rtsp_stream_url` attribute contains the address of the stream, which we will be converting. It is a sample stream prepared for the purpose of the demo. +The `rtsp_stream_url` attribute contains the address of the stream, which we will be converting. If you want to receive a stream from some accessible RTSP server, you can pass it's URL here. In this demo we'll run our own, simple server: + +```bash +mix run server.exs +``` Now we can start the application: -```console +```bash mix run --no-halt ``` -The pipeline will start playing, after a couple of seconds the HLS files should appear in the `@output_path` directory. In order to play the stream we need to first serve them. We can do it using simple python server. - -```console -python3 -m http.server 8000 -``` +The pipeline will start playing, after a couple of seconds the HLS files should appear in the `@output_path` directory. Then we can play the stream using [ffmpeg](https://ffmpeg.org/), by pointing to the location of the manifest file: -```console +```bash ffplay http://YOUR_MACHINE_IP:8000/rtsp_to_hls/hls_output/index.m3u8 ``` diff --git a/broadcasting/10_ConnectionManager.md b/broadcasting/10_ConnectionManager.md deleted file mode 100644 index bcf7980..0000000 --- a/broadcasting/10_ConnectionManager.md +++ /dev/null @@ -1,71 +0,0 @@ -Now let's focus on how the Connection Manager works. As mentioned previously its role is to establish the RTSP connection with the RTSP server. - -The `ConnectionManager` module will use the [`Connection`](https://hexdocs.pm/connection/Connection.html) behaviour, which provides additional callbacks to [`GenServer`](https://hexdocs.pm/elixir/GenServer.html) behaviour, aiding with building a connection process. - -First of all we are defining the `ConnectionStatus` struct, which we will use to keep the state of the ConnectionManager: - -##### lib/connection_manager.ex -```elixir -defmodule ConnectionStatus do - @moduledoc false - @type t :: %__MODULE__{ - stream_url: binary(), - rtsp_session: pid(), - pipeline: pid(), - keep_alive: pid(), - pipeline_options: keyword() - } - - @enforce_keys [ - :stream_url, - :pipeline, - :pipeline_options - ] - - defstruct @enforce_keys ++ - [ - :rtsp_session, - :keep_alive - ] -end -``` - -It holds the `rtsp_session`, which is the pid of a process started with `Membrane.RTSP.start/1`. The `Membrane.RTSP` allows us to execute RTSP client commands. You can read more about it [here](https://hexdocs.pm/membrane_rtsp/readme.html). -The `pipeline` field is the pid of the pipeline, we will need it to notify the pipeline, that the RTSP connection is ready. In such notification we send `pipeline_options`, which contain necessary information about the stream. -The `keep_alive` is a process which repeatedly pings the RTSP server, in order to keep the connection alive and prevent a timeout. - -Let's take a look at the `connect/2` callback, which is called immediately after the `init/1`: - -##### lib/connection_manager.ex -```elixir -def connect(_info, %ConnectionStatus{} = connection_status) do - rtsp_session = start_rtsp_session(connection_status) - connection_status = %{connection_status | rtsp_session: rtsp_session} - - if is_nil(rtsp_session) do - {:backoff, @delay, connection_status} - else - with {:ok, connection_status} <- get_rtsp_description(connection_status), - :ok <- setup_rtsp_connection(connection_status), - {:ok, connection_status} <- start_keep_alive(connection_status), - :ok <- play(connection_status) do - - send( - connection_status.pipeline, - {:rtsp_setup_complete, connection_status.pipeline_options} - ) - - {:ok, connection_status} - else - {:error, error_message} -> - {:backoff, @delay, connection_status} - end - end -end -``` - -In the callback we go through the whole process of establishing RTSP connection - first starting the RTSP session, then getting the video parameters with, setting up the session, starting the keep alive process and finally playing the stream. -If all those steps succeed we can notify the pipeline, otherwise we back off and try to set up the connection after a `@delay` amount of time. - -What might seem unclear to you is the `get_sps_pps` function. -It is responsible for getting the [sps and pps](https://www.cardinalpeak.com/blog/the-h-264-sequence-parameter-set) parameters from the RTSP DESCRIBE method. In short, sps and pps are parameters used by the H.264 codec and are required to decode the stream. Once the RTSP connection is complete we are sending them to the pipeline. \ No newline at end of file diff --git a/broadcasting/10_RTSP_Pipeline.md b/broadcasting/10_RTSP_Pipeline.md new file mode 100644 index 0000000..02b80f6 --- /dev/null +++ b/broadcasting/10_RTSP_Pipeline.md @@ -0,0 +1,57 @@ +As explained in the [Architecture chapter](08_RTSP_Architecture.md), the pipeline will consist of a RTSP Source and a HLS Sink. + +The initial pipeline will consist of the `RTSP Source`, which will start establishing the connection with the RTSP server, and the `HLS Sink Bin`. For now we won't connect this elements in any way, since we don't have information about what tracks we'll receive from the RTSP server which we're connecting with. + +##### lib/pipeline.ex +```elixir +@impl true +def handle_init(_context, options) do + spec = [ + child(:source, %Membrane.RTSP.Source{ + transport: {:udp, options.port, options.port + 5}, + allowed_media_types: [:video, :audio], + stream_uri: options.stream_url, + on_connection_closed: :send_eos + }), + child(:hls, %Membrane.HTTPAdaptiveStream.SinkBin{ + target_window_duration: Membrane.Time.seconds(120), + manifest_module: Membrane.HTTPAdaptiveStream.HLS, + storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{ + directory: options.output_path + } + }) + ] + + {[spec: spec], %{parent_pid: options.parent_pid}} +end +``` + +Once we receive the `{:set_up_tracks, tracks}` notification from the source we have the information what tracks have been set up during connection establishment and what we should expect. First we filter these tracks, so that we have at most one video and audio track each. Then we can create specs that will connect output pads of the source with input pads of the sink appropriately - audio to audio and video to video. + +##### lib/pipeline.ex +```elixir +@impl true +def handle_child_notification({:set_up_tracks, tracks}, :source, _ctx, state) do + track_specs = + Enum.uniq_by(tracks, & &1.type) + |> Enum.filter(&(&1.type in [:audio, :video])) + |> Enum.map(fn track -> + encoding = + case track do + %{type: :audio} -> :AAC + %{type: :video} -> :H264 + end + + get_child(:source) + |> via_out(Pad.ref(:output, track.control_path)) + |> via_in(:input, + options: [encoding: encoding, segment_duration: Membrane.Time.seconds(4)] + ) + |> get_child(:hls) + end) + + {[spec: track_specs], state} +end +``` + +By doing this we are prepared to receive the streams when a `PLAY` request is eventually sent by the source and the server starts streaming. diff --git a/broadcasting/11_RTSP_Pipeline.md b/broadcasting/11_RTSP_Pipeline.md deleted file mode 100644 index 21c3e71..0000000 --- a/broadcasting/11_RTSP_Pipeline.md +++ /dev/null @@ -1,89 +0,0 @@ -As explained in the [Architecture chapter](08_RTSP_Architecture.md), the pipeline will consist of a couple of elements, that will be processing the RTP stream. - -The flow of the pipeline will consist of three steps. First, when the pipeline is initialized we will start the Connection Manager, which will set up the RTP stream via the RTSP. -Once that is finished, we will set up two initial elements in the pipeline - the `UDP Source` and `RTP SessionBin`, which will allow us to receive RTP packets and process them. -When the SessionBin detects that the RTP stream has been started, it will notify the pipeline with the `:new_rtp_stream` notification. Later on, we will add the remaining elements to the pipeline, allowing for the whole conversion process to take place. - -Those steps take place, respectively, in the: `handle_init/1`, `handle_other/3` and `handle_notification/4` callbacks. While the `handle_init/1` is rather intuitive, we will describe in detail what's happening in the other callbacks. - -Let us explain what's going on in the `handle_other` callback: - -##### lib/pipeline.ex -```elixir -@impl true -def handle_other({:rtsp_setup_complete, options}, _ctx, state) do - children = %{ - app_source: %Membrane.UDP.Source{ - local_port_no: state[:port], - recv_buffer_size: 500_000 - }, - rtp: %Membrane.RTP.SessionBin{ - fmt_mapping: %{96 => {:H264, 90_000}} - }, - hls: %Membrane.HTTPAdaptiveStream.Sink{ - manifest_module: Membrane.HTTPAdaptiveStream.HLS, - target_window_duration: 120 |> Membrane.Time.seconds(), - target_segment_duration: 4 |> Membrane.Time.seconds(), - storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{ - directory: state[:output_path] - } - } - } - - links = [ - link(:app_source) - |> via_in(Pad.ref(:rtp_input, make_ref())) - |> to(:rtp) - ] - - spec = %ParentSpec{children: children, links: links} - { {:ok, spec: spec}, %{state | video: %{sps: options[:sps], pps: options[:pps]}} } -end -``` - -When we receive the `rtsp_setup_complete` message, we first define the new children for the pipeline, and links between them - the UDP Source and the RTP SessionBin. We also create the HLS Sink, however we won't be linking it just yet. With the message we receive the sps and pps inside the options, and we add them to the pipeline's state. - -Only after we receive the `:new_rtp_stream` notification we add the rest of the elements and link them with each other: - -##### lib/pipeline.ex -```elixir -@impl true -def handle_notification({:new_rtp_stream, ssrc, 96, _extensions}, :rtp, _ctx, state) do - actions = - if Map.has_key?(state, :rtp_started) do - [] - else - children = %{ - video_nal_parser: %Membrane.H264.FFmpeg.Parser{ - sps: state.video.sps, - pps: state.video.pps, - skip_until_keyframe?: true, - framerate: {30, 1}, - alignment: :au, - attach_nalus?: true - }, - video_payloader: Membrane.MP4.Payloader.H264, - video_cmaf_muxer: Membrane.MP4.Muxer.CMAF - } - - links = [ - link(:rtp) - |> via_out(Pad.ref(:output, ssrc), - options: [depayloader: Membrane.RTP.H264.Depayloader] - ) - |> to(:video_nal_parser) - |> to(:video_payloader) - |> to(:video_cmaf_muxer) - |> via_in(:input) - |> to(:hls) - ] - - [spec: %ParentSpec{children: children, links: links}] - end - - { {:ok, actions}, Map.put(state, :rtp_started, true) } -end -``` - -First we check, if the stream hasn't started yet. That's because if we are restarting the pipeline there might be a previous RTP stream still being sent, so we might receive the `:new_rtp_stream` notification twice - once for the old and then for the new stream. We want to ignore any notification after the first one, as we want only a single copy of each media processing element. -Notice the sps and pps being passed to the H264 parser - they are necessary for decoding the stream. diff --git a/broadcasting/12_Summary.md b/broadcasting/11_Summary.md similarity index 100% rename from broadcasting/12_Summary.md rename to broadcasting/11_Summary.md diff --git a/broadcasting/13_H264_codec.md b/broadcasting/12_H264_codec.md similarity index 100% rename from broadcasting/13_H264_codec.md rename to broadcasting/12_H264_codec.md diff --git a/broadcasting/assets/rtsp_pipeline.drawio.png b/broadcasting/assets/rtsp_pipeline.drawio.png index 4975b8a..643392c 100644 Binary files a/broadcasting/assets/rtsp_pipeline.drawio.png and b/broadcasting/assets/rtsp_pipeline.drawio.png differ diff --git a/broadcasting/index.md b/broadcasting/index.md index 6eb4283..935aae9 100644 --- a/broadcasting/index.md +++ b/broadcasting/index.md @@ -9,14 +9,13 @@ part: 7 | ------ | -------------------------------- | -------------------------------- | | 1 | General introduction | 01_General_Introduction.md | | 2 | RTMP introduction | 02_RTMP_Introduction.md | -| 3 | RTMP to HLS system architecture | 03_RTMP_SystemArchitecture.md | -| 4 | Running the RTMP to HLS demo | 04_RTMP_RunningTheDemo.md | +| 3 | RTMP to HLS system architecture | 03_RTMP_Architecture.md | +| 4 | Running the RTMP to HLS demo | 04_RTMP_Running_The_Demo.md | | 5 | RTMP to HLS - pipeline | 05_RTMP_Pipeline.md | | 6 | Web player | 06_WebPlayer.md | | 7 | RTSP to HLS introduction | 07_RTSP_Introduction.md | | 8 | RTSP to HLS system architecture | 08_RTSP_Architecture.md | -| 9 | Running the RTSP to HLS demo | 09_RTSP_RunningDemo.md | -| 10 | Connection manager | 10_ConnectionManager.md | -| 11 | RTSP to HLS - pipeline | 11_RTSP_Pipeline.md | -| 12 | Summary | 12_Summary.md | -| 13 | (Suplement) H264 codec | 13_H264_codec.md | +| 9 | Running the RTSP to HLS demo | 09_RTSP_Running_The_Demo.md | +| 10 | RTSP to HLS - pipeline | 10_RTSP_Pipeline.md | +| 11 | Summary | 11_Summary.md | +| 12 | (Suplement) H264 codec | 12_H264_codec.md |