Support directly convert SRT with WebRTC, vice versa.

[winlinvip]

We may need to make a very important change in our structure. Previously, SRT and GB were converted to RTMP and then to RTC, but they should have the ability to convert directly.

Since SRT, GB, and RTC are essentially RTP packets, direct conversion is more efficient. RTMP, FLV, and HLS are frame-based protocols, and they are similar, so converting to RTMP and then to HLS is not a problem.

That is, packet-based protocols should be able to convert directly, and frame-based protocols can be converted directly. When converting from packet to frame, RTMP is needed. GB and RTC can be converted directly. SRT and RTC can be converted directly. For GB/SRT to HLS, it should be GB/SRT to RTMP to HLS. This way, SRT can be directly converted to RTC, with lower latency and higher efficiency.

In practice, it has been found that low-latency live streaming may be more suitable for SRT, but the player can only choose the SRT protocol, which limits the scope of application. For end-to-end low-latency live streaming, FLV cannot be used for low-latency players, as there will be cumulative latency. The only choice is the RTC player, but currently, SRT to RTMP and then to RTC has a large latency and a long link.

SRT

Each SRT UDP packet contains multiple TS packets. There may be issues with packet loss, out-of-order packets, and retransmissions.

Since TS packets can be directly converted to H.264 annexb packets, SRT UDP packets can be directly converted to RTC RTP packets. Packet loss, out-of-order packets, and retransmissions can be directly mapped.

However, FEC could be a problem.

GB

GB can be H.264 encapsulation or PS encapsulation, with most being PS encapsulation.

If it is H.264 encapsulation, it can directly correspond to RTC RTP packets. If it is PS encapsulation, it cannot be directly converted and requires receiving a complete Frame before converting to RTP.

Note: Currently, GB supports RTMP Client and direct connection to RTMP Source (default).

RTC

RTC RTP packets contain H.264 raw data, which can also be in annexb format. Both types can be parsed.

Sometimes, many small P-frames can be actively packaged into one RTP packet, as referenced:

    # Whether to merge multiple NALUs into one.
    # @see https://github.com/ossrs/srs/issues/307#issuecomment-612806318
    # default: off
    merge_nalus off;

[xiaozhihong]

For non-RTP (SRT-TS), once a frame is collected, it is directly converted.
For RTP (28181), each packet is directly converted.

The reason why many non-RTP protocols cannot directly convert each packet to RTP is that these protocols themselves also have out-of-order, packet loss, etc. They often need to be framed and then converted to RTP. If a complete frame cannot be formed, then converting to RTP is not playable and has little significance. Frame-to-RTP packet conversion itself does not introduce delay.

[zhouxiaojun2008]

The boss said that framing TS and then converting it to RTP packets brings delay, which refers to the frame-level delay brought by framing, right? The delay of the conversion itself is negligible.
Directly converting MPEGTS to RTP packets seems a bit risky, as the TS stream is also a frame split into several TS packets. I remember there is no length field, and it relies on the payload_unit_start_indicator in each TS packet header. When converting a TS packet to RTP, how do you know whether this TS packet needs to be split into FU-A or is a separate small frame that can be directly converted?
We know the starting position of the FU-A header, but we don't know which TS packet is the ending position. We need to read the payload_unit_start_indicator to know that the previous packet is the last packet of a frame.

[magicffourier]

如果是264封装，则可以直接和RTC的RTP包对应。如果是PS封装，则无法直接转换，需要收到完整的Frame后再转RTP。

gb其实不一定一个帧编码，而且还存在data partition的情况

[winlinvip]

If it's SRT to RTMP to WebRTC, it's actually just a Frame delay, and it should be the same as direct conversion.

So where is the delay in this process? It may still need to be looked at in detail, and it may not necessarily be the delay caused by converting to RTMP.

The so-called conversion to RTMP is actually just Message framing, at most a Frame delay, which is 40ms delay when fps=25.

The GB delay is not actually due to RTMP, but because of the PS stream and the irregularity of the camera's own packet sending, which causes the server to buffer more data to correctly parse.

GB to RTMP to WebRTC, the delay test is around 350ms. Reference: #3176 (comment)