Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix RX side dropping frames at the beginning. #917

Merged

Conversation

DawidWesierski4
Copy link
Collaborator

Address an issue where the RX is sending
ARP replies to the TX while still being completely uninitialized, which leads to the session dropping frames. The problem worsens as the TX outperforms
the RX in terms of initialization speed.

Resolve this by halting the FFmpeg plugin
initialization and starting the scheduled tasklets because we don't want the CNI thread to respond to the ARP requests sent in the tv_attach function
until the RX is ready to receive those packets.

@frankdjx
Copy link
Collaborator

Not understand why it need this change. What's the problem if some frames are missing at the initial stage? Please note the typical case in video production is using multicast address, no one will use unicast address. Even with this change, the similar drop for the first frame parsing is still there with multicast mode.

ecosystem/ffmpeg_plugin/mtl_st20p_rx.c Outdated Show resolved Hide resolved
ecosystem/ffmpeg_plugin/mtl_st20p_rx.c Outdated Show resolved Hide resolved
Address an issue where the RX is sending
ARP replies to the TX while still being completely
uninitialized, which leads to the session dropping
frames. The problem worsens as the TX outperforms
the RX in terms of initialization speed.

Resolve this by halting the FFmpeg plugin
initialization and starting the scheduled tasklets
because we don't want the CNI thread to respond to
the ARP requests sent in the tv_attach function
until the RX is ready to receive those packets.
@DawidWesierski4
Copy link
Collaborator Author

Although the scenario is unlikely, isn't less frame drop better?

@frankdjx
Copy link
Collaborator

It doesn't make sense, it has more codes but not fix anything really.

@DawidWesierski4
Copy link
Collaborator Author

DawidWesierski4 commented Jun 28, 2024

It doesn't make sense, it has more codes but not fix anything really.

In my testing if the TX outperforms the TX the RX Informs that the first received packet does not start with the sequence number 0

With this change the RX side will start responding to ARP messages and will start the initialization of the TX session after the RX session is initialized

Although this only occurs with unicast it makes it so that the RX side receives the first packets after this change

I can provide the logs confirming this

To replicate the issue the easiest way is to have debug log enabled and a very high performance / low latency scenario
You will see the first received packet would not be 0 and a few frames dropped at the beginning

The issue is bigger the more TX outperforms the RX (it allows for faster initialization )

@DawidWesierski4
Copy link
Collaborator Author

DawidWesierski4 commented Jun 28, 2024

The fix works basically via moving the mtl_start that enables the ARP response further down the line of RX initialization in ffmpeg as for now the RX side is not catching up to the TX side

here an example where the first recived packet has the p_counter 189

MTL: 2024-06-17 22:18:08, rv_attach(0), w 1920 h 1080 fmt ST20_FMT_YUV_422_10BIT packing 0 pt 112 flags 0x0 frame time 40.000000ms fps 25.000000
MTL: 2024-06-17 22:18:08, st22_rx_create, succ on sch 0 session 0
MTL: 2024-06-17 22:18:08, rv_slot_by_tmstamp(0): new tmstamp 41793993
MTL: 2024-06-17 22:18:08, rv_get_frame(0), find frame at 0
MTL: 2024-06-17 22:18:08, rv_slot_by_tmstamp(0): assign slot 0 framebuff 0x3205a17000 for tmstamp 41793993
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3227 kmode 0 trans_order 1
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3227 p_counter 187 sep_counter 0
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), get seq_id 3227 tmstamp 41793993, p_counter 187 sep_counter 0, payload_length 1280
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3228 kmode 0 trans_order 1
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3228 p_counter 188 sep_counter 0
MTL: 2024-06-17 22:18:08, st22p_rx_create(0), codestream fmt JPEGXS_CODESTREAM, output fmt: YUV422PLANAR10LE, flags 0x8000
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3229 kmode 0 trans_order 1
MTL: 2024-06-17 22:18:08, rv_handle_st22_pkt(0,0), seq_id 3229 p_counter 189 sep_counter 0

I did not experience this after this change

The video received usually had a few frames cropped from the beginning

@DawidWesierski4 DawidWesierski4 marked this pull request as draft July 1, 2024 18:32
@DawidWesierski4 DawidWesierski4 marked this pull request as ready for review July 3, 2024 11:24
@frankdjx
Copy link
Collaborator

frankdjx commented Jul 3, 2024

Please help to fix lint fail.

/github/workspace/ecosystem/ffmpeg_plugin/mtl_st22p_rx.c:318:7: error: code should be clang-formatted [-Wclang-format-violations]
    /* 
      ^
/github/workspace/ecosystem/ffmpeg_plugin/mtl_st22p_rx.c:324:17: error: code should be clang-formatted [-Wclang-format-violations]
      if (frame)

Just run ./foramt_coding.sh

Add retries to the initial read packet in FFmpeg
to prevent imminent failures. This will aid
in the synchronization of unicast FFmpeg streams
and allow for some leniency in the RX session
establishment.
@frankdjx frankdjx merged commit bbf9847 into OpenVisualCloud:main Jul 8, 2024
30 checks passed
DawidWesierski4 added a commit to DawidWesierski4/Media-Transport-Library that referenced this pull request Jul 17, 2024
Address an issue where the RX is sending
ARP replies to the TX while still being completely uninitialized, which
leads to the session dropping frames. The problem worsens as the TX
outperforms
the RX in terms of initialization speed.

Resolve this by halting the FFmpeg plugin
initialization and starting the scheduled tasklets because we don't want
the CNI thread to respond to the ARP requests sent in the tv_attach
function
until the RX is ready to receive those packets.
awilczyns pushed a commit that referenced this pull request Jul 17, 2024
Address an issue where the RX is sending
ARP replies to the TX while still being completely uninitialized, which
leads to the session dropping frames. The problem worsens as the TX
outperforms
the RX in terms of initialization speed.

Resolve this by halting the FFmpeg plugin
initialization and starting the scheduled tasklets because we don't want
the CNI thread to respond to the ARP requests sent in the tv_attach
function
until the RX is ready to receive those packets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants