Refax: extract receiver buffer insertion handling to a separate function #2500

ethouris · 2022-10-25T15:12:59Z

Extracted the part handling the insertion of the newly coming in packet into the buffer in CUDT::processData to a separate function.

This fragment will be, in perspective, added an alternative handling for the implementation of the group-common receiver buffer.

…ster

…avis report)

srtcore/core.h

srtcore/core.cpp

srtcore/core.h

maxsharabayko · 2022-11-03T08:57:09Z

srtcore/core.h

+    /// @return 0 The call was successful (regardless if the packet was accepted or not).
+    /// @return -1 The call has failed: no space left in the buffer.
+    /// @return -2 The incoming packet exceeds the expected sequence by more than a length of the buffer (irrepairable discrepancy).
+    int handleSocketPacketReception(const std::vector<CUnit*>& incoming, bool& w_new_inserted, bool& w_was_sent_in_order, CUDT::loss_seqs_t& w_srt_loss_seqs);


The name sounds confusing. Especially "socket" and "reception" parts seem unrelated or redundand.
processDataPackets? 🤔

In the group development branch, there's a second function named handleGroupPacketReception.

srtcore/core.cpp

gou4shi1 · 2022-11-08T11:16:27Z

srtcore/core.cpp

+        // This is executed only when bonding is enabled and only
+        // with the new buffer (in which case the buffer is in the group).
+
+        const int avail_bufsize = (int) getAvailRcvBufferSizeNoLock();


This function doesn't match the codes removed from processData() below, several PRs were lost.

What exactly? I can see only changes in logs and slight improvement for the offset counting. Some parts are left up to the calling function (emergency closing is handled through returning -2). You can cut off since this line:

UniqueLock recvbuf_acklock(m_RcvBufferLock);

and except the last two branches (which remain at the original location and are unchanged) you can see no significant changes in the handleSocketPacketReception.

e.g. #2465 #2467

I can see only changes in logs and slight improvement for the offset counting.

Your refactor is great, but I don't think we can silently revert PRs.

Hey, one moment. @maxsharabayko please join.

Since when is m_iRcvLastAck the base sequence number for the receiver buffer?

This was so in the old UDT, and it was sharing the number that was latest received as ACK as well as the sequence number of the first cell in the receiver buffer. This has changed since the first versions of SRT when the TLPKTDROP was introduced and so a separate field m_iRcvLastSkipAck, which has become the true beginning of the receiver buffer. Here is an excerpt from the CUDT::tsbpd function taken from 1.4.4 version, which executes the drop functionality in order to reach the after-drop packets:

/* Packet ready to play according to time stamp but... */ int seqlen = CSeqNo::seqoff(self->m_iRcvLastSkipAck, skiptoseqno); if (skiptoseqno != SRT_SEQNO_NONE && seqlen > 0) { /* * skiptoseqno != SRT_SEQNO_NONE, * packet ready to play but preceeded by missing packets (hole). */ self->updateForgotten(seqlen, self->m_iRcvLastSkipAck, skiptoseqno); self->m_pRcvBuffer->skipData(seqlen); self->m_iRcvLastSkipAck = skiptoseqno;

Here you can see that the call to skipData is getting only the number of cells to drop as the old receiver buffer had no idea about the sequence numbers at all. The code is almost the same in version 1.2.0, the very first line added to Github. The base sequence number to calculate seqlen is taken from m_iRcvLastSkipAck and this is the real base sequence number for the receiver buffer.

Actually from my production logs, I think m_iRcvLastSkipAck behaved better than m_iRcvLastAck as the base seq of recv buffer.

So the conclusion is using m_iRcvLastSkipAck as the recv buffer base seq? Then I would suggest to change getAvailRcvBufferSizeNoLock() to return m_pRcvBuffer->getAvailSize(m_iRcvLastSkipAck); to keep offset consistent with avail_bufsize.
This doesn't change the result of ACKD_BUFFERLEFT, because it happens after ackDataUpTo(), as #2467 (comment) said .

And m_pRcvBuffer->strFullnessState(qrlog.Warn.CheckEnabled(), m_iRcvLastAck, steady_clock::now()) should be fixed to m_iRcvLastSkipAck also

The difference is that at the moment when empty cells are being removed from the buffer - per TLPKTDROP - then m_iRcvLastSkipAck is updated, but m_iRcvLastAck is not. The latter contains the ACK number that was last received, and when it happens, the cells are also being removed from the receiver buffer up to this number, and both fields are then updated. Hence, m_iRcvLastSkipAck contains the sequence number of the cell 0 of the receiver buffer, while m_iRcvLastAck is either equal to m_iRcvLastSkipAck, or points to a sequence in the past (already cut off from the buffer).

In any case when there's a need to get some value to operate with the buffer, m_iRcvLastAck shall never be used.

gou4shi1 · 2022-11-08T11:25:04Z

srtcore/core.cpp

@@ -9665,20 +9665,10 @@ bool srt::CUDT::overrideSndSeqNo(int32_t seq)
    return true;
 }

-int srt::CUDT::processData(CUnit* in_unit)
+int srt::CUDT::checkLazySpawnLatencyThread()


Suggested change

int srt::CUDT::checkLazySpawnLatencyThread()

int srt::CUDT::checkLazySpawnTsbpdThread()

gou4shi1 · 2022-11-08T11:28:27Z

srtcore/core.cpp

+        const int pktrexmitflag = m_bPeerRexmitFlag ? (rpkt.getRexmitFlag() ? 1 : 0) : 2;
+        const bool retransmitted = pktrexmitflag == 1;
+
+        int buffer_add_result;


buffer_add_result defined without initial value, why not define in line 9790, i.e.
int buffer_add_result = m_pRcvBuffer->insert(u);

gou4shi1 · 2022-11-08T11:35:10Z

srtcore/core.cpp

+            continue;
+        }
+
+        // This is executed only when bonding is enabled and only


What does This stands for?

The below call. Although this comment is false and comes from a version with changes that are not brought here.

gou4shi1 · 2022-11-08T11:42:29Z

srtcore/core.cpp

+        buffer_add_result = m_pRcvBuffer->insert(u);
+        if (buffer_add_result < 0)
+        {
+            // addData returns -1 if at the m_iLastAckPos+offset position there already is a packet.


Suggested change

// addData returns -1 if at the m_iLastAckPos+offset position there already is a packet.

// insert() returns -1 if at the m_iLastAckPos+offset position there already is a packet.

gou4shi1 · 2022-11-08T11:43:31Z

srtcore/core.cpp

+            // addData returns -1 if at the m_iLastAckPos+offset position there already is a packet.
+            // So this packet is "redundant".
+            IF_HEAVY_LOGGING(exc_type = "UNACKED");
+            adding_successful = false;


Also need to update w_new_inserted?

Suggested change

adding_successful = false;

adding_successful = false;

w_new_inserted = false;

This is initialized to false before passing. Although you are partially right - the style for reference passing forbids relying on the initial values set before passing (this is only allowed in case of in-out protocol), so it should be at least initialized in the beginning of the function.

it should be at least initialized in the beginning of the function.

Agreed.

gou4shi1 · 2022-11-08T11:46:11Z

srtcore/core.cpp

@@ -9853,208 +10077,54 @@ int srt::CUDT::processData(CUnit* in_unit)
    }
 #endif

+    // NULL time by default
+    time_point next_tsbpd_avail;


next_tsbpd_avail is not used

Is this used in your following PRs?

Yes. See #2527

OK, it's a big change, I would find some time to study it.

maxsharabayko · 2022-11-10T16:00:08Z

Wow, so many review comments! Thank you, @gou4shi1 for the follow up review!
@ethouris Could you please address those comments in a dedicated PR?

ethouris · 2022-11-10T16:01:00Z

Sure, will go.

Mikołaj Małecki added 4 commits September 22, 2022 17:25

Exported fragment to a separate function

496e6a7

Applied changes from PR 2467-before merging

03214ff

Adjusted to the latest changes in the group branch, per changes in ma…

96e92dd

…ster

Updated and post-fixed

bdc6fff

ethouris added Priority: High Type: Maintenance Work required to maintain or clean up the code [core] Area: Changes in SRT library core labels Oct 25, 2022

ethouris requested a review from maxsharabayko October 25, 2022 15:12

Mikołaj Małecki added 4 commits October 26, 2022 16:26

Updated for latest master

a68d5cf

Merge branch 'master' into dev-processdata-refax

d2d1c90

Added lacking fix for AEAD

89e927f

Fixed: block static variable that might be unused when no logging (tr…

a56c84d

…avis report)

ethouris force-pushed the dev-processdata-refax branch from 2246e1b to a56c84d Compare October 27, 2022 14:24

maxsharabayko added this to the v1.6.0 milestone Nov 1, 2022

maxsharabayko reviewed Nov 1, 2022

View reviewed changes

srtcore/core.h Show resolved Hide resolved

srtcore/core.h Outdated Show resolved Hide resolved

srtcore/core.cpp Outdated Show resolved Hide resolved

Fixed after PR comments

0168c5e

This comment was marked as off-topic.

Sign in to view

maxsharabayko reviewed Nov 3, 2022

View reviewed changes

Mikołaj Małecki added 3 commits November 3, 2022 12:08

Renamed getPacketPTS. Moved call to obtain PTS under belated condition

870509f

Merge branch 'master' into dev-processdata-refax

f9baf83

Fixed the description for getPktTsbPdTime

bcd1fd5

maxsharabayko reviewed Nov 8, 2022

View reviewed changes

srtcore/core.cpp Outdated Show resolved Hide resolved

srtcore/core.cpp Outdated Show resolved Hide resolved

Smaller post-review fixes

3f840aa

maxsharabayko merged commit 491e6e8 into Haivision:master Nov 8, 2022

gou4shi1 reviewed Nov 8, 2022

View reviewed changes

ethouris mentioned this pull request Nov 10, 2022

Refax/postfix: further fixes after last refax changes #2528

Merged

gou4shi1 mentioned this pull request Nov 11, 2022

[core] Fixed usages of m_iRcvLastAck. #2530

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refax: extract receiver buffer insertion handling to a separate function #2500

Refax: extract receiver buffer insertion handling to a separate function #2500

ethouris commented Oct 25, 2022

This comment was marked as off-topic.

maxsharabayko Nov 3, 2022

ethouris Nov 4, 2022

gou4shi1 Nov 8, 2022 •

edited

ethouris Nov 8, 2022

gou4shi1 Nov 8, 2022

gou4shi1 Nov 8, 2022 •

edited

ethouris Nov 8, 2022

gou4shi1 Nov 8, 2022

gou4shi1 Nov 10, 2022 •

edited

gou4shi1 Nov 10, 2022 •

edited

ethouris Nov 10, 2022

gou4shi1 Nov 8, 2022

gou4shi1 Nov 8, 2022

gou4shi1 Nov 8, 2022

ethouris Nov 8, 2022

gou4shi1 Nov 8, 2022

gou4shi1 Nov 8, 2022

ethouris Nov 8, 2022

gou4shi1 Nov 10, 2022 •

edited

gou4shi1 Nov 8, 2022

gou4shi1 Nov 10, 2022 •

edited

ethouris Nov 10, 2022

gou4shi1 Nov 10, 2022

maxsharabayko commented Nov 10, 2022

ethouris commented Nov 10, 2022

	int srt::CUDT::checkLazySpawnLatencyThread()
	int srt::CUDT::checkLazySpawnTsbpdThread()

	// addData returns -1 if at the m_iLastAckPos+offset position there already is a packet.
	// insert() returns -1 if at the m_iLastAckPos+offset position there already is a packet.

	adding_successful = false;
	adding_successful = false;
	w_new_inserted = false;

Refax: extract receiver buffer insertion handling to a separate function #2500

Refax: extract receiver buffer insertion handling to a separate function #2500

Conversation

ethouris commented Oct 25, 2022

This comment was marked as off-topic.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gou4shi1 Nov 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gou4shi1 Nov 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gou4shi1 Nov 10, 2022 • edited

Choose a reason for hiding this comment

gou4shi1 Nov 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gou4shi1 Nov 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gou4shi1 Nov 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxsharabayko commented Nov 10, 2022

ethouris commented Nov 10, 2022

gou4shi1 Nov 8, 2022 •

edited

gou4shi1 Nov 8, 2022 •

edited

gou4shi1 Nov 10, 2022 •

edited

gou4shi1 Nov 10, 2022 •

edited

gou4shi1 Nov 10, 2022 •

edited

gou4shi1 Nov 10, 2022 •

edited