Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix regression (crash) in transport-cc feedback generation #1339

Merged
merged 6 commits into from Feb 20, 2024

Conversation

ibc
Copy link
Member

@ibc ibc commented Feb 20, 2024

Details

  • Regression introduced in PR Make transport-cc feedback work similarly to libwebrtc #1088, which has many issues.
  • Basically there are cases in which this->transportCcFeedbackPacket is reset so it's a complete new packet and hence its baseSet is false.
  • However in TransportCongestionControlServer::FillAndSendTransportCcFeedback() in which we just call this->transportCcFeedbackPacket.SetBase() once at the top, but then we enter a loop in which the packet maybe full so it's sent (or other things) and hence we reset the packet while in the loop. And then at the top of the loop this->transportCcFeedbackPacket.AddPacket() is called so the assertion fails because its SetBase() was not called on that fresh packet.
  • So this PR calls SetBase() conditionally in every iteration of the loop by checking if current feedback has the base set or not.
  • Also add a missing break in a case block in that loop.
  • Also set proper packet count in all cases in which we reset this->transportCcFeedbackPacket.
  • Fix crash scenario in which feedback-cc packet is full but cannot be sent: b8a411b

- Fixes #1336

### Details

- Regression introduced in PR #1088, which has many issues.
- Basically there are cases in which `this->transportCcFeedbackPacket` is reset so it's a complete new packet and hence its `baseSet` is `false`.
- However in `TransportCongestionControlServer::FillAndSendTransportCcFeedback()` in which we just call `this->transportCcFeedbackPacket.SetBase()` once at the top, but then we enter a loop in which the packet maybe full so it's sent (or other things) and hence we **reset** the packet while in the loop. And then at the top of the loop `this->transportCcFeedbackPacket.AddPacket()` is called so the assertion fails because its `SetBase()` was not called on that fresh packet.
- Also add a missing `break` in a `case` block in that loop.
- Also set proper packet count in all cases in which we reset `this->transportCcFeedbackPacket`.

### TODO

This is not done yet. The issue is mainly identified but I don't know yet how to properly fix it without doing other wrong things. Basically I'm not sure if the whole logic is ok after PR #1088:

- As said before we only set the base of the packet once at the beginning of the loop, and such a `SetBase()` is called with `this->transportCcFeedbackWideSeqNumStart` and a computed `timestamp`.
- However, within the loop below, the packet may become full and hence `SendTransportCcFeedback()` is called, which updates `++this->transportCcFeedbackPacketCount` and `this->transportCcFeedbackWideSeqNumStart`, and **also** resets the packet (so missing base again).
- So in order to fix the crash, we must call `SetBase()` again, but which which values? Because the values given initially at the top of `TransportCongestionControlServer::FillAndSendTransportCcFeedback()` were the value of `this->transportCcFeedbackWideSeqNumStart` at that time and a timestamp. But that `this->transportCcFeedbackWideSeqNumStart` has now changed, so then what?
- Basically every time `this->transportCcFeedbackPacket.reset()` is called (and there are various cases in the whole `TransportCongestionControlServer.cpp` file) **we must** call ``SetBase()` on the fresh packet **with** proper values. And which are those proper values if we are within that loop?
@ibc ibc requested a review from jmillan February 20, 2024 13:21
@ibc
Copy link
Member Author

ibc commented Feb 20, 2024

This is a draft PR because I need help. See TODO above.

@ibc
Copy link
Member Author

ibc commented Feb 20, 2024

I think I know how to solve it properly. Just a question:

Does somebody remember what is feedbackPacketCount in FeedbackRtpTransportPacket? It's obviously part of the packet fields but I don't remember its meaning. Specifically if an endpoint sends 2 sequential CC feedback packets, what should be the value of that field in those packets?

And why are we setting it when initiating transport server control?

	/* Instance methods. */

	TransportCongestionControlServer::TransportCongestionControlServer(
	  RTC::TransportCongestionControlServer::Listener* listener,
	  RTC::BweType bweType,
	  size_t maxRtcpPacketLen)
	  : listener(listener), bweType(bweType), maxRtcpPacketLen(maxRtcpPacketLen)
	{
		MS_TRACE();

		switch (this->bweType)
		{
			case RTC::BweType::TRANSPORT_CC:
			{
				// Create a feedback packet.
				this->transportCcFeedbackPacket.reset(new RTC::RTCP::FeedbackRtpTransportPacket(0u, 0u));

				// Set initial packet count.
				this->transportCcFeedbackPacket->SetFeedbackPacketCount(this->transportCcFeedbackPacketCount);

@ibc
Copy link
Member Author

ibc commented Feb 20, 2024

Ah, is the number of CC feedback packet sent, right? so it's just a counter that must be incremented every time a feedback packet is sent. Right?

@ibc ibc marked this pull request as ready for review February 20, 2024 13:54
@ibc
Copy link
Member Author

ibc commented Feb 20, 2024

Concerns resolved in theory. PR ready for review.

Copy link
Member

@jmillan jmillan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a couple cosmetic suggestions

worker/src/RTC/TransportCongestionControlServer.cpp Outdated Show resolved Hide resolved
@ibc ibc changed the title Fix regression (crash) in Transport CC handling Fix regression (crash) in transport-cc feedback generation Feb 20, 2024
@ibc
Copy link
Member Author

ibc commented Feb 20, 2024

I've done this change to test:

uint16_t FeedbackRtpTransportPacket::maxMissingPackets{ (1 << 13) - 1 };
// uint16_t FeedbackRtpTransportPacket::maxPacketStatusCount{ (1 << 16) - 1 };

and it crashes:

(ABORT) RTC::RTCP::FeedbackRtpTransport::AddPacket() | failed assertion `!IsFull()': packet is full

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	    0x7ff80d79b1e2 __pthread_kill + 10
1   libsystem_pthread.dylib       	    0x7ff80d7d2ee6 pthread_kill + 263
2   libsystem_c.dylib             	    0x7ff80d6f9b45 abort + 123
3   mediasoup-worker              	       0x1095e6ea0 RTC::RTCP::FeedbackRtpTransportPacket::AddPacket(unsigned short, unsigned long long, unsigned long) (.cold.2) + 64
4   mediasoup-worker              	       0x1092b4c47 RTC::RTCP::FeedbackRtpTransportPacket::AddPacket(unsigned short, unsigned long long, unsigned long) + 231
5   mediasoup-worker              	       0x109274e22 RTC::TransportCongestionControlServer::FillAndSendTransportCcFeedback() + 210

Why? Because when this->transportCcFeedbackPacket->IsFull() we call SendTransportCcFeedback() which is supposed to send the feedback and reset this->transportCcFeedbackPacket so in the next iteration of the loop there is a fresh feedback packet which obviously is not full, however there are cases in which SendTransportCcFeedback() does NOT send the feedback and does NOT reset it:

if (!this->transportCcFeedbackPacket->IsSerializable())
{
	return;
}

So here another regression in that PR or perhaps

@ibc
Copy link
Member Author

ibc commented Feb 20, 2024

Fixing it as follows:
CleanShot-2024-02-20-at-17 37 57@2x

@ibc ibc merged commit a1cc550 into v3 Feb 20, 2024
23 checks passed
@ibc ibc deleted the fix-transport-cc-regression-issue-1336 branch February 20, 2024 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

(ABORT) RTC::RTCP::FeedbackRtpTransport::AddPacket() | failed assertion `baseSet': base not set
2 participants