Skip to content

Commit

Permalink
Sample TWCC implementation (#1957)
Browse files Browse the repository at this point in the history 
* encoder bitrate change based on twcc

* Change to 5% inc and dec

* modify percentages

* ema based calc

* EMa fix

* Nits

* Readme

* flip

* memset remove

* Readme update, move enable flags to createSampleConfiguration

* Add codecov token
  • Loading branch information
@disa6302 @niyatim23
disa6302 authored and niyatim23 committed May 2, 2024
1 parent d9dfcb9 commit ee9925e
Show file tree
Hide file tree
Showing 5 changed files with 243 additions and 74 deletions.
1 change: 1 addition & 0 deletions .github/workflows/codecov.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ jobs:
runs-on: ubuntu-20.04
env:
AWS_KVS_LOG_LEVEL: 2
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
permissions:
id-token: write
contents: read
Expand Down
53 changes: 53 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,33 @@ createLwsIotCredentialProvider(
freeIotCredentialProvider(&pSampleConfiguration->pCredentialProvider);
```
## TWCC support
Transport Wide Congestion Control (TWCC) is a mechanism in WebRTC designed to enhance the performance and reliability of real-time communication over the internet. TWCC addresses the challenges of network congestion by providing detailed feedback on the transport of packets across the network, enabling adaptive bitrate control and optimization of media streams in real-time. This feedback mechanism is crucial for maintaining high-quality audio and video communication, as it allows senders to adjust their transmission strategies based on comprehensive information about packet losses, delays, and jitter experienced across the entire transport path.
The importance of TWCC in WebRTC lies in its ability to ensure efficient use of available network bandwidth while minimizing the negative impacts of network congestion. By monitoring the delivery of packets across the network, TWCC helps identify bottlenecks and adjust the media transmission rates accordingly. This dynamic approach to congestion control is essential for preventing degradation in call quality, such as pixelation, stuttering, or drops in audio and video streams, especially in environments with fluctuating network conditions.
To learn more about TWCC, check [TWCC spec](https://datatracker.ietf.org/doc/html/draft-holmer-rmcat-transport-wide-cc-extensions-01)
### Enabling TWCC support
TWCC is enabled by default in the SDK samples (via `pSampleConfiguration->enableTwcc`) flag. In order to disable it, set this flag to `FALSE`.
```c
pSampleConfiguration->enableTwcc = FALSE;
```

If not using the samples directly, 2 things need to be done to set up Twcc:
1. Set the `disableSenderSideBandwidthEstimation` to `FALSE`:
```c
configuration.kvsRtcConfiguration.disableSenderSideBandwidthEstimation = FALSE;
```
2. Set the callback that will have the business logic to modify the bitrate based on packet loss information. The callback can be set using `peerConnectionOnSenderBandwidthEstimation()`:
```c
CHK_STATUS(peerConnectionOnSenderBandwidthEstimation(pSampleStreamingSession->pPeerConnection, (UINT64) pSampleStreamingSession,
sampleSenderBandwidthEstimationHandler));
```
## Use Pre-generated Certificates
The certificate generating function ([createCertificateAndKey](https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-c/Dtls__openssl_8c.html#a451c48525b0c0a8919a880d6834c1f7f)) in createDtlsSession() can take between 5 - 15 seconds in low performance embedded devices, it is called for every peer connection creation when KVS WebRTC receives an offer. To avoid this extra start-up latency, certificate can be pre-generated and passed in when offer comes.
Expand Down Expand Up @@ -476,6 +503,32 @@ To disable threadpool, run `cmake .. -DENABLE_KVS_THREADPOOL=OFF`

Starting version 1.10.0, threadpool usage provides latency improvements in connection establishment. Note, that increasing the number of minimum threads can increase stack memory usage. So, ensure to increase with caution.

### Set up TWCC
TWCC is a mechanism in WebRTC designed to enhance the performance and reliability of real-time communication over the Internet. TWCC addresses the challenges of network congestion by providing detailed feedback on the transport of packets across the network, enabling adaptive bitrate control and optimization of
media streams in real-time. This feedback mechanism is crucial for maintaining high-quality audio and video communication, as it allows senders to adjust their transmission strategies based on comprehensive information about packet losses, delays, and jitter experienced across the entire transport path.
The importance of TWCC in WebRTC lies in its ability to ensure efficient use of available network bandwidth while minimizing the negative impacts of network congestion. By monitoring the delivery of packets across the network, TWCC helps identify bottlenecks and adjust the media transmission rates accordingly.
This dynamic approach to congestion control is essential for preventing degradation in call quality, such as pixelation, stuttering, or drops in audio and video streams, especially in environments with fluctuating network conditions. To learn more about TWCC, you can refer to the [RFC draft](https://datatracker.ietf.org/doc/html/draft-holmer-rmcat-transport-wide-cc-extensions-01)

In order to enable TWCC usage in the SDK, 2 things need to be set up:

1. Set the `disableSenderSideBandwidthEstimation` to FALSE. In our samples, the value is set using `enableTwcc` flag in `pSampleConfiguration`

```c
pSampleConfiguration->enableTwcc = TRUE; // to enable TWCC
pSampleConfiguration->enableTwcc = FALSE; // to disable TWCC
configuration.kvsRtcConfiguration.disableSenderSideBandwidthEstimation = !pSampleConfiguration->enableTwcc;
```

2. Set the callback that will have the business logic to modify the bitrate based on packet loss information. The callback can be set using `peerConnectionOnSenderBandwidthEstimation()`.

```c
CHK_STATUS(peerConnectionOnSenderBandwidthEstimation(pSampleStreamingSession->pPeerConnection, (UINT64) pSampleStreamingSession,
sampleSenderBandwidthEstimationHandler));
```
By default, our SDK enables TWCC listener. The SDK has a sample implementation to integrate TWCC into the Gstreamer pipeline via the `sampleSenderBandwidthEstimationHandler` callback. To get more details, look for this specific callback.
### Setting ICE related timeouts
There are some default timeout values set for different steps in ICE in the [KvsRtcConfiguration](https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-c/structKvsRtcConfiguration.html). These are configurable in the application. While the defaults are generous, there could be applications that might need more flexibility to improve chances of connection establishment because of poor network.
Expand Down
95 changes: 76 additions & 19 deletions samples/Common.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -400,9 +400,14 @@ STATUS initializePeerConnection(PSampleConfiguration pSampleConfiguration, PRtcP
// Set this to custom callback to enable filtering of interfaces
configuration.kvsRtcConfiguration.iceSetInterfaceFilterFunc = NULL;

// disable TWCC
configuration.kvsRtcConfiguration.disableSenderSideBandwidthEstimation = !(pSampleConfiguration->enableTwcc);
DLOGI("TWCC is : %s", configuration.kvsRtcConfiguration.disableSenderSideBandwidthEstimation ? "Disabled" : "Enabled");

// Set the ICE mode explicitly
configuration.iceTransportPolicy = ICE_TRANSPORT_POLICY_ALL;

configuration.kvsRtcConfiguration.enableIceStats = pSampleConfiguration->enableIceStats;
// Set the STUN server
PCHAR pKinesisVideoStunUrlPostFix = KINESIS_VIDEO_STUN_URL_POSTFIX;
// If region is in CN, add CN region uri postfix
Expand Down Expand Up @@ -536,8 +541,12 @@ STATUS createSampleStreamingSession(PSampleConfiguration pSampleConfiguration, P

ATOMIC_STORE_BOOL(&pSampleStreamingSession->terminateFlag, FALSE);
ATOMIC_STORE_BOOL(&pSampleStreamingSession->candidateGatheringDone, FALSE);

pSampleStreamingSession->peerConnectionMetrics.peerConnectionStats.peerConnectionStartTime = GETTIME() / HUNDREDS_OF_NANOS_IN_A_MILLISECOND;

if (pSampleConfiguration->enableTwcc) {
pSampleStreamingSession->twccMetadata.updateLock = MUTEX_CREATE(TRUE);
}

CHK_STATUS(initializePeerConnection(pSampleConfiguration, &pSampleStreamingSession->pPeerConnection));
CHK_STATUS(peerConnectionOnIceCandidate(pSampleStreamingSession->pPeerConnection, (UINT64) pSampleStreamingSession, onIceCandidateHandler));
CHK_STATUS(
Expand Down Expand Up @@ -577,8 +586,10 @@ STATUS createSampleStreamingSession(PSampleConfiguration pSampleConfiguration, P
CHK_STATUS(transceiverOnBandwidthEstimation(pSampleStreamingSession->pAudioRtcRtpTransceiver, (UINT64) pSampleStreamingSession,
sampleBandwidthEstimationHandler));
// twcc bandwidth estimation
CHK_STATUS(peerConnectionOnSenderBandwidthEstimation(pSampleStreamingSession->pPeerConnection, (UINT64) pSampleStreamingSession,
sampleSenderBandwidthEstimationHandler));
if (pSampleConfiguration->enableTwcc) {
CHK_STATUS(peerConnectionOnSenderBandwidthEstimation(pSampleStreamingSession->pPeerConnection, (UINT64) pSampleStreamingSession,
sampleSenderBandwidthEstimationHandler));
}
pSampleStreamingSession->startUpLatency = 0;
CleanUp:

Expand Down Expand Up @@ -629,6 +640,12 @@ STATUS freeSampleStreamingSession(PSampleStreamingSession* ppSampleStreamingSess
}
MUTEX_UNLOCK(pSampleConfiguration->sampleConfigurationObjLock);

if (pSampleConfiguration->enableTwcc) {
if (IS_VALID_MUTEX_VALUE(pSampleStreamingSession->twccMetadata.updateLock)) {
MUTEX_FREE(pSampleStreamingSession->twccMetadata.updateLock);
}
}

CHK_LOG_ERR(closePeerConnection(pSampleStreamingSession->pPeerConnection));
CHK_LOG_ERR(freePeerConnection(&pSampleStreamingSession->pPeerConnection));
SAFE_MEMFREE(pSampleStreamingSession);
Expand Down Expand Up @@ -679,27 +696,61 @@ VOID sampleBandwidthEstimationHandler(UINT64 customData, DOUBLE maximumBitrate)
DLOGV("received bitrate suggestion: %f", maximumBitrate);
}

// Sample callback for TWCC. Average packet is calculated with exponential moving average (EMA). If average packet lost is <= 5%,
// the current bitrate is increased by 5%. If more than 5%, the current bitrate
// is reduced by percent lost. Bitrate update is allowed every second and is increased/decreased upto the limits
VOID sampleSenderBandwidthEstimationHandler(UINT64 customData, UINT32 txBytes, UINT32 rxBytes, UINT32 txPacketsCnt, UINT32 rxPacketsCnt,
UINT64 duration)
{
UNUSED_PARAM(customData);
UNUSED_PARAM(duration);
UNUSED_PARAM(rxBytes);
UNUSED_PARAM(txBytes);
UINT64 videoBitrate, audioBitrate;
UINT64 currentTimeMs, timeDiff;
UINT32 lostPacketsCnt = txPacketsCnt - rxPacketsCnt;
UINT32 percentLost = lostPacketsCnt * 100 / txPacketsCnt;
UINT32 bitrate = 1024;
if (percentLost < 2) {
// increase encoder bitrate by 2 percent
bitrate *= 1.02f;
} else if (percentLost > 5) {
// decrease encoder bitrate by packet loss percent
bitrate *= (1.0f - percentLost / 100.0f);
}
// otherwise keep bitrate the same

DLOGS("received sender bitrate estimation: suggested bitrate %u sent: %u bytes %u packets received: %u bytes %u packets in %lu msec, ", bitrate,
txBytes, txPacketsCnt, rxBytes, rxPacketsCnt, duration / 10000ULL);
DOUBLE percentLost = (DOUBLE) ((txPacketsCnt > 0) ? (lostPacketsCnt * 100 / txPacketsCnt) : 0.0);
SampleStreamingSession* pSampleStreamingSession = (SampleStreamingSession*) customData;

if (pSampleStreamingSession == NULL) {
DLOGW("Invalid streaming session (NULL object)");
return;
}

// Calculate packet loss
pSampleStreamingSession->twccMetadata.averagePacketLoss =
EMA_ACCUMULATOR_GET_NEXT(pSampleStreamingSession->twccMetadata.averagePacketLoss, ((DOUBLE) percentLost));

currentTimeMs = GETTIME();
timeDiff = currentTimeMs - pSampleStreamingSession->twccMetadata.lastAdjustmentTimeMs;
if (timeDiff < TWCC_BITRATE_ADJUSTMENT_INTERVAL_MS) {
// Too soon for another adjustment
return;
}

MUTEX_LOCK(pSampleStreamingSession->twccMetadata.updateLock);
videoBitrate = pSampleStreamingSession->twccMetadata.currentVideoBitrate;
audioBitrate = pSampleStreamingSession->twccMetadata.currentAudioBitrate;

if (pSampleStreamingSession->twccMetadata.averagePacketLoss <= 5) {
// increase encoder bitrate by 5 percent with a cap at MAX_BITRATE
videoBitrate = (UINT64) MIN(videoBitrate * 1.05, MAX_VIDEO_BITRATE_KBPS);
// increase encoder bitrate by 5 percent with a cap at MAX_BITRATE
audioBitrate = (UINT64) MIN(audioBitrate * 1.05, MAX_AUDIO_BITRATE_BPS);
} else {
// decrease encoder bitrate by average packet loss percent, with a cap at MIN_BITRATE
videoBitrate = (UINT64) MAX(videoBitrate * (1.0 - pSampleStreamingSession->twccMetadata.averagePacketLoss / 100.0), MIN_VIDEO_BITRATE_KBPS);
// decrease encoder bitrate by average packet loss percent, with a cap at MIN_BITRATE
audioBitrate = (UINT64) MAX(audioBitrate * (1.0 - pSampleStreamingSession->twccMetadata.averagePacketLoss / 100.0), MIN_AUDIO_BITRATE_BPS);
}

// Update the session with the new bitrate and adjustment time
pSampleStreamingSession->twccMetadata.newVideoBitrate = videoBitrate;
pSampleStreamingSession->twccMetadata.newAudioBitrate = audioBitrate;
MUTEX_UNLOCK(pSampleStreamingSession->twccMetadata.updateLock);

pSampleStreamingSession->twccMetadata.lastAdjustmentTimeMs = currentTimeMs;

DLOGI("Adjustment made: average packet loss = %.2f%%, timediff: %llu ms", pSampleStreamingSession->twccMetadata.averagePacketLoss, timeDiff);
DLOGI("Suggested video bitrate %u kbps, suggested audio bitrate: %u bps, sent: %u bytes %u packets received: %u bytes %u packets in %lu msec",
videoBitrate, audioBitrate, txBytes, txPacketsCnt, rxBytes, rxPacketsCnt, duration / 10000ULL);
}

STATUS handleRemoteCandidate(PSampleStreamingSession pSampleStreamingSession, PSignalingMessage pSignalingMessage)
Expand Down Expand Up @@ -888,6 +939,12 @@ STATUS createSampleConfiguration(PCHAR channelName, SIGNALING_CHANNEL_ROLE_TYPE
pSampleConfiguration->pregenerateCertTimerId = MAX_UINT32;
pSampleConfiguration->signalingClientMetrics.version = SIGNALING_CLIENT_METRICS_CURRENT_VERSION;

// Flag to enable SDK to calculate selected ice server, local, remote and candidate pair stats.
pSampleConfiguration->enableIceStats = FALSE;

// Flag to enable/disable TWCC
pSampleConfiguration->enableTwcc = TRUE;

ATOMIC_STORE_BOOL(&pSampleConfiguration->interrupted, FALSE);
ATOMIC_STORE_BOOL(&pSampleConfiguration->mediaThreadStarted, FALSE);
ATOMIC_STORE_BOOL(&pSampleConfiguration->appTerminateFlag, FALSE);
Expand Down
19 changes: 19 additions & 0 deletions samples/Samples.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,12 @@ extern "C" {
#define MAX_SIGNALING_CLIENT_METRICS_MESSAGE_SIZE 736 // strlen(SIGNALING_CLIENT_METRICS_JSON_TEMPLATE) + 20 * 10
#define MAX_ICE_AGENT_METRICS_MESSAGE_SIZE 113 // strlen(ICE_AGENT_METRICS_JSON_TEMPLATE) + 20 * 2

#define TWCC_BITRATE_ADJUSTMENT_INTERVAL_MS 1000 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND
#define MIN_VIDEO_BITRATE_KBPS 512 // Unit kilobits/sec. Value could change based on codec.
#define MAX_VIDEO_BITRATE_KBPS 2048000 // Unit kilobits/sec. Value could change based on codec.
#define MIN_AUDIO_BITRATE_BPS 4000 // Unit bits/sec. Value could change based on codec.
#define MAX_AUDIO_BITRATE_BPS 650000 // Unit bits/sec. Value could change based on codec.

typedef enum {
SAMPLE_STREAMING_VIDEO_ONLY,
SAMPLE_STREAMING_AUDIO_VIDEO,
Expand Down Expand Up @@ -159,6 +165,8 @@ typedef struct {

PCHAR rtspUri;
UINT32 logLevel;
BOOL enableIceStats;
BOOL enableTwcc;
} SampleConfiguration, *PSampleConfiguration;

typedef struct {
Expand All @@ -178,6 +186,16 @@ typedef struct {

typedef VOID (*StreamSessionShutdownCallback)(UINT64, PSampleStreamingSession);

typedef struct {
MUTEX updateLock;
UINT64 lastAdjustmentTimeMs;
UINT64 currentVideoBitrate;
UINT64 currentAudioBitrate;
UINT64 newVideoBitrate;
UINT64 newAudioBitrate;
DOUBLE averagePacketLoss;
} TwccMetadata, *PTwccMetadata;

struct __SampleStreamingSession {
volatile ATOMIC_BOOL terminateFlag;
volatile ATOMIC_BOOL candidateGatheringDone;
Expand All @@ -197,6 +215,7 @@ struct __SampleStreamingSession {
UINT64 startUpLatency;
RtcMetricsHistory rtcMetricsHistory;
BOOL remoteCanTrickleIce;
TwccMetadata twccMetadata;

// this is called when the SampleStreamingSession is being freed
StreamSessionShutdownCallback shutdownCallback;
Expand Down
Loading

0 comments on commit ee9925e

Please sign in to comment.