codec + configuration #19

ibc · 2016-03-11T20:48:13Z

mediasoup needs to hold a map of payloadType and codecs+configuration. So I need to know what "codec+configuration" means (note that some codecs have their own configuration parameters...).

Mail sent to AVT WG: https://www.ietf.org/mail-archive/web/avt/current/msg17068.html

https://en.wikipedia.org/wiki/RTP_audio_video_profile

The text was updated successfully, but these errors were encountered:

ibc · 2016-03-22T16:10:47Z

Response by Magnus:

10.1.2. Payload Type (PT) Value Reuse

Multiple bundled "m=" lines might represent RTP based media. As all
RTP based media associated with a BUNDLE group belong to the same RTP
session, in order for a given payload type value to be used inside
more than one bundled "m=" line, all codecs associated with the
payload type number MUST share an identical codec configuration.
This means that the codecs MUST share the same media type, encoding
name, clock rate and any parameter that can affect the codec
configuration and packetization.
[I-D.ietf-mmusic-sdp-mux-attributes] lists SDP attributes, whose
attribute values must be identical for all codecs that use the same
payload type value.

So identical a=rtpmap and a=fmtp lines would meet this criteria.

However, two different m= blocks configuring a particular PT for a codec could possibly use different values for an optional parameter that can be received correctly by either configurations.

Codecs that can produce different encoding bit-rates can in many cases be used at different bit-rates and fulfil this requirement.

The underlying issue is that what is identical codec configurations are codec specific. Thus hard to write a precise definition of it.

ibc · 2016-03-23T12:13:20Z

So let's check how Chrome Canary M51 and Firefox Nightly 48 offer "codecs":

Chrome:

m=audio 49606 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10; useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60

m=video 58734 UDP/TLS/RTP/SAVPF 100 101 116 117 96 97 98
a=rtpmap:100 VP8/90000
a=rtpmap:101 VP9/90000
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=100
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=101
a=rtpmap:98 rtx/90000
a=fmtp:98 apt=116

Firefox:

m=audio 64761 UDP/TLS/RTP/SAVPF 109 9 0 8
a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 53852 UDP/TLS/RTP/SAVPF 120 126 97
a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

ibc · 2016-03-23T12:48:22Z

As I commented in the AVT ML:

Offer by Firefox 48:

m=audio 49706 UDP/TLS/RTP/SAVPF 109 9 0 8
a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1
a=rtpmap:9 G722/8000/1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 61083 UDP/TLS/RTP/SAVPF 120 126 97
a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

Answer by Chrome 51:

m=audio 61998 UDP/TLS/RTP/SAVPF 109 9 0 8
a=rtpmap:109 opus/48000/2
a=fmtp:109 minptime=10; useinbandfec=1
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 9 UDP/TLS/RTP/SAVPF 120
a=rtpmap:120 VP8/90000

1a) opus in Firefox:

a=rtpmap:109 opus/48000/2
a=fmtp:109 maxplaybackrate=48000;stereo=1

1b) opus in Chrome:

a=rtpmap:109 opus/48000/2
a=fmtp:109 minptime=10; useinbandfec=1

So I must assume that maxplaybackrate, stereo, minptime and useinbandfec are optional params for opus, or that their default values (if omitted) match those values, so both codecs+configuration are the "same".

2a) VP8 in Firefox:

a=rtpmap:120 VP8/90000
a=fmtp:120 max-fs=12288;max-fr=60

2b) VP8 in Chrome:

a=rtpmap:120 VP8/90000

Here also I assume that max-fs and max-fr are optional parameters that do not prevent both offered codec+configuration to match.

3a) G722 in Firefox:

a=rtpmap:9 G722/8000/1

3b ) G722 in Chrome:

a=rtpmap:9 G722/8000

Do they match? Is "1" the default value for the 'encoding' field in a rtpmap entry?

4) H264 in Firefox:

a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1

a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1

Since Firefox offers two H264 codecs I must assume they are, indeed, different codecs. So it seems that "packetization-mode=1" make them different (which also means that its default value is not 1). Right?

This is terribly complex and unmanageable. And this requires the SFU developer to become an expert in codec specific parameters (regardless the SFU won't decode media).

ibc · 2016-03-24T15:26:25Z

Regarding H264 packetization-mode parameter:

https://tools.ietf.org/html/rfc6184#section-6

6.2. Single NAL Unit Mode

This mode is in use when the value of the OPTIONAL packetization-mode
media type parameter is equal to 0 or the packetization-mode is not
present. All receivers MUST support this mode.

So valid values are 0 (default if not set), 1 and 2. Mandatory for WebRTC are 0 and 1.

ibc · 2016-03-29T10:18:16Z

Chrome 50 with H264 support (enabled in chrome://flags):

a=rtpmap:107 H264/90000

Just that. No fmtp line. Since no packetization-mode is given then it gets its default value 0. What about profile-level-id?

/cc @saghul

saghul · 2016-03-29T10:21:14Z

If the attribute is absent the default is used: 420010.

ibc · 2016-03-29T10:22:09Z

Wow, so indeed Firefox and Chrome do not interop at H264 level...

ibc · 2016-03-29T10:25:39Z

Moving comments form #18 here:

UPDATE: After a good chat with @jmillan, the room will be provided with mandatory videoCodec and audioCodec parameters so endpoints MUST support them and MUST use them (and no other codecs). Period.

Still we need an internal list of "valid" audio/video codecs in mediasoup. The list (to be completed) would start with:

Audio:
- opus (default)
- alaw
- ulaw
Video:
- VP8 (default)
- H264 packetization-mode=0
- H264 packetization-mode=1

@saghul said:

If 2 different endpoints use different H264 profile-level-id attributes someone will get confused, I'm afraid.

The profile-level-id can also make H264 codecs different. A low powered device might not be able to handle the "high" profile, for example.
On the implementations I've worked with, we always accept whatever the remote wants, but I guess this need not be the case for eveyone.

@ibc said:

If so, different profile-level-id values would also generate different codec units. May you please detail those values?

@saghul said:

They could. There are different profiles and levels. Also check https://tools.ietf.org/html/rfc6184#section-8.1

@ibc said:

For example, as shown in #19, Firefox offers two H264 configurations:
a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;level-asymmetry-allowed=1
Does it mean that it does not support other profile-level-id?

@saghul said:

No. Those 2 configurations use the same profile-level-id, but different packetization-mode, hence the need for 2 configs.

saghul · 2016-03-29T10:32:27Z

Wow, so indeed Firefox and Chrome do not interop at H264 level...

Not necessarily. One of them could just accept the offer and use the supplied profile-level-id. Here they both use the same profile (42, baseline) and different level (1f == 31 == 3.1 vs 10 == 1.0). Different levels could cause problems, but more importantly, different profiles could (42 == baseline vs 4d == main), for example.

ibc · 2016-03-29T10:56:46Z

Interesting.... so what? XD

ibc · 2016-03-29T10:59:32Z

At the end I need specific and separate "codec configurations" to be thread as the "same codec+configuration".

Note however that codec parameters given by each peer will be properly indicated to each receiver peer (so Chrome will be the H264 params of Firefox and vice-versa), but still I need a proper list of "codec + configurations".

notedit · 2016-04-16T15:25:20Z

no vp9 codecs？

ibc · 2016-04-16T15:27:11Z

mediasoup is a SFU so it does not encode/decode codecs. This means that any codec will be supported.

ollyde · 2022-07-11T19:01:57Z

I have an issue where https://v3demo.mediasoup.org/?roomId=aaaahh works

But using the default codecs as explained here fails locally, does anyone know what config the demo server is running?

Fails for Web (Chrome)

{
	kind      : 'audio',
	mimeType  : 'audio/opus',
	clockRate : 48000,
	channels  : 2
},
{
	kind       : 'video',
	mimeType   : 'video/VP8',
	clockRate  : 90000,
	parameters :
	{
		'x-google-start-bitrate' : 1000
	}
},

ibc added the help wanted label Mar 11, 2016

ibc self-assigned this Mar 11, 2016

ibc added this to the 1.X.Y milestone Mar 11, 2016

ibc mentioned this issue Mar 29, 2016

BUNDLE and payload values constraint #18

Closed

ibc assigned ibc and unassigned ibc Jun 20, 2016

ibc removed this from the 1.0.0 milestone Jan 2, 2017

ibc closed this as completed Mar 27, 2017

Ambyjkl mentioned this issue Jun 26, 2020

C++ crashes when using PipeTransport (maybe) #429

Closed

euthyphro8 mentioned this issue Jun 3, 2021

Worker dies as soon as client transport connects #580

Closed

snyk-bot mentioned this issue Mar 12, 2022

[Snyk] Upgrade randomatic from 3.0.0 to 3.1.1 BTDevelop/mediasoup#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

codec + configuration #19

codec + configuration #19

ibc commented Mar 11, 2016

ibc commented Mar 22, 2016

ibc commented Mar 23, 2016

ibc commented Mar 23, 2016

1a) opus in Firefox:

1b) opus in Chrome:

2a) VP8 in Firefox:

2b) VP8 in Chrome:

3a) G722 in Firefox:

3b ) G722 in Chrome:

4) H264 in Firefox:

ibc commented Mar 24, 2016

ibc commented Mar 29, 2016

saghul commented Mar 29, 2016

ibc commented Mar 29, 2016

ibc commented Mar 29, 2016

saghul commented Mar 29, 2016

ibc commented Mar 29, 2016

ibc commented Mar 29, 2016

notedit commented Apr 16, 2016

ibc commented Apr 16, 2016

ollyde commented Jul 11, 2022 •

edited

codec + configuration #19

codec + configuration #19

Comments

ibc commented Mar 11, 2016

ibc commented Mar 22, 2016

ibc commented Mar 23, 2016

ibc commented Mar 23, 2016

1a) opus in Firefox:

1b) opus in Chrome:

2a) VP8 in Firefox:

2b) VP8 in Chrome:

3a) G722 in Firefox:

3b ) G722 in Chrome:

4) H264 in Firefox:

ibc commented Mar 24, 2016

ibc commented Mar 29, 2016

saghul commented Mar 29, 2016

ibc commented Mar 29, 2016

ibc commented Mar 29, 2016

saghul commented Mar 29, 2016

ibc commented Mar 29, 2016

ibc commented Mar 29, 2016

notedit commented Apr 16, 2016

ibc commented Apr 16, 2016

ollyde commented Jul 11, 2022 • edited

ollyde commented Jul 11, 2022 •

edited