AV1 support

[aixinaxc]

When will AV1 support be added to webrtc？

[Sean-Der]

Hi @aixinaxc

It works today! You can call RegisterCodec on the MediaEngine

we don’t have a packetizer yet however. If you are interested in contributing that would be great. Then we can turn on by default.

thanks

[tmatth]

Is there a depacketizer?

[Sean-Der]

@tmatth we have neither yet unfortunately!

If this is actually blocking anyone happy to make a priority. This would be a fun first contribution if anyone is interested.

if you are new to Pion feel free to ping me on Slack with questions.

[tmatth]

Reference:
https://aomediacodec.github.io/av1-rtp-spec/

[jech]

Do any browsers implement it yet?

[Sean-Der]

Chrome does Release Announcement

It looks like this could be implement pretty quickly! Payload Header is all we need to worry about.

For testing we should add a var args optional parameters to the ivfwriter like SampleBuilder that allows someone to set the FOURCC

[lnogueir]

Do any browsers implement it yet?

It has been available in libwebrtc for a while, so technically browsers that use that WebRTC implementation could support it.

Check these articles for details: http://webrtcbydralex.com/?s=Av1

Some people have been able to set up AV1 encoding on Chrome Canary, as explained here: https://medooze.medium.com/mastering-the-av1-svc-chains-a4b2a6a23925

[jech]

Confirmed, I've just managed to push an AV1 flow through Galène. Essentially, it consisted in adding

webrtc.RTPCodecCapability{
		"video/AV1X", 90000, 0,
		"",
		nil,
}

at the right point in the code. The browser is Brave 1.23.71 based on Chromium 90 (just because it's the easiest way to get a recent build of Chromium with H.264 support for Debian, please don't take that as an endorsment of Brave or of anything related to cryptocurrencies).

There are a few things that I find weird:

• the SDP type is AV1X, not AV1 as specified in the RTP format;
• the SDP doesn't contain a fmtp with the profile;
• the ptype is 35, which is unassigned according to the IANA registry (it should probably be in the dynamically assigned range?).

I haven't looked at the packet contents yet, so I don't know if they're generating useful dependency descriptors (without that, it's going to be tricky to detect keyframes). I also haven't looked at how difficult it would be to write a depacketiser.

And if you're wondering — the CPU usage is high (25% of a four-core laptop for a single flow), but not prohibitive.

[jech]

There's no dependency descriptor (or perhaps I'm not negotiating it correctly).

[Sean-Der]

One requirement for AV1 support is to add support to our IVFWriter.

We should add a *Option... argument and allow users to specify VP8, VP9 or AV1.

[jech]

I think there's also an issue with the samplebuilder. The samplebuilder assumes that it is enough to concatenate packet bodies, but I don't think that's going to work for AV1, since the size of the last OBU in a packet is omitted.

[NadeenUdantha]

maybe useful: https://github.com/NadeenUdantha/DependencyDescriptor

It looks like this could be implement pretty quickly! Payload Header is all we need to worry about.

https://gist.github.com/NadeenUdantha/a864b617c19639e8cc506684a2d95745

[dannymn]

maybe useful: https://github.com/NadeenUdantha/DependencyDescriptor

It looks like this could be implement pretty quickly! Payload Header is all we need to worry about.

https://gist.github.com/NadeenUdantha/a864b617c19639e8cc506684a2d95745

any news regarding AV1 codec support ?
I'm using this new codec for almost a year now inside both chrome browser as mentioned in the thread and also with my native c++ libwebrtc code (shared with chrome as well...)

chrome sdp video line indeed includes -> "AV1/90000" (not AV1X any more starting with chrome 91 and above)

in order to manipulate chrome default encoder or my libWebRTC based streamer app I put in "m=video" line the following order starting with payload 35 / 36 (chrome/libwebrtc regsiter AV1 under this payload type id) so default codec priority is AV1 -> vp8 -> vp9 -> h264 etc...
////AV1 support
string newCodecOrder = "m=video 9 UDP/TLS/RTP/SAVPF 35 36 96 97 98 99 100 101";
sdpStr.replace(posCodecOrder, codecOrder.length(), newCodecOrder);

can i ask you guys to add this to ion-sfu as soon as possible? AV1 outperforms vp8 quite easily with both quality and bandwidth usage...with minimal CPU overhead/impact...when dealing with Camera HD/High Res streaming, I also use AV1 for streaming Screen Share (1920X1080...) working prety nice as well....

many thanks!
Danny

[jech]

chrome sdp video line indeed includes -> "AV1/90000" (not AV1X any more starting with chrome 91 and above)

Thanks for the info. I've just adapted Galene to the new name jech/galene@6fbdf0e

[dannymn]

I don't think it is enough->
i believe work needed to be added here in order to support AV1 packet buffer under \ion-sfu\pkg\buffer directory similar to ->

buffer.go:

switch b.mime {
case "video/vp8":
vp8Packet := VP8{}
if err := vp8Packet.Unmarshal(p.Payload); err != nil {
return
}
ep.Payload = vp8Packet
ep.KeyFrame = vp8Packet.IsKeyFrame
case "video/h264":
ep.KeyFrame = isH264Keyframe(p.Payload)
}

if b.minPacketProbe < 25 {
	if sn < b.baseSN {
		b.baseSN = sn
	}

	if b.mime == "video/vp8" {
		pld := ep.Payload.(VP8)
		mtl := atomic.LoadInt32(&b.maxTemporalLayer)
		if mtl < int32(pld.TID) {
			atomic.StoreInt32(&b.maxTemporalLayer, int32(pld.TID))
		}
	}

	b.minPacketProbe++
}

helpers.go

type VP8 struct {
TemporalSupported bool
// Optional Header
PictureID uint16 /* 8 or 16 bits, picture ID /
PicIDIdx int
MBit bool
TL0PICIDX uint8 / 8 bits temporal level zero index */
TlzIdx int

// Optional Header If either of the T or K bits are set to 1,
// the TID/Y/KEYIDX extension field MUST be present.
TID uint8 /* 2 bits temporal layer idx*/
// IsKeyFrame is a helper to detect if current packet is a keyframe
IsKeyFrame bool

}

[jech]

I think there might be a misunderstanding. This bug tracker is about Pion, a WebRTC library. This issue is about changes to Pion WebRTC that might be required for full AV1 support.

Galene and Ion-SFU are two distinct projects that use Pion. Galene has basic support for AV1, even though the support in Pion is incomplete. I have no idea about Ion-SFU.

If you have patches against Pion to submit, you should submit them here. If you have patches against Ion-SFU, you should submit them at https://github.com/pion/ion-sfu

[dannymn]

Dear Juliusz,
my remarks regarding Pion missing AV1 support has nothing with Galene project.
I'm interested in adding AV1 support in ion-SFU, as far as i understand and maybe i'm wrong here, ion-SFU and Pion sharing same WebRTC stack code (or perhaps better say ion-sfu using Pion webrtc support) , so adding AV1 support in Pion, should find its way quickly into relying Ion-Sfu, am i wrong here? i could try posting it on ion-sfu, but as far as i checked, i didn't see any open issue regarding AV1 support in ion-SFU yet...cheers!