Add support for Opus encoding/decoding

[ru-ka]

Describe the project you are working on

I'm working on a multiplayer game with voice chat.

Describe the problem or limitation you are having in your project

Vorbis is not a great choice since is almost legacy and opus outperforms vorbis in every aspect (also the cpu usage is almost identical in decoding).

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This feature could improve every area that uses audio such as multiplayer voice chat and high quality audio playback (opus could offer a very high fidelity, similar to the one of aac).

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

It would be cool to use ogg opus files as sounds, by implementing AudioStreamOggOpus.

If this enhancement will not be used often, can it be worked around with a few lines of script?

It is not fast and convenient to implement that feature as a script, because of the limitations of GDExtension.

Is there a reason why this should be core and not an add-on in the asset library?

Opus should be the default for the reasons mentioned above. Other than that is impossible to overcome certain gdextension's limitations and provide a complete integration with this codec since such modifications touch a core part of the engine.

[KoBeWi]

Opus was actually supported in the past, but got removed due to streaming issues. See godotengine/godot#7496

[DeeJayLSP]

Streaming issues due to being RT unsafe were the reason Speex and Opus got ditched and libvorbis got replaced with stb_vorbis in 3.0.

Considering how 4.0 brought libvorbis back (along with the new OggPacketSequence class which I believe it has the role of making it RT safe) I don't think the streaming issues will be back.

[Calinou]

• Related to Add support for FLAC decoding #1190.

It is not fast and convenient to implement that feature as a script, because of the limitations of GDExtension.

I believe we should look into fixing and improving GDExtension 🙂

While I'm not opposed to adding Opus support in core, GDExtension being incomplete isn't a reason to keep adding more features to core endlessly.

opus could offer a very high fidelity, similar to the one of aac).

Ogg Vorbis already outperforms AAC in terms of quality to file size ratio with modern encoder versions and settings.

[DeeJayLSP]

Ogg Vorbis already outperforms AAC in terms of quality to file size ratio with modern encoder versions and settings.

Opus outperforms both. You can have a reasonably good quality at just 96kbps and transparency at 128kbps. You just need to avoid old/non-official encoder versions.

It was designed to be a successor to Speex and Vorbis anyway.

[teknico]

You can have a reasonably good quality at just 96kbps and transparency at 128kbps.

With v. 1.3 or higher, you can have quite good quality for voice at 32kbps and for music at 64kbps, both stereo. See https://jmvalin.ca/opus/opus-1.3/

V. 1.3 was released four years ago.

[Jimmio92]

Can we get this implemented for 4.2? Being able to slap some samples into an Opus stream and toss them over the network to other players would be really awesome, on top of the fact it was intended to replace vorbis and speex, the former being used pretty heavily by users of Godot.

[Calinou]

Can we get this implemented for 4.2?

There is no ETA for this being implemented, as nobody has looked into it yet. Audio contributors are difficult to find, so it may take a while to happen.

[Faless]

It is not fast and convenient to implement that feature as a script, because of the limitations of GDExtension.

Currently, there doesn't seem to be any limitation preventing this from being a GDExtension as shown by this FLAC implementation

[ru-ka]

It is not fast and convenient to implement that feature as a script, because of the limitations of GDExtension.

Currently, there doesn't seem to be any limitation preventing this from being a GDExtension as shown by this FLAC implementation

I've not seen the extension in detail but it's primary use is for local media playback and not, also, for voice communications or audio streaming over the network. Although they're both audio codecs the usage changes a lot.

[Faless]

I've not seen the extension in detail but it's primary use is for local media playback and not, also, for voice communications or audio streaming over the network. Although they're both audio codecs the usage changes a lot.

Well, if you found there are limitations in GDExtension for the specific "voice communications" use case, then those limitations should be stated.

The original proposal states:

It would be cool to use ogg opus files as sounds, by implementing AudioStreamOggOpus.

and that is indeed possible to implement as a GDExtension, as shown by the FLAC extension which implements AudioStreamFLAC.

[goatchurchprime]

I've done some initial work porting the libopus library into Godot4 https://github.com/goatchurchprime/libopus-gdnative/tree/4.1

In practice, the Opus library is not a lot of good on its own. It only makes sense when it is part of a VOIP feature. And for that you also have to have the networking system, the functions to break down the AudioStream from the microphone into the right size frames, and even more functions and buffers to reassemble them at the receiving end while handling dropped packets and delays.

Keeping the binding of the Opus library out of the godotengine core and as a GDExtension means it will be easier to make changes to its functional interface as a VOIP application gets developed.

[atirut-w]

I'd love for Godot to get built-in Opus support also for its unreasonably good quality-per-bitrate ratio, which means audio can take less space for what is practically the same quality you get with other codecs. Games with a lot of audio assets (or very long ones) could save a lot of space with Opus.

[unfa]

I wonder what were the issues with Opus in regards to streaming and playback - it's widely used for VoIP and is designed to allow using extremely small stream packets for low latency if need be.

Maybe it is a matter of the encoder/decoder used? The most lightweight implementation might not be suitably robust for our needs.

It might be a good idea to create something like an AudioStreamTransmitter effect (encodes and sends audio from a bus via network) and AudioStreamReciever resource (listens on a port and decodes)- having audio streaming be part of the high-level multiplayer toolset.

With audio streaming, especially for VoIP there's a need for good dropout handling. Usually it means the decoder repeats last received data for a moment to mask the gap in the signal (it's way less distracting that having gaps of dead silence), I don't know if this is part of the Opus decoder we'd use. It probably should be.

[atirut-w]

encodes and sends audio from a bus via network

Mayhaps off topic, but does that allow for microphone input?