Enhance audio codecs

[flying1314]

Such as opus over rtmp,so that it can be passed to webrtc,no need to transcode

[veovera]

We are looking at adding additional audio codecs as part of the enhancement to the RTMP. Opus is one of the codecs we are investigating. Stay tuned...

[flying1314]

Good, Opus for webrtc, AC3 for dolby effect, and any other audio codecs not listed

[umekzg]

Multi-channel audio!

[veovera]

There are many ways to accomplish multi-channel audio. Care to point to what you had in mind?

[krakow10]

Multi-channel audio!

Did you mean multiple audio tracks? Is that already supported in enhanced rtmp?

[veovera]

fyi: today RTMP streams are limited to a single audio track

[krakow10]

Does enhanced rtmp support multiple audio tracks?

[veovera]

not today

[krakow10]

It should 😉
That way it could replace the custom implementation Twitch is using for VOD audio. Thanks for the information.

[jprjr]

FLAC would be a great codec to add.

Use-case is pushing a stream to something that winds up transcoding to multiple formats.

There is linear PCM in the flv spec but that's limited to 44.1kHz.

[TroyKomodo]

Multiple audio tracks are technically supported by the rtmp spec, since you can either reuse the same connection and publish a second stream for audio, or you can have a 4th stream id with the additional audio track.

Currently most implementations dont let you do this however, but it could be very easily implemented and im not exactly sure why it was never implemented.
@veovera @krakow10

[veovera]

FLAC would be a great codec to add.

Use-case is pushing a stream to something that winds up transcoding to multiple formats.

There is linear PCM in the flv spec but that's limited to 44.1kHz.

@jprjr Perhaps can you created a seperate issue to support FLAC. I am trying to avoid overloading the same issue wit multiple request for ease of tracking?

I understand the use case, but can you point me to solutions/services/apps shipping today which would benefit from FLAC support

[veovera]

Multiple audio tracks are technically supported by the rtmp spec, since you can either reuse the same connection and publish a second stream for audio, or you can have a 4th stream id with the additional audio track.

Currently most implementations dont let you do this however, but it could be very easily implemented and im not exactly sure why it was never implemented. @veovera @krakow10

• Yes you can multiplex on the server, but you can only send one audio stream at a time and the switch would have a delay. My guess is that it has been done by some custom clients to support multi language.
• sending a stream on a separate stream id means you have to have client side logic to sync multiple streams. That has it's own set of challenges and complexities, but not impossible. I suspect that is the reason that has not been done. There could be some interesting use cases if a client supported this capability.

[zenomt]

the major complication for synchronizing playback of multiple NetStreams/stream-IDs is that the normal behavior of Adobe Media Server (which other servers mimic) is to start a new stream's timestamps at 0. if you could ask the server to not do that and provide the original timestamps from the publisher, a client could trivially synchronize alternate tracks published as different streams.

tcserver supports that with the asis: stream name prefix. :)

[zenomt]

for supporting other codecs, note that there are 5 defined audio codec code points with the high bit set (G.711 µ-Law, AAC, Speex, MP3 8kHz, and "device specific"), so all of the bits for an audio message's first byte have a meaning. that means a similar encoding used for enhanced video codecs can't be used for audio. and note that some folks (such as myself ;) ) use the "device specific" code point today (in my case i'm using it for Opus right now), so that one shouldn't be taken even though it has the tasty and tempting "all 1s" bit pattern.

if it was me, i'd choose code point 9 or 13 for "extended", with 4 bytes of fourcc and one byte of packet type (similar to AAC and the enhanced video packet types).

[Thulinma]

I'd like to quick point out that @TroyKomodo is completely right and multi-track (both video and audio) is already supported by the base RTMP spec. (And indeed, hardly any implementations support it, for reasons that baffle me as well.)
It would be nice to "formally" state this method should be supported (or to signal such support with a flag somewhere or somesuch) when using enhanced RTMP, as it makes it possible to do e.g. multi-angle streaming (more than one video track) and multi-language streaming (more than one audio track) and similar use cases.

Note that for multi-track, you wouldn't connect multiple times or send multiple streams, but instead use different chunkstream ID for each track. On a side note, I always found it strange that almost all implementations use a single chunkstream ID for both audio and video, as it makes the RTMP headers much less efficient to do so. 🤔

To stay more on topic: I'd +1 both Opus and FLAC audio support as well. Was excited to find an effort to extend RTMP finally happening, but disappointed to see it only adds 3 video codecs and that's it. Let's make it happen ^_^

[zenomt]

Note that for multi-track, you wouldn't connect multiple times or send multiple streams, but instead use different chunkstream ID for each track.

i hope no implementations actually do this, because this is both a layering violation and is inconsistent with the semantics of RTMP, RTMFP, and FLV. the RTMP chunkstream is responsible for framing and interleaving the higher-layer RTMP messages in a reliable in-order stream. chunkstream IDs are not an additional semantic dimension (like an additional track) for RTMP messages.

an RTMP stream ID has at most one video track and at most one audio track.

[zenomt]

I always found it strange that almost all implementations use a single chunkstream ID for both audio and video, as it makes the RTMP headers much less efficient to do so.

most implementations treat RTMP as a framing format and completely ignore that the "RT" stands for "Real-Time". most implementations also don't do any real-time treatment at all when using RTMP, such as prioritizing audio over video, or having transmission deadlines. many implementations can't handle when a message has been abandoned partway through transmission.

[zenomt]

for clarification on why using the chunkstream ID as an additional dimension is a layering violation and inconsistent with the semantics and intent of RTMP messages, please see the first two paragraphs of Section 6 of the RTMP spec and the third paragraph of the introduction to RFC 7425 (Adobe's RTMFP Profile for Flash Communication).

[Thulinma]

That might be the intended meaning, but I'd argue that neither spec actually explicitly says this is not allowed. Instead, it's left entirely ambiguous, leading to this kind of misunderstanding. It states that each chunk stream ID carries a single message type from a single stream ID. (This is already broadly ignored as in practically all implementations I've encountered, the audio and video message types are multiplexed over a single chunk stream ID). It however says nothing about what happens when a single message type and stream ID are multiplexed over multiple different chunk IDs. Some implementations might consider those separate tracks, some implementations might consider it a single track that can handle "slow" and "fast" messages independently (which, for video and audio message types, is not useful within the supported codecs because as far as I'm aware there is no such concept in any of them). Since I've never encountered an implementation that uses multiple chunk stream IDs for a single message type ID and stream ID, my implementation considers them separate tracks when receiving this kind of data. As far as I'm aware, in the past ~12 years, this has not caused issues even once with any existing other implementation.

I'm suggesting we remove the ambiguity and explicitly allow this usage of chunkstream IDs. Alternatively, we could use multiple stream IDs instead - but this has the downside of there being no explicit or implicit synchronization between the timestamps of multiple stream IDs (it, too, is left entirely ambiguous as the spec says nothing about whether any constraints or expectations exist or not). Defining that synchronization would also allow multi-track, albeit with far more overhead than using multiple chunksteams would (unless we say this is not needed, of course - I'm not sure how many existing implementations support multiple stream IDs over a single connection to begin with).

I'm not too concerned with any of that breaking compatibility for such streams with FLV or RTMFP, because those are not in common use anymore (unlike RTMP) and it would be trivial to mux the data into a more suitable muxing format that does support multi-track after receipt. As long as this method is only applied when support was signaled in advance, I don't see any practical harm to it.

Either way and separately from all this, it might be wise to "patch" the RTMP spec to remove the missing, ambiguous (or even blatantly wrong) parts. Over the years, these have been a source of needless reverse engineering, headaches and bugs that should all really not have been necessary. Since RTMP is apparently not going away any time soon and new implementations keep popping up, having a spec that is both complete and correct seems critical. Especially the RTMP handshake being wrong (as far as I can tell purposefully so) in an attempt to make Flash Player only play H264 when connecting to Adobe software is jarring. Thankfully all non-Adobe implementations ignore the contents of the handshake and Adobe implementations are rapidly going extinct, so today this is no longer really an issue.

[zenomt]

Either way and separately from all this, it might be wise to "patch" the RTMP spec to remove the missing, ambiguous (or even blatantly wrong) parts.

+1. the error in the RTMP spec i'm most embarrassed about is missing that the "cs id - 64" field in the 3-byte Chunk Basic Header form is little-endian, not big-endian as implied by the spec (EDIT: very careful reading of the formula does show it's little-endian, but i missed that at first in my own implementation, and it's certainly not prominent). i blame not writing a new implementation based just on the spec while editing it all those years ago for missing that. i missed a lot more too. i've wanted to write a corrigendum and a "missing stuff you really want to know for interoperation" doc for a long time.

in the spirit of staying on topic for this Issue though, further discussion of this and ways of signaling/transporting additional tracks probably belong in a Discussion or two.

[zenomt]

Either way and separately from all this, it might be wise to "patch" the RTMP spec to remove the missing, ambiguous (or even blatantly wrong) parts.

apologies for continuing to be off-topic, but because of this thread i've started an RTMP Errata and Addenda document (source link) to finally try to address some of the errors, omissions, and ambiguities in the RTMP spec.

[veovera]

We're pleased to announce the integration of FLAC and Opus audio codecs into the Enhanced RTMP specification. For more details on this update, please see our #26. You can access the detailed specification here.