Red5 client fails to decode RTMP from FFmpeg server

[JonathanMurray]

Hello, and thank you for this project!

Problem summary

I'm using a Red5 RTMP client for connecting to and playing audio from an FFmpeg process (ffmpeg -re -f lavfi -i sine -f flv -c copy -flvflags no_duration_filesize -listen 1 rtmp://localhost:1935/live/testing). Before any audio is received by Red5, a decoding error occurs, due to a class cast exception inside RTMPProtocolDecoder::readTransactionId (class java.lang.String cannot be cast to class java.lang.Number).

Details

I've tried looking at this on both the Red5 side and FFmpeg side, and my understanding can be summed as follows:

• The decoding error happens when Red5 receives a "@setDataFrame onMetaData" message from FFmpeg. "@setDataFrame" is being interpreted as an action, and where Red5 excepts to find a transaction ID in the message it instead gets the string "onMetaData".
• The reason Red5 tries to parse the message as an action is this if-statement: https://github.com/Red5/red5-server-common/blob/master/src/main/java/org/red5/server/net/rtmp/codec/RTMPProtocolDecoder.java#L525
• FFmpeg sends the "@setDataFrame onMetaData" message with a Message Stream ID of 0.

Please correct me if the above doesn't make sense, or if you think I have misunderstood something. Based on the above, I'd like some help to answer the following questions:

1. Why does Red5 parse messages differently based on header.getStreamId() being 0 or not? Is this rooted in something from the RTMP specification that can be referred to? From what I understand, this behaviour has been here since (at least) 2014 when code was pulled in from a fork.

2. Is it incorrect of FFmpeg to send "@setDataFrame on Metadata" with Message Stream ID = 0? Is there something in the RTMP specification that provides more details on this topic?

Reproduction steps

Run an FFmpeg RTMP server:
ffmpeg -re -f lavfi -i sine -f flv -c copy -flvflags no_duration_filesize -listen 1 rtmp://localhost:1935/live/testing

Run the following Java program for connecting to the FFmpeg server with a Red5 client:

import org.red5.client.net.rtmp.RTMPClient;

/*
This example uses the following mvn dependencies:
<dependency>
  <groupId>org.red5</groupId>
  <artifactId>red5-client</artifactId>
  <version>1.2.12</version>
</dependency>
<dependency>
  <groupId>org.red5</groupId>
  <artifactId>red5-server-common</artifactId>
  <version>1.2.12</version>
</dependency>
 */

public class Red5AndFfmpeg {
  public static void main(String[] args) {
    var red5Client = new RTMPClient();
    red5Client.setStreamEventDispatcher(event -> System.out.println("Received event: " + event));
    red5Client.connect(
        "[::1]", // change to "localhost" for IPv4
        1935,
        "live",
        connectCall -> {
          assert connectCall.isSuccess();
          red5Client.createStream(
              streamCall -> {
                assert streamCall.isSuccess();
                double streamId = (Double) streamCall.getResult();
                red5Client.play(streamId, "testing", 0, -1);
              });
        });
  }
}

When running the above, I see the following output from the Java program:

15:18:11.868 [NioProcessor-2] DEBUG org.red5.io.amf.Input - String: @setDataFrame
15:18:11.868 [NioProcessor-2] DEBUG org.red5.io.object.Deserializer - Datatype: String
15:18:11.868 [NioProcessor-2] DEBUG org.red5.io.amf.Input - Length: 10 limit: 171
15:18:11.868 [NioProcessor-2] DEBUG org.red5.io.amf.Input - String: onMetaData
15:18:11.868 [NioProcessor-2] WARN org.red5.server.net.rtmp.codec.RTMPProtocolDecoder - Failed to decodeBuffer: pos 0, limit 970, chunk size 128, buffer 040000000000ab120000000002000d40736574446174614672616d6502000a6f6e4d657461446174610800000006000d617564696f6461746172617465004085888000000000000f617564696f73616d706c65726174650040e5888000000000000f617564696f73616d706c6573697a65004030000000000000000673746572656f0100000c617564696f63c46f64656369640040080000000000000007656e636f64657202000d4c61766635392e31362e3130300000090400000000080108000000003e00000001ff01fd02f803ee04e005cc06b0078d0862092d0aee0aa40b4e0cec0c7c0dff0d750edb0e330f7b0fb40fdd0ff60fff0ff80fe10fba0f830f3c0fe60e820e0e0e8d0dfd0c610cb80b040b440a7a09a708ca07e606fc050b0515041a031e021e011e001dff1dfe20fd25fc2dfb3bfa4ff969f88bf7b5f6e9f527f570c4f4c5f326f393f20ff298f130f1d6f08cf052f027f00cf001f006f01cf041f076f0bbf00ff172f1e4f164f2f2f28df334f4e7f4a6f56ff641f71cf8fff8eaf9dafacffbc8fcc5fdc5fec5ffc600c601c402bf03b704aa0597067d075c083309000ac30a7b0b280cc90c5d0de30d5b0ec50e200f6c0fa90fd50ff20ffe0ffb0fe7c40fc40f910f4e0ffb0e9a0e2a0eab0d1f0d860ce00b2d0b700aa909d708fd071b07320642054d04540357025801580057ff57fe59fd5dfc65fb72fa84f99df8bdf7e6f617f653f599f4ebf349f3b3f22cf2b2f146f1e9f09cf05df02ff011f002f004f015f037f069f0aaf0fbf05bf1c9f146f2d0f268f30df4bef47af540f610c4f7e9f7cbf8b3f9a2fa97fb90fc8cfd8bfe8bff8c008c018b0287037f047305610649072a080209d209970a530b020ca60c3c0dc60d410eaf0e0d0f5c0f9c0fcc0fed0ffd0ffd0fed0fcd0f9e0f5e0f0f0fb10e440ec90d400da90c060c580b9d0ad709080930084f076706790585048d0391029201920091ff91fe92fd96fc9ec4fba9fabaf9d1f8eff716f745f67ff5c2f411f46cf3d4f249f2ccf15df1fdf0acf06af038f016f004f002f010f02ef05cf09af0e7f043f1aff128f2b0f244f3e6f394f44ef512f6e0f6b7f797f87ef96cfa5ffb57fc53fd51fe51ff5200510150024d0347043c052b061507f807d208a4096c0a290bdb0b820c1b0da80d260e97c40ef90e4c0f8f0fc30fe70ffb0fff0ff20fd60faa0f6e0f220fc80e5e0ee70d610dcd0c2d0c800bc80a050a3809620883079c06b005bd04c503ca02cd01cd00ccffccfeccfdd0fcd6fbe0faeff905f922f846f774f6aaf5ecf438f491f3f6f268f2e7f175f112f1bdf078f042f01df007f001f00bf026f050f08bf0d4f02df195c4
15:18:11.870 [NioProcessor-2] WARN org.red5.server.net.rtmp.codec.RTMPProtocolDecoder - Closing connection because decoding failed: RTMPMinaConnection 0:0:0:0:0:0:0:1:1935 to null client: null session: VM1VKW8XESSKU state: connected
org.red5.server.net.protocol.ProtocolException: Error during decoding
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.decode(RTMPProtocolDecoder.java:195)
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.decodeBuffer(RTMPProtocolDecoder.java:120)
	at org.red5.client.net.rtmp.codec.RTMPMinaCodecFactory$1.decode(RTMPMinaCodecFactory.java:58)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:254)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
	at org.red5.client.net.rtmpe.RTMPEIoFilter.messageReceived(RTMPEIoFilter.java:79)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1224)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1213)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap')
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.readTransactionId(RTMPProtocolDecoder.java:785)
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.decodeAction(RTMPProtocolDecoder.java:761)
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.decodeMessage(RTMPProtocolDecoder.java:528)
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.decodePacket(RTMPProtocolDecoder.java:308)
	at org.red5.server.net.rtmp.codec.RTMPProtocolDecoder.decode(RTMPProtocolDecoder.java:181)
	... 22 common frames omitted

This is the FFmpeg version I'm using:

$ ffmpeg -version
ffmpeg version 5.0 Copyright (c) 2000-2022 the FFmpeg developers
built with Apple clang version 13.0.0 (clang-1300.0.29.30)
configuration: --prefix=/usr/local/Cellar/ffmpeg/5.0 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
libavutil      57. 17.100 / 57. 17.100
libavcodec     59. 18.100 / 59. 18.100
libavformat    59. 16.100 / 59. 16.100
libavdevice    59.  4.100 / 59.  4.100
libavfilter     8. 24.100 /  8. 24.100
libswscale      6.  4.100 /  6.  4.100
libswresample   4.  3.100 /  4.  3.100
libpostproc    56.  3.100 / 56.  3.100