[BUG] Restart a stream doesn't work anymore in OBS Studio

[DJ-PD]

Platform

Operating system and version: Windows 10 v1909 64bit
OBS Studio version: v25.0.8

Expected Behavior

Start Streaming = start the stream (it's ok and works today)
Stop Streaming = stop the stream (it's ok and works today)
Start Streaming again after a stop = shall start the stream as task 1 as with OBS Studio v24.0.3

Current Behavior

Start Streaming = start the stream (it's ok and works today)
Stop Streaming = stop the stream (it's ok and works today)
Start Streaming again after a stop = stream doesn't start, control button "Start/Stop Streaming" is totally grey and I can't do any thing with this button. Exit OBS studio works but the program still runs in background and process task need to be ended via Windows task manager. Thereafter can OBS Studio be started and used again until next stop.

OBS Studio v25.0.8 part of log file when issue occur

11:29:13.744: ---------------------------------
11:29:13.744: [FFmpeg aac encoder: 'avc_aac_stream'] bitrate: 320, channels: 2, channel_layout: 3
11:29:13.744:
11:29:13.746: [rtmp stream: 'adv_stream'] Connecting to RTMP URL rtmp://192.168.0.174/live/...
11:29:13.750: [rtmp stream: 'adv_stream'] Interface: Intel(R) 82574L Gigabit Network Connection (ethernet, 1000 mbps)
11:29:16.507: [rtmp stream: 'adv_stream'] Connection to rtmp://192.168.0.174/live/ successful
11:29:16.514: ==== Streaming Start ===============================================
11:29:52.048: [rtmp stream: 'adv_stream'] User stopped the stream
11:29:52.048: Output 'adv_stream': stopping
11:29:52.048: Output 'adv_stream': Total frames output: 880
11:29:52.048: Output 'adv_stream': Total drawn frames: 957
11:29:52.053: ==== Streaming Stop ================================================
11:29:52.116: warning: 2 frames left in the queue on closing
11:32:28.898: [jim-nvenc: 'streaming_h264'] settings:
11:32:28.898: rate_control: VBR
11:32:28.898: bitrate: 2500
11:32:28.898: cqp: 20
11:32:28.898: keyint: 125
11:32:28.898: preset: llhq
11:32:28.898: profile: high
11:32:28.898: width: 1920
11:32:28.898: height: 1080
11:32:28.898: 2-pass: false
11:32:28.898: b-frames: 2
11:32:28.898: lookahead: false
11:32:28.898: psycho_aq: true
11:32:28.898:
11:32:28.920: ---------------------------------
11:32:28.921: [FFmpeg aac encoder: 'avc_aac_stream'] bitrate: 320, channels: 2, channel_layout: 3
11:32:28.921:
11:32:28.922: [rtmp stream: 'adv_stream'] Connecting to RTMP URL rtmp://192.168.0.174/live/...
11:32:28.923: [rtmp stream: 'adv_stream'] Interface: Intel(R) 82574L Gigabit Network Connection (ethernet, 1000 mbps)
11:32:28.927: HandleInvoke, error decoding invoke packet

Steps to Reproduce

Start Streaming = start the stream
Stop Streaming = stop the stream
Start Streaming again after a stop = stream doesn't start

Additional information

Tested both v24.0.3 and v25.0.8 / uninstalled and installed 3 times and totally left the server side.
Version 24.0.3 works as a charm every time but 25.0.8 fails on the second "Start Streaming" all the time.
I refer to closed case #2849 and ask for a investigation before closing this new support ticket.

[notr1ch]

What RTMP server are you streaming to?

[DJ-PD]

MistServer

[armpdq]

i can confirm, same issue with flussonic rtmp server

[govz]

The same issue is happening with Flussonic Media Servers, the exact same issue.
And I got the same in logs

21:30:54.208: [rtmp stream: 'adv_stream'] Connecting to RTMP URL rtmp://xx-xx/static...
21:30:54.444: HandleInvoke, error decoding invoke packet

[chgordonchoi]

I can confirm such issue also occur when using Castr multistreaming service. The error log also ends at

HandleInvoke, error decoding invoke packet
before manually shutting down.

In our case, we have the same issue since v25.0 and all the way up to v25.0.8

[ZloyDyadka]

Same issue with Flussonic media server
Additional: https://obsproject.com/forum/threads/obs-multiplatform-refuses-to-start-stream-by-2nd-try.36394/

[eric]

I'm running into problems on v25.0.8 properly reconnecting to an nginx-rtmp-module streaming server. It sometimes will reconnect but other times will not. I am using rtmp:// and not rtmps://.

[Thulinma]

Hi there! MistServer dev here.
I've found the reason for this bug: the librtmp state isn't being correctly reset on reconnect.
This causes the connection to be started with an incorrect max chunk size (whatever was in use when the last disconnect happened, and not the 128 byte RTMP default), and thus fails to parse multi-part chunks.
Most servers set a new max chunk size only right after sending the _result response to the connect call, since some RTMP implementations fail if you do so before responding to the connect call.

In short: please reset the librtmp internal state in between reconnect attempts, as fixing this on the media server side will break other client implementations instead. 🙁

[Banderi]

Oh, I've been trying to find the culprit to this for ages... Google served me no luck and yet it was reported here all along! I'm glad the cause was narrowed down and I wasn't just going crazy. Hopefully it's an easy fix!

[eric]

I found that my issue was not this problem but was caused by #3078.

[LosTigeros]

I've tested it on master and it still can't connect after one disconnect.

[rtmp stream: 'simple_stream'] Connecting to RTMP URL rtmp://stream.tokyo.[redacted]...
[rtmp stream: 'simple_stream'] Interface: Intel(R) PRO/1000 MT Desktop Adapter (ethernet, 1000 mbps)
[rtmp stream: 'simple_stream'] Connection to rtmp://stream.tokyo.[redacted] successful
obs_encoder_audio: Null 'encoder' parameter
==== Streaming Start ===============================================
[rtmp stream: 'simple_stream'] User stopped the stream
Output 'simple_stream': stopping
Output 'simple_stream': Total frames output: 311
Output 'simple_stream': Total drawn frames: 404
==== Streaming Stop ================================================
[rtmp stream: 'simple_stream'] Connecting to RTMP URL rtmp://stream.tokyo.[redacted]...
[rtmp stream: 'simple_stream'] Interface: Intel(R) PRO/1000 MT Desktop Adapter (ethernet, 1000 mbps)
RTMPSockBuf_Fill, remote host closed connection
[rtmp stream: 'simple_stream'] Connection to rtmp://stream.tokyo.[redacted] failed: -3
==== Streaming Stop ================================================

And in the rtmp server logs:

client connected '***.***.***.***'
too big message: 5242880, client: ***.***.***.***, server: 0.0.0.0:1935
disconnect, client: ***.***.***.***, server: 0.0.0.0:1935
deleteStream, client: ***.***.***.***, server: 0.0.0.0:1935

After increasing max_message to 10M on the server OBS just freezes while reconnecting.

[rtmp stream: 'simple_stream'] Connecting to RTMP URL rtmp://stream.tokyo.[redacted]...
[rtmp stream: 'simple_stream'] Interface: Intel(R) PRO/1000 MT Desktop Adapter (ethernet, 1000 mbps)

And after connecting for the 2nd time it just get's stuck at connecting.