Server crashing - panic: runtime error: index out of range [0] with length 0

[convicte]

Hi all,

My apologies if this issue falls under any of the existing tickets, but I was not able to identify similarities or lack there of.

My setup was working flawlessly and was providing nearly no latency with six 4K streams, loading immediate on a single card.
Unfortunately, recently the server start to crash intermittently.
Since I updated to HA 2022.8.X, did a Mikrotik update to fw v.7.1.X, as well as played with a lot of different things over the last weeks, it's really difficult to narrow what exactly changed in my setup.

The error I am getting is the following:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-rtsp-to-web: executing...
[01:20:50] INFO: Updating configuration
[cont-init.d] 01-rtsp-to-web: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[01:20:50] INFO: Starting discovery service
[01:20:50] INFO: Starting RSTPtoWeb
time="2022-08-17T01:20:50+02:00" level=info msg="Server CORE start" func=main module=main
time="2022-08-17T01:20:50+02:00" level=info msg="Server HTTP start" call=Start func=RTSPServer module=http_server
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.stairs
time="2022-08-17T01:20:50+02:00" level=info msg="Server start success a wait signals" func=main module=main
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.gate
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.garden
time="2022-08-17T01:20:50+02:00" level=info msg="Server RTSP start" call=Start func=RTSPServer module=rtsp_server
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.taras_2
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.vto
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.backyard
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.vto_main
time="2022-08-17T01:20:50+02:00" level=info msg="Run stream" call=Run channel=0 func=StreamServerRunStreamDo module=core stream=camera.garden_ptz
[01:20:50] INFO: Successfully send discovery information to Home Assistant (8083).
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.taras_2
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.stairs
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.garden
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.vto
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.backyard
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.gate
time="2022-08-17T01:20:51+02:00" level=info msg="Success connection RTSP" call=Start channel=0 func=StreamServerRunStream module=core stream=camera.garden_ptz
time="2022-08-17T01:20:51+02:00" level=error msg="Stream error restart stream Camera send statusRTSP/1.0 404 Not Found" call=Restart
panic: runtime error: index out of range [0] with length 0
goroutine 37 [running]:
main.StreamServerRunStreamRTMP(0xc00040b4c0, 0xa, 0x12dd5e0, 0x1, 0xc0000beb00, 0x0, 0x0, 0x0)
/workspace/streamCore.go:265 +0x1291
main.StreamServerRunStream(0xc00040b4c0, 0xa, 0x12dd5e0, 0x1, 0xc0000beb00, 0x0, 0x0, 0x0)
/workspace/streamCore.go:67 +0x14cb
main.StreamServerRunStreamDo(0xc00040b4c0, 0xa, 0x12dd5e0, 0x1)
/workspace/streamCore.go:47 +0x4fa
created by main.(*StorageST).StreamChannelRunAll
/workspace/storageStreamChannel.go:42 +0x18a
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

I am running the HA_OS in a VirtualBox VM, with 6 cores passed onto the machin, and 8GB of RAM.

Any assistance would be greatly appreciated.
If I am missing any pertinent information, please let me know.

Cheers!

[deepch]

you use rtmp stream? not rtsp? change rtmp:// to rtsp://

[convicte]

Thanks for the prompt response.

I am sorry if I am not understanding the issue, but I am using RTSP streams to bring the cameras into HA through Frigate.

To make sure we are on the same page—the add-on often starts and runs for half a day or even two days, but then crushes with the same error, unexpectedly.

I've not made any changes in Frigate configuration prior to the issues occurring.

Could you point me to where I would be able to find any setting for rtsp/rtmp being used in your add-on?
I can't see any indication of rtmp in the camera settings beyond Frigate streaming RTMP, if need be.

[deepch]

strings.ToLower(url.Scheme) == "rtmp"
---->main.StreamServerRunStreamRTMP
fix you url check all stream

close port 1935

[deepch]

and update last version source

[convicte]

@deepch, I understand it may be frustrating to deal with someone that doesn't follow, but I really don't understand what you mean:

1. "fix you url check all stream" - should I disable the additional RTMP streams in Frigate, because they are causing your add-on some issues? I really don't understand which stream URL you are referring to, if the only URL I use in Frigate is RTSP.
   BTW, it didn't, until now.

2. I do not see any updates available - could you please be more specific in your other request - update last version source
   Just in the meantime I removed the add-on and HA integration and reinstalled it anew - current version is 1.2.2.
   This did not resolve the issue, btw.

Thank you and please accept my apologies for not catching as quickly as you probably would hope.

[deepch]

ok im test it.

[deepch]

send me all stream's list

[convicte]

Here is the appropriate part of the Frigate config.

cameras:
  Gate:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.142:554//h265Preview_01_main
          roles:
            - record
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.142:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 896,720,1280,720,1280,681,893,684
        - 239,0,358,0,352,31,348,61,324,96,278,94,244,87

  Garden:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.159:554//h265Preview_01_main
          roles:
            - record
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.159:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 892,720,1280,720,1280,683,891,683
        - 580,40,240,61,146,71,112,0,65,63,0,150,313,125,715,119,1280,128,1280,37,696,34

  Taras:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.134:554//h265Preview_01_main
          roles:
            - record
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.134:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 899,720,1280,720,1280,667,894,668
        - 215,237,0,348,0,248,187,162,384,84,399,142
        - 0,133,159,73,136,0,0,56

  Backyard:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.148:554//h265Preview_01_main
          roles:
            - record
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.148:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 887,720,1280,720,1280,682,1094,682,890,684
        - 0,0,211,0,637,0,576,24,539,34,506,44,470,53,447,79,429,112,392,147,344,162,277,158,227,154,77,123,0,90

  Stairs:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.144:554//h265Preview_01_main
          roles:
            - record
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.144:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 897,720,1280,720,1280,665,895,667
        - 898,198,905,0,825,0,739,0,676,0,591,0,432,0,413,38,390,71,380,119,366,204,430,319,477,501,722,511,902,500
        - 51,326,37,502,0,529,0,293,0,84,92,106

  VTO:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:vto123jud@XXX.XXX.XXX.6/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://ZZZZZZZ:vto123jud@XXX.XXX.XXX.6/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 0,720,529,720,531,664,231,659,0,660
        - 617,0,1280,0,1280,332,1149,350,819,348,511,355,293,323,226,312,171,301,167,233,243,127
        - 0,0,122,0,124,91,127,202,139,236,92,327,0,344


  Garden_PTZ:
    ffmpeg:
      inputs:
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.114:554//h265Preview_01_main
          roles:
            - record
        - path: rtsp://ZZZZZZZ:XXXXXXXXX@XXX.XXX.XXX.114:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 890,720,1280,720,1280,671,1095,668,886,671
        - 294,0,950,0,935,62,823,103,638,123,419,154,316,130
    zones:
      Balcony:
        coordinates: 0,0,0,720,480,720,1166,720,748,347,396,430,329,171,125,0

[convicte]

Here are all currently streaming entities I have:

I've had a duplication in the VTO camera (since it's integrated via RTSP and a separate integration for Dahua (to capture button presses). It's disabled now.

[allenporter]

I think you sent your frigate config and not the rtsp to web config.

(My impression is frigate creates rtmp urls for consumption by others)

[convicte]

I think you sent your frigate config and not the rtsp to web config.

(My impression is frigate creates rtmp urls for consumption by others)

Unless I am being really thick, there is no RTSPtoWeb config.

The whole installation process is:

1. Install the addon from Git
2. Go to integrations and press on the RTSPtoWeb.

As long as your cameras are integrated already, that is the extent of it.
Honestly, it used to work just fine until maybe a week ago, some something have gone bad.

[convicte]

The issue continues daily. The server runs for a few hours, then crashes and won't start, just to randomly start again and work for 2-3 hours, etc.

[convicte]

@deepch is there anything else I can do to aid in the investigation?

Thank you again and in advance for all the help!!

[mrdmiller2]

I think my problem is the same as yours.. (crashing after a few seconds).. with the same or similar error.