Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Support]: Record stream doesn't recover after camera restart #3122

Closed
kspearrin opened this issue Apr 19, 2022 · 22 comments
Closed

[Support]: Record stream doesn't recover after camera restart #3122

kspearrin opened this issue Apr 19, 2022 · 22 comments

Comments

@kspearrin
Copy link

Describe the problem you are having

This seems related to #1713

I have 10 Amcrest/Dahua turret cameras running on Frigate (see config below).

If a camera is restarted, Frigate will recover the event and live stream, however recordings stop and never recover until I restart Frigate.

I can simulate this by just manually restarting the camera.

Version

0.10.1-83481af

Frigate config file

database:
  path: /db/frigate.db

mqtt:
  host: *************
  user: *************
  password: *************

detectors:
  coral1:
    type: edgetpu
    device: pci:0

record:
  enabled: True
  retain:
    days: 14
    mode: all
  events:
    retain:
      default: 14

detect:
  width: 1280
  height: 720

model:
  labelmap:
    15: animal
    16: animal
    17: animal
    18: animal
    19: animal
    20: animal
    21: animal
    22: animal
    23: animal
    24: animal

objects:
  track:
    - person
    - bicycle
    - car
    - motorcycle
    - animal

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
  input_args: -avoid_negative_ts make_zero -rtsp_transport tcp
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy

cameras:

  front_yard_ne:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    zones:
      front_yard_ne_driveway:
        coordinates: ..........

  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_door.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_door.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_door.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal
    zones:
      front_door_at_door:
        coordinates: ..........

  front_yard_drive:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_yard_drive.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_yard_drive.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_yard_drive.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    zones:
      front_yard_drive_driveway:
        coordinates: ..........

  garage:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@garage.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@garage.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@garage.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal

  front_yard_se:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect

  side_yard_se:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@side_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@side_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@side_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal

  back_yard_sw:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@back_yard_sw.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@back_yard_sw.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@back_yard_sw.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal
        - boat

  back_yard_nw:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@back_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@back_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@back_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal
        - boat

  side_yard_nw:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@side_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin:***************@side_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@side_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal

  boathouse:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
      inputs:
        - path: rtsp://admin::***************@@boathouse.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@boathouse.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
            - detect
    detect:
      width: 640
      height: 480
    live:
      height: 480
    objects:
      track:
        - person
        - animal
        - boat
    zones:
      boathouse_main_deck:
        coordinates: ..........

Relevant log output

[2022-04-18 21:22:18] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
Starting migrations
[2022-04-18 21:22:18] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-04-18 21:22:18] peewee_migrate                 INFO    : There is nothing to migrate
[2022-04-18 21:22:18] frigate.app                    INFO    : Output process started: 213
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_yard_ne: 215
[2022-04-18 21:22:18] ws4py                          INFO    : Using epoll
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_door: 218
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_yard_drive: 221
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for garage: 224
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_yard_se: 225
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for side_yard_se: 227
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for back_yard_sw: 229
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for back_yard_nw: 231
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for side_yard_nw: 232
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for boathouse: 234
[2022-04-18 21:22:18] frigate.app                    INFO    : Capture process started for front_yard_ne: 236
[2022-04-18 21:22:18] frigate.app                    INFO    : Capture process started for front_door: 238
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for front_yard_drive: 241
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for garage: 250
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for front_yard_se: 255
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for side_yard_se: 259
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for back_yard_sw: 268
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for back_yard_nw: 274
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for side_yard_nw: 285
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for boathouse: 297
[2022-04-18 21:22:19] ws4py                          INFO    : Using epoll
W driver/beagle/beagle_kernel_top_level_handler.cc:131] Could not set performance expectation : 92 (Inappropriate ioctl for device)
[2022-04-18 21:22:18] detector.coral1                INFO    : Starting detection process: 212
[2022-04-18 21:22:18] frigate.edgetpu                INFO    : Attempting to load TPU as pci:0
[2022-04-18 21:22:19] frigate.edgetpu                INFO    : TPU found
[2022-04-18 21:22:37] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58174]



(I restarted camera front_yard_ne at this point)



[2022-04-18 21:23:59] watchdog.front_yard_ne         INFO    : No frames received from front_yard_ne in 20 seconds. Exiting ffmpeg...
[2022-04-18 21:23:59] watchdog.front_yard_ne         INFO    : Waiting for ffmpeg to exit gracefully...
[2022-04-18 21:24:29] watchdog.front_yard_ne         INFO    : FFmpeg didnt exit. Force killing...
[2022-04-18 21:24:29] frigate.video                  ERROR   : front_yard_ne: Unable to read frames from ffmpeg process.
[2022-04-18 21:24:29] frigate.video                  ERROR   : front_yard_ne: Unable to read frames from ffmpeg process.
[2022-04-18 21:24:29] frigate.video                  ERROR   : front_yard_ne: ffmpeg process is not running. exiting capture thread...
[2022-04-18 21:24:29] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58174]
[2022-04-18 21:24:39] watchdog.front_yard_ne         ERROR   : Ffmpeg process crashed unexpectedly for front_yard_ne.
[2022-04-18 21:24:39] watchdog.front_yard_ne         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

FFprobe output from your camera

$ docker exec frigate ffprobe 'rtsp://admin:******************@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=0'
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 33 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 33 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 22 packets
[h264 @ 0x55ab6e569780] concealing 28713 DC, 28713 AC, 28713 MV errors in I frame
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 25 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 17 packets
[h264 @ 0x55ab6e569780] error while decoding MB 63 54, bytestream -5
[h264 @ 0x55ab6e569780] concealing 19426 DC, 19426 AC, 19426 MV errors in P frame
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 14 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 23 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 23 packets
Input #0, rtsp, from 'rtsp://admin:******************@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.001000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 3840x2160 [SAR 7713:7712 DAR 857:482], 15 fps, 100 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: aac, 16000 Hz, mono, fltp

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Coral version

M.2

Network connection

Wired

Camera make and model

Amcrest IP8M-T2499EB-28MM

Any other information that may be helpful

Same issue seems to happen with all 10 of my cameras.

@NickM-27
Copy link
Sponsor Collaborator

When you say recording stop, what does this mean?

What symptoms are you seeing that show recordings are not working?

@kspearrin
Copy link
Author

kspearrin commented Apr 19, 2022

When you say recording stop, what does this mean?

What symptoms are you seeing that show recordings are not working?

When I try to view events for that camera, I get errors like:

image

I assume this is because there is no recordings file for that timeframe.

I then look in the recordings directory and observe that no files are being created anymore for that camera. When I reboot Frigate, recordings files start showing up again every 10 seconds.

@NickM-27
Copy link
Sponsor Collaborator

Do you see new recordings files in the container's /tmp/cache for that camera?

@kspearrin
Copy link
Author

kspearrin commented Apr 19, 2022

Do you see new recordings files in the container's /tmp/cache for that camera?

I observe that /tmp/cache has files being created and removed while the camera is operating normally. Whenever I restart the camera and the problem described above starts, the /tmp/cache directory has the last residual file that was created before the camera restarted and no new files are being created or deleted for that camera any longer. When I reboot Frigate, that residual file gets cleaned up and new ones start being created again, as expected.

@hawkeye217
Copy link
Collaborator

hawkeye217 commented Apr 19, 2022

I notice the same thing with two of my cameras as well. Over the past few days I have been changing settings on my router and rebooting it. Frigate never records anything for those two particular cameras after the router reboots unless I restart Frigate as well.

Detect streams are still working - it's just recording that I have a problem with.

I may dig more into this later this week if there's no immediate solution.

@s-cuba
Copy link

s-cuba commented Apr 27, 2022

Im having the same issues.

Im using 8 reolinks via rmtp streams and other than this and timestamps not working everything seems fine.

Whenever a camera leave the network, then rejoins frigate will not pick the stream back up.

Log
[2022-04-27 14:21:43] watchdog.reo_2 INFO : No frames received from reo_2 in 20 seconds. Exiting ffmpeg...
[2022-04-27 14:21:43] watchdog.reo_2 INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-27 14:22:13] watchdog.reo_2 INFO : FFmpeg didnt exit. Force killing...
[2022-04-27 14:22:13] frigate.video ERROR : reo_2: Unable to read frames from ffmpeg process.
[2022-04-27 14:22:13] frigate.video ERROR : reo_2: ffmpeg process is not running. exiting capture thread...
[2022-04-27 14:22:23] watchdog.reo_2 ERROR : Ffmpeg process crashed unexpectedly for reo_2.
[2022-04-27 14:22:23] watchdog.reo_2 ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

Nothing is being recorded after it leaves. In the camera recodings tab it shows no times past the disconnect. It will for some reason still catch events, never record, just save a snapshot. When trying to view recording of said snapshot, the ui says network failure or incorrect format. Logs show

[2022-04-27 14:24:47] frigate.http ERROR : No recordings found for the requested time range

@s-cuba
Copy link

s-cuba commented Apr 28, 2022

Well I have seemed to solve it, not sure what did it i changed some things in config, removed global args for ffmpeg as they were the same as default.

Heres my current config

#####
mqtt:
  host: mosquitto_frigate
  topic_prefix: xxx
  client_id: xxx
  user: xxx
  password: xxx

####
ffmpeg:
  input_args:
    - -avoid_negative_ts
    - make_zero
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -rw_timeout
    - '5000000'
    - -use_wallclock_as_timestamps
    - "1"
    - -f
    - live_flv
###
detect:
  width: 640
  height: 480
  fps: 5
  enabled: True

###
objects:
  track:
    - person
    - car
    - bicycle
    - bus
    - cat
    - dog

###
record:
  enabled: True

  retain:
    days: 14
    mode: all
  events:
    retain:
      default: 14
      mode: active_objects

###
snapshots:
  enabled: True
  retain:
    default: 14

###
timestamp_style:
  position: "tl"
  format: "%m/%d/%Y %H:%M:%S"

####
cameras:

#1
  reo_0:
    ffmpeg:
      inputs:
        - path: rtmp://10.10.20.50/bcs/channel0_sub.bcs?channel=0&stream=0&user=xxx&password=xxx
          roles:
            - detect
            - rtmp
        - path: rtmp://10.10.20.50/bcs/channel0_main.bcs?channel=0&stream=0&user=xxx&password=xxx
          roles:
            - record

Im going to do some more testing to be sure its solved. All logs show the exact same info when a disconnect happens, but now never show these lines which show up in both of our error logs

[2022-04-18 21:23:59] watchdog.front_yard_ne INFO : No frames received from front_yard_ne in 20 seconds. Exiting ffmpeg...
[2022-04-18 21:23:59] watchdog.front_yard_ne INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-18 21:24:29] watchdog.front_yard_ne INFO : FFmpeg didnt exit. Force killing...

So it seems as if frigate trying to kill ffmpeg causes the issue

@stale
Copy link

stale bot commented Jun 1, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jun 1, 2022
@kspearrin
Copy link
Author

Still an issue.

@stale stale bot removed the stale label Jun 1, 2022
@stale
Copy link

stale bot commented Jul 1, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jul 1, 2022
@kspearrin
Copy link
Author

Still an issue

@stale stale bot removed the stale label Jul 1, 2022
@mikegleasonjr
Copy link

mikegleasonjr commented Jul 2, 2022

When I was recording my RTSP streams manually with ffmpeg I had to add the option -stimeout 30000000 for ffmpeg to timeout whenever my camera/router was restarted. So I think it's the behavior of ffmpeg not Frigate. Otherwise ffmpeg would just hang without returning and without recording anything...

@salleq
Copy link

salleq commented Aug 2, 2022

My workaround for this (having Reolink cameras that fail sometimes) is making Frigate be restarted on error.
Basically I'm reading Home Assistan event log for:
homeassistant.components.stream.stream.camera.CAMERANAME

and if that is found I send MQTT message to restart frigate.

I lose few seconds of reconrding whenever this happens but it's way better than to notice it many days later that a camera hasn't been recording (again).

@kspearrin
Copy link
Author

My workaround for this (having Reolink cameras that fail sometimes) is making Frigate be restarted on error. Basically I'm reading Home Assistan event log for: homeassistant.components.stream.stream.camera.CAMERANAME

and if that is found I send MQTT message to restart frigate.

I lose few seconds of reconrding whenever this happens but it's way better than to notice it many days later that a camera hasn't been recording (again).

Are you doing this with a automation script of some sort or another way? Could you share more details?

@salleq
Copy link

salleq commented Aug 2, 2022

Are you doing this with a automation script of some sort or another way? Could you share more details?

I do it with Node-Red. I believe it's doable with HA automations as well but I can't immediately say how to do that.
I linked my flow here, it's not too complicated: https://pastebin.com/rEwcUvez

@phedders
Copy link

phedders commented Nov 1, 2022

It would be really nice if Frigate could restart streams after they fail....

@blakeblackshear
Copy link
Owner

Frigate does restart streams when they fail. Pretty sure this is happening because the default input args were modified to remove the -timeout parameter.

@phedders
Copy link

phedders commented Nov 2, 2022

Thankyou! I have input_args set - and wasnt aware of the defaults being overwritten... I'll poke it a bit without setting to see what I am missing.

@kspearrin
Copy link
Author

@phedders Please let me know what you find. I was unable to get my streams working with the timeout input arg.

@phedders
Copy link

phedders commented Nov 2, 2022

-rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1

Was missing from the default options - so I have added that to my input_args and will see if it is more stable. Cheers

@bagobones
Copy link

bagobones commented Nov 14, 2022

Once or twice now I have found ALL of my camera feeds unresponsive old snapshot and no feed when opening the camera in the frigate web UI only two of my 6 cameras have custom input lines and the two that do have custom input still have the timeout.

In one of these cases I was able to restart frigate via the web UI restart, the other time I had to restart the docker..

I will try to capture logs next time but the WebUI it self still loaded. It was just the camera feeds being dead and the restart option did nothing in the one case.

@NickM-27
Copy link
Sponsor Collaborator

Fixed in 0.12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants