Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Screen Images Freezes after 30ish minutes #39

Closed
Nicarlo opened this issue Feb 2, 2020 · 10 comments
Closed

Screen Images Freezes after 30ish minutes #39

Nicarlo opened this issue Feb 2, 2020 · 10 comments

Comments

@Nicarlo
Copy link

Nicarlo commented Feb 2, 2020

Describe the bug
After about 30-45 minutes of runtime the images freeze up.
I checked htop and there is no resource exhaustion happening however I do see the omxplayer.bin rtsp running in the process list. If I check the network traffic via something like iftop or vnstat you can see that the traffic has dropped dramatically to the point where i feel comfortable saying that the traffic received is related to the typicaly idle noise you'd see.

Some things that I've tried to do to troubleshoot:

  1. Attempted to restart the NVR to see if it was related to the NVR
  2. Ran a single RTSP feed from the debug omxplayer mode as found in the READM.md
  3. Changed the power supply of the PI to ensure that it wasnt related to power.

To Reproduce
Steps to reproduce the behavior:

  1. Install per README.md
  2. Configure my feeds in layout.conf.default
  3. Run 'sudo systemctl restart displaycameras.service'
  4. Wait about 30-45 minutes until the images freeze
  5. Run 'sudo systemctl restart displaycameras.service' again to restart the sessions.

Expected behavior
The images should not freeze up

Raspberry Pi (please complete the following information):

  • Hardware model: Raspberry Pi 3 Model B
  • OS: Rasbian Buster Lite
  • Console: Text Console
  • gpu memory split: 256MB
  • displaycameras Version 0.8.3.3

**Display Configuration

  • Display/Monitor resolution: 1440 x 900
  • Window matrix: 4x4
  • Feed resolution: 704 x 480, Compression H.264H, 15 FPS, 256/Kb/S Bit Rate, Bit Rate Type CBR

Camera information (please complete the following information):

  • Number of cameras: 4
  • Camera models (if not Ubiquiti): Dalhua 2 x 4K cameras and 2 x 4MP Cameras
  • Managed

Features enabled (please complete the following information):

  • Rotation: no
  • Display Blanking: no
  • Display Detection: no

NVR information (please complete the following information):
@Anonymousdog
Copy link
Owner

Anonymousdog commented Feb 3, 2020 via email

@Nicarlo
Copy link
Author

Nicarlo commented Feb 4, 2020

Hey Andy,

thanks for reaching out

Here's the layout.conf.default

# This is the camera feed/windows layout configuration file for the
# displaycameras service.  It ONLY configures the layout and feeds for
# the cameras; the rest of the configuration is in displaycameras.conf.
# See the comments in that file for notes on configuring the below.

# This example defines seven 1/2-HD windows, three of which are off-screen to the right,
# through which the service rotates six camera feeds (it actually uses only six windows)
# on a full-HD monitor.  If this suites your needs, modify only the camera names to taste
# and feed URLs to what your cameras or NVR provides.

# Window names

# 2x2 screen with 3 off-screen windows
windows=(upper_left upper_right lower_left lower_right)
# Make sure to account for each window above in the list below.

# Windows positions

window_positions=(
#First Row
#upper_left
# 960x540
"0 0 719 449" \
#upper_right
# 960x540
"720 0 1439 449" \

#Second Row (missing all but the far right window because large_left is double size
#lower_left
# 960x540
"0 450 719 899" \
#lower_right
# 960x540
"720 450  1439 899" \

#off-screen
# 960x540 window just off-screen to the right
"1920 0 2879 539" \
# 960x540 window just below the other
"1920 540 2879 1079" \
# 960x540 window just off-screen to the left
"2880 0 3839 539" \
)

# Camera Names

camera_names=(NE SE South SW )
# Make sure to account for each camera above in the list of feeds below.

# Camera Feeds

camera_feeds=( \
# Mid-Res if your RPi can handle the load
# "rtsp://xxx.xxx.xxx.xxx/yyyyy_1" \
# Low-Res otherwise
# "rtsp://xxx.xxx.xxx.xxx/yyyyy_2" \
#Front Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1" \
#Driveway
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1" \
#Side Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1" \
#Street View
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1" \
#West
"<Camera stream URL>" \
#Vestibule
"<Camera stream URL>" \
)

# Are we rotating cameras through the window matrix? (default false if not set here)
# rotate="true"

output of the sudo /usr/bin/displaycameras status

pi@raspberrypi:~ $ sudo /usr/bin/displaycameras status
NE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1
at 15485sec
SE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1
at 15726sec
South is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1
at 15720sec
SW is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1
at 15713sec

@chienlaid
Copy link

chienlaid commented Feb 4, 2020 •
edited
Loading

I'm having the same problem, it freezes every 2 hours. My setup is exacly the same as Nicarlo. Using a LOREX NVR with same type of rtsp stream. If I do a restart everything is good for another couple hours. Stream is at low res H.264 352x240 512kbps....i've tried multiple resolutions with the same result.

@Anonymousdog
Copy link
Owner

Anonymousdog commented Feb 5, 2020 via email

@Nicarlo
Copy link
Author

Nicarlo commented Feb 6, 2020

@Anonymousdog

the sudo /usr/bin/displaycameras status command returned results that were being incremented every time i would run the command.

After reading your post I started instead focusing my attention on omxplayer. After making changes to the resolution of the substream on the NVR, the feeds started playing again without me having to restart the session.

@chienlaid try playing around with the resolution on your lorex NVR and let us know if this also fixes your issue

@Nicarlo
Copy link
Author

Nicarlo commented Feb 7, 2020

A little update on this.

I must have spoken to soon because the issue started happening again but this time after a longer time.

I did ran the following command directly to get a log output of the omxplayer instance for a single feed.
sudo omxplayer --no-keys --fps 30 --refresh --video_queue 4 --avdict rtsp_transport:tcp "rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=2" --live -n -1 --timeout 0 -g

As @Anonymousdog mentioned his script is only a wrapper to make it easier to manage the omxplayer so I am convinced this is an issue with the omxplayer.

For those interested here is the last few lines of the log output before the feed freezes

23:22:23 T:18446744071756935099   DEBUG: Normal M:7352767473 (A:0 V:47459890844) P:0 A:-7352.77 V:40107.12/T:0.70 (0,0,1,1) A:0% V:99% (0.00,0.00)
23:22:23 T:18446744071756935183   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:0
23:22:23 T:18446744071756935226   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:1
23:22:23 T:18446744071756935609   DEBUG: Live: 40109.27 (40107.12) S:1.010 T:0.70
23:22:23 T:18446744071756956109   DEBUG: Normal M:7352788690 (A:0 V:47459890844) P:0 A:-7352.79 V:40107.10/T:0.70 (0,0,1,1) A:0% V:99% (0.00,0.00)
23:22:23 T:18446744071756956193   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:0
23:22:23 T:18446744071756956237   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:1
23:22:23 T:18446744071756956617   DEBUG: Live: 40109.25 (40107.10) S:1.010 T:0.70
23:22:23 T:18446744071756966868   DEBUG: OMXClock::OMXStop
23:22:23 T:18446744071756978203   DEBUG: OMXThread::Run - Exited thread with  id -1389367616
23:22:23 T:18446744071756978746   DEBUG: OMXThread::StopThread - Thread stopped
23:22:23 T:18446744071756990474   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_scheduler handle 0xac900b40
23:22:23 T:18446744071757010371   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_decode handle 0xb4e860
23:22:23 T:18446744071757012491   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_render handle 0xac900688
23:22:23 T:18446744071757013160   ERROR: COMXPlayer::interrupt_cb - Timed out
23:22:23 T:18446744071757020050   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.clock handle 0xb5a090

Additionally I wanted to rule out that it wasnt an issue with my NVR so I initiated a stream from another computer using VLC and left it running overnight. When I returned the stream was still working properly which furthers my thinking that its an issue with omxplayer.

Lastly, I had a raspberry pi 4 4GB version laying around so I loaded that unit with raspbian lite and went through the installation of the displaycamera script and copied the old layout.default configuration to this unit. After some time, I still continue get a frozen feed.

some poking around on the omxplayer github issue page I've come across quite a few threads with people reporting similar issues.

I've listed some of them below but have not yet tried them all

popcornmix/omxplayer#555

https://stackoverflow.com/questions/37053495/omxplayer-freezes-when-playing-video

I will continue troubleshooting this issue and will let you know what I do find.

@Nicarlo
Copy link
Author

Nicarlo commented Feb 8, 2020

Alright another update on this.

The current maintainer of omxplayer is sayingthat omxplayer will be depreciated over vlc
see popcornmix/omxplayer#752 (comment)

Now I did try running the same rtsp stream over vlc and I had no issues with it.
@Anonymousdog , any plans to do the same as you did for omxplayer with vlc?

@Anonymousdog
Copy link
Owner

Anonymousdog commented Feb 11, 2020 via email

@AnonWSU86
Copy link

@Nicarlo - Have you had any luck since tinkering with this back in Feb? I am experiencing the same issue you described above. The image freezes and it takes a restart of the service to get it back online.
sudo systemctl restart displaycameras.service

  • If I run a system status, it says the stream is running and the # of seconds continue to increase if I keep running the status

  • If I look at network activity, it still looks the PI is pulling the stream

  • If I try to repair the status it does NOT resolve the is.

  • If I kill omxplayer, and then run the repair, this does solve this issue - Is this the same thing as restarting the service?
    sudo killall omxplayer.bin
    sudo /usr/bin/displaycameras repair

  • I have tried this with a single camera, full-screen stream on a 800x480 screen.

  • I have also tried this with a 4 Camera (2x2) stream, one the same 800x480 screen.

I have yet to try tinkering with any of the timing settings, could those make a difference?

I currently have a cron job that just restarts the service every 30 minutes. But I still seem to catch the stream frozen from time to time.

@Anonymousdog - Thank you for taking the time to put this together. It is a very neat solution. I would be very interested if you do the same with VLC.

@Nicarlo
Copy link
Author

Nicarlo commented May 1, 2020

@AnonWSU86 I ended up settling for a single stream on vlc. Vlc does have the ability to do something similar however their documentation has gaps in it and no one appears to know how to get it working.

As @Anonymousdog mentioned this isn’t an issue with displaycameras but instead an issue with the omxplayer that it uses. I did check out their gitlab and omxplayer will eventually get depreciated for vlc so there are no plans for them to put out a fix for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Nicarlo @Anonymousdog @chienlaid @AnonWSU86