Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MotionEye / Motion / Cameras just stop working #1582

Closed
IAmOrion opened this issue Jul 12, 2018 · 66 comments
Closed

MotionEye / Motion / Cameras just stop working #1582

IAmOrion opened this issue Jul 12, 2018 · 66 comments

Comments

@IAmOrion
Copy link

Preliminary Docs

I confirm that I have read the CONTRIBUTING guide before opening this issue. (YES)

I confirm that I have read the FAQ before opening this issue. (YES)

motionEyeOS Version

I am running motionEyeOS version: (insert your version here, e.g. 20180314).

motionEye Version | 0.39.2
Motion Version | 4.1.1
OS Version | motionEyeOS 20180627

Board Model

RPi Zero W

Camera

MMAL

Network Connection

WiFi

Peripherals

Log Files

Very odd, but my cameras had been working fine for a while now, running and recording 24/7.

I updated the newest stable release (20180627) and now for some reason, everything "freezes". I can access the Web UI, and via SSH, but the camera image won't show, it's no longer recording and even running reboot in ssh does nothing (well, it just hangs so to speak) meaning I have to physically unplug and re plug in the device.

I currently have 2 devices - identical RPi Zero W with the same RPi Cameras. (As mentioned, they have been working fine for ages without trouble) ... this morning, as coincidental as it is, they BOTH stopped at 7:15am. BOTH!! What are the chances of them both stopping within 30 seconds of each other?

What logs would help in this instance?

@xadox-1st
Copy link

Same for me.

@ccrisan
Copy link
Collaborator

ccrisan commented Jul 12, 2018

Do you save media files remotely? Please attach all of your logs, since they all might contain useful information as to why your system hangs.

@xadox-1st
Copy link

Yes, I am storing the captures on SMB.
Will post logs later.

@IAmOrion
Copy link
Author

IAmOrion commented Jul 12, 2018

Do you save media files remotely? Please attach all of your logs, since they all might contain useful information as to why your system hangs.

@ccrisan Yes I do, I have a 3rd Raspberry Pi (Pi 3) running as a central hub with 4TB attached hard drive. I have NOT yet updated the hub. That's currently on motionEyeOS 20180602

Logs attached. I've prepended them cam1_ and cam2_ (and appended .log to messages file)

cam_1dmesg.log
cam1_boot.log
cam1_motion.log
cam1_motioneye.log
cam1_messages.log

cam2_boot.log
cam2_dmesg.log
cam2_messages.log
cam2_motion.log
cam2_motioneye.log

@xadox-1st
Copy link

xadox-1st commented Jul 12, 2018

Here are my logs:
boot.log
dmesg.log
messages.log
motion.log
motioneye.log

@tigernero79
Copy link

me with release 27062018 I had many problems, with release 28062018 and last pre release 12072018 everything is fine

@ccrisan
Copy link
Collaborator

ccrisan commented Jul 13, 2018

Guys, the problem here is that the SMB protocol version has recently changed. Some users are fine with the old version, while others are fine with the new one. I personally don't know what to do about it.

@IAmOrion
Copy link
Author

@ccrisan hmmm interesting. Did the logs indicate what actually happens? Does the SMB protocol go to sleep or "crash"? My 2 cameras lasted about 55 hours this time, before they both rebooted. At least this time they did reboot without getting stuck in limbo so that's something I guess.

Perhaps I should go back to the older version.

@matucker
Copy link

I'm having the same issue since updating to motionEyeOS 20180627
motioneye.log
messages.log

@tigernero79
Copy link

install release 20180712

@SealedJoy
Copy link

SealedJoy commented Jul 20, 2018

same problem here except only saving locally no smb used. Pi zero w, working fine before update to 20180627. Using a generic usb cam.

@ccrisan
Copy link
Collaborator

ccrisan commented Aug 1, 2018

Guys, the latest nightly-dev includes an option to manually choose SMB protocol version. Please test it and see if it fixes your problems. To upgrade to latest nightly-build, use fwupdate upgrade nightly-dev from command line, making sure that you have Prereleases enabled in Expert Settings.

@IAmOrion
Copy link
Author

IAmOrion commented Aug 7, 2018

Thanks, will give this a try and report back...

EDIT, @ccrisan - could you clarify please what was the "Old" version from previous stable release, and what is the "new version" in the latest stable release. I'm assuming the previous stable release was version 1.0 but only guessing

screenshot 2018-08-07 at 17 22 03

@VegasIOT
Copy link

VegasIOT commented Aug 8, 2018

I tried the nightly build and had the same issue, I'm going to try to fall back to 0602 on my pi zero W. I've literally replaced all the hardware. It just seems to die after a few days

@matucker
Copy link

I have the Samba server turned off and I'm storing the files locally. I did install the latest nightly and I'm still having the same issues on my Pi Zero W's. I didn't change the samba version because I'm storing locally and don't have the option to change samba versions unless I change it to a network share. Looking at my logs I still see smb mounts stopping and starting, but why would they if I have the samba service turned off?

motioneye.log
messages.log
motion.log

@ccrisan
Copy link
Collaborator

ccrisan commented Aug 21, 2018

@matucker your issue is then not related to SMB itself so you can safely ignore the SMB-related comments. Judging form your logs, lines like Aug 10 18:48:44 ZeroFive user.notice wifi: disconnected indicate that you have a connectivity issue.

@IAmOrion
Copy link
Author

IAmOrion commented Aug 27, 2018

I know it's been a while now, but just to report back and say no matter what SMB protocol I choose, I still have the same problem, it just seems to be random with it's timing.

Worth noting though - and this is what actually makes this a real problem for me, is that once it does freeze up so to speak - I can't reboot it remotely. It "hangs". Reboot and Reboot -f do absolutely nothing. There's no way to reboot it, I've tried stopping the motioneye services etc etc before rebooting but nope - still won't work. I have to go and pull the plug - which is a huge pita when it's supposed to be a remote cctv unit :(

I could live with rebooting it every so often and not be too concerned about the issue - if only the reboot command worked

@tigernero79
Copy link

Idem for me, command reboot ssh not reboot pi zero also gui web is blocked

@artesdev
Copy link

Same thing for me this morning. Suddenly, the GUI of motioneye show nothing, and can't login anymore. :-(

@surlymatt
Copy link

I have the Pi Zero W problem where it no longer shows up a camera.
Yes, I'm saving the files via a samba share and the motioneye.log is showing a cURL upload failed/Failure estabishing an SSH session.
I can reboot and it all comes good.
I'm thinking of setting up a cron job to check if the latest file on the smb is within 5 minutes of the current most image file, and if not, then just reboot the whole thing?
Good idea or not?

@quantum8
Copy link

quantum8 commented Nov 1, 2018

Not sure if I'm having a similar issue or not. I have two zero w set up as fast cameras and a 3b set as the server. Every few hours the second zero loses the camera connection. I can still log in to reboot it to get the image back. The first zero hasn't skipped a beat.

Logs.zip

@tigernero79
Copy link

with the release 20181008 it seems that the problem has been solved at least for me on my pizero w no longer freezes.

https://github.com/ccrisan/motioneyeos/releases/download/nightly-dev/motioneyeos-raspberrypi-dev20181008.img.xz

@quantum8
Copy link

quantum8 commented Nov 2, 2018

Just tried that @tigernero79, it still failed a few hours later.

@VegasIOT
Copy link

VegasIOT commented Nov 2, 2018 via email

@starbasessd
Copy link

I have the 'loss of camera as of' error on 20180627 on RPi3B. Happens to a USB attached camera. Doesn't matter where the files are being saved (local or network) I tried to update to dev20181126 but during install, it downloads, unpacks, then tries to apply, but runs out of space on the partition. (8GB SD Card). File system is trashed afterwards. Goes into a reboot loop. Unable to recover data.
Re-imaged with 20180627. Booted, Went into KVM to console. Ran fwupdate upgrade dev20181126 before doing anything else. Same thing. unable to get logs off of SD Card.

@ccrisan
Copy link
Collaborator

ccrisan commented Dec 3, 2018

@starbasessd I'm sorry for your inconvenience, but the partition layout has been recently changed. You'll need to write the new OS image from scratch. See https://github.com/ccrisan/motioneyeos/releases/tag/20181129.

@jasaw
Copy link
Collaborator

jasaw commented Dec 4, 2018

@ccrisan I think there's a way to support fwupdate from old partition layout to the new one. The data partition is created by S00datapart if the partition does not exist.

  1. We could tar up a few configuration files like wpa_supplicant.conf and motioneye.conf, put them in the boot partition temporarily.
  2. We then delete the data partition and reboot. On device boot up, S00datapart will create a new data partition while reserving 1GB for boot + root.
  3. Untar the backed up config files from boot back into data partition and reboot.
  4. Now we can upgrade to the new latest firmware using the usual process.

Re-doing the data partition could be worth it. We could use it to support doing factory reset. Factory reset + "bring up wifi AP if not configured" will greatly improve the user experience. Aim is to minimize the need for users to reflash SD card.

What do you think?

@JohnWPB
Copy link

JohnWPB commented Jan 11, 2019

motioneye.zip
Hello, I found my way here from a Google Search.

I am having the same exact problem described many times above.

I start the raspberry Pi and MotionEye, and all is well. I can view the camera, and it records motion when detected. Randomly it loses the camera image. I can still access the web interface, and change and apply settings. The only way to get the camera working again is to reboot the Pi. It will work again for an undetermined time before it loses the camera again.

I have used 2 different Raspberry Pi 3 B+'s, and two different Class 10 32 gig Micro SD cards. (I installed from scratch, and did not just use the same SD card in the 2nd Pi.)

Specs:
Raspberry 3 B+ with heat sinks, in a 3D printed case with a cooling fan.
Connected: Via WiFi
Camera: Logitec C920
MotionEye Version: 0.39.3
Services: Samba server & FTP are disabled in MotionEye settings
Recording Location: local MicroSD card
3000 mA power supply

MotionEye Log is attached

@IAmOrion
Copy link
Author

IAmOrion commented Jan 15, 2019

@ccrisan ok, after being up for some time, a good week or so I think actually, it's just "died" again.
When I viewed the Web interface, I had the "No Image" picture. It was using SMB 2.1 at the time. I tried my usual fix of changing the SMB to 3.0 and clicking apply. That's been a hit and miss 'fix' as sometimes it works, sometimes it doesn't. In this instance, it did NOT work. Now I can't even access the web interface.

Rebooting remotely also refuses to work.

I can however access it via SSH / SFTP - so, before I battle my way to yanking the plug out of the bloody thing - what log files would you like? I'll download then via SFTP whilst it's in it's purgatory state of being useless and refusing to reboot.

I believe my versions are still the same as my original post:

motionEye Version | 0.39.2
Motion Version | 4.1.1
OS Version | motionEyeOS 20180627

Is there a way to double check those via terminal or the log files?

Just the usual? Eg:

boot.log
dmesg.log
messages.log
motion.log
motioneye.log

Any others?

@IAmOrion
Copy link
Author

@ccrisan

My logs are attached as a zip. I've also attached a screenshot that shows what happens when I type the reboot command.... the cursor moves to the next line, but nothing actually happens. I can't even ctrl+c to get out of it. I have to close terminal and reconnect :(

screenshot 2019-01-16 at 09 09 28

logs.zip

@ccrisan
Copy link
Collaborator

ccrisan commented Jan 23, 2019

@IAmOrion I simply cannot tell what's going on in this case. Rebooting should especially never fail since the first thing it does is to kill the watchdog feeder. Is this something 100% reproducible?

@IAmOrion
Copy link
Author

IAmOrion commented Jan 24, 2019

@IAmOrion I simply cannot tell what's going on in this case. Rebooting should especially never fail since the first thing it does is to kill the watchdog feeder. Is this something 100% reproducible?

@ccrisan It happens frequently, but I couldn't tell you how to specifically reproduce it on cue so to speak. But every time the image from my camera "dies" (So I get the static no image picture) I can no longer reboot. I did discover literally the other day, that, bizarrely, if I restart the router (thus killing wifi temporarily) it sometimes restarts - I presume the watchdog does it... but I just can't figure out what's preventing it manually (eg by me typing it in terminal in remote ssh window)

SSH/SFTP still works, and I can practically do everything else remotely still APART from reboot. It's very odd

@ccrisan
Copy link
Collaborator

ccrisan commented Jan 24, 2019

@IAmOrion when you find your unit in this bad state, could you please run the internal commands contained by /sbin/reboot one by one?

This is what /sbin/reboot looks like:

#!/bin/bash

# carry on with the script in case of error
set +e

# write buffers to disk
/bin/sync

# allow the shutdown script 20 seconds to shut down,
# after which we stop feeding the watchdog
(sleep 20 && /usr/bin/killall -STOP watchdog) &

# actual reboot command
/bin/busybox reboot

If anyone can spot a problem with this script, I'd be happy to address it.

@IAmOrion
Copy link
Author

@ccrisan Sure thing! I'll hold off updating this particular unit (I've updated a couple last night) because it will happen eventually! I'll report back when it does, thanks

@ir-fuel
Copy link

ir-fuel commented Jan 27, 2019

I've got similar stability issues on a new install (one day old). Right now the web interface works, the camera image does not show (but opening the URL to the IP cam feed in another tab works), and ssh hangs when trying to connect to the Pi.
I already had it stop recording twice the last 24h, but this is the first time ssh actually froze up.

@ir-fuel
Copy link

ir-fuel commented Jan 27, 2019

And now half an hour after rebooting it doesn't capture videos anymore, although the video feed still works in the web gui.

@ccrisan
Copy link
Collaborator

ccrisan commented Jan 27, 2019

@ir-fuel what OS version?

@ir-fuel
Copy link

ir-fuel commented Jan 27, 2019 via email

@ccrisan
Copy link
Collaborator

ccrisan commented Jan 27, 2019

@ir-fuel see if 20190119 fixes this problem for you.

@ir-fuel
Copy link

ir-fuel commented Jan 30, 2019

Installing the latest version in the prerelease list right now. It worked fine for 2 days and right now the system was blocked (no web server, no video uploads). I could SSH into the Raspberry Pi though. I'll let you know my findings.

@ccrisan
Copy link
Collaborator

ccrisan commented Jan 30, 2019

@ir-fuel please make sure you installed 20190119 and not some dev nightly build.

@ir-fuel
Copy link

ir-fuel commented Jan 30, 2019

How do I do this? If I only allow official releases it tells me I am on the latest version, and if I toggle to prerelease it shows me 20190130 as build to install.

@ir-fuel
Copy link

ir-fuel commented Jan 30, 2019

Ok got it:

note: do not attempt to update to this version automatically! Due to the different partition layout, you'll need to rewrite the OS image from scratch. Using the automatic update mechanism will, at best, fail with an error message.

I'll wipe my sd card and install it.

@IAmOrion
Copy link
Author

@ir-fuel
Copy link

ir-fuel commented Jan 31, 2019

FYI It's still running since I upgraded the version. Will see if it stays like this.

@IAmOrion
Copy link
Author

IAmOrion commented Feb 2, 2019

@ccrisan (@jasaw) unbelievably even on 20190119 pre-release, this STILL causes me problems!

I wonder if the reboot is failing because of writing buffers to disc part - which perhaps is stuck because Samba has died or something??

screenshot 2019-02-02 at 01 06 17

screenshot 2019-02-02 at 01 08 59

I tried to change Samba version, but when I clicked "Apply" it just got stuck and I get the standard browser error that the url is not responding

screenshot 2019-02-02 at 01 17 38

I can still connect via SSH, but can't reboot. Following your response re: the shut down script, this is what happened:

screenshot 2019-02-02 at 01 14 22

It froze after the bin/sync (I say froze, I can press enter and any other keys for that matter, but nothing ever happens.)

If I quit and reopen terminal, I can connect again and follow up the next set, but again, after the busybox reboot command, it freezes and never does anything more.

screenshot 2019-02-02 at 01 20 46

It'll sit like that forever if I let it! can't ctrl+c out of it or anything. I just end up quitting Terminal (And I could reconnect fine if I wanted). In this particular instance, the busybox reboot DID seem to complete (as you can see I'm on a new line in terminal) - but it never reboots.

Log files (in case they're of use)

Logs.zip

or individually:

boot.log
dmesg.log
messages.log
motion.log
motioneye.log
wpa_supplicant.log
wtmp.log

Samba Logs:

log.nmbd.log
log.smbd.log

@ccrisan
Copy link
Collaborator

ccrisan commented Feb 2, 2019

@IAmOrion reboot definitely fails due to waiting indefinitely for sync, in your case. Problems with syncing buffers are often linked to faulty SD cards. Can you please try a different (type of) SD card and see if the problem persists?

I can't tell if this particular problem can be caused by SMB, but you could temporarily disable SMB usage on your system and see if that's the cause. I would go with SD card change first, though.

@GeoffMorg
Copy link

FWIW I am noob to the Pi and this is only my second project. I am having the same problem. I set it up earlier today and was successfully sending images to my NAS (using SMB 2.0) for a few hours but decided to rebuild and relocate it, and since then (and further rebuilds) the camera just drifts off after a while - and I've not even got to the point of using my NAS, so it's all local.

I'm using it headless, just accessing through the browser and I can report that the power off button works OK.

I have tried three different SD cards (brand and size) and it continues to be a problem, I have another new Zero here so will try that tomorrow.

I'm using motioneyeos-raspberrypi-20190119 and would be happy to assist with any logs as required.

btw having said that it looks like a perfect product with the ability to do just what I need.

@GeoffMorg
Copy link

GeoffMorg commented Feb 6, 2019

Further to my post above, I had only successfully built a working ca,era 1 time out of maybe 15 times. So I took everything really slowly and documented each step with the idea that I might isolate which particular piece of config was breaking it, And of course it worked. It worked for a full day then I powered off overnight, and at next power on I have no camera.

I did backup the config so I will re-create the card with the software, and if that is working just reload my config.

And I do now have the ability to put it on a monitor to see what's happening, if that might assist you.

@Joseph-Muc
Copy link

Joseph-Muc commented Apr 10, 2019

dear friends of motioneyeos,
same Problem here with exact the same configuration.
Running a few days very nice and find it a very usefull system.
a lot of /bin/sync lines after ps command (Stat D).
There is no possibility to kill them.

greetings
Joseph

@ccrisan
Copy link
Collaborator

ccrisan commented Apr 12, 2019

@Joseph-Muc sync commands are there to make sure your changed data is written to the SD card as soon as possible during boot, so that in the event of a power loss, you don't lose your configuration.

It is expected not to be able to kill it but it's definitely not normal that it runs for long periods of time. Please try replacing the SD card with a new/faster one and see if the problem persists.

@C4PT41ND34DP00L
Copy link

C4PT41ND34DP00L commented Aug 16, 2019

I am having an issue similar to this.

MotionEye version is 20190427

It was working then had it powered down for a bit. I plugged it back in and connected to the web interface but the raspberry pi camera just spins and never loads.

Running on a raspberry pi zero w

image

motion.log
motioneye.log
messages.log
boot.log
dmesg.log

@starbasessd
Copy link

starbasessd commented Aug 16, 2019 via email

@C4PT41ND34DP00L
Copy link

@starbasessd Thank you for the information. It works in Firefox. Was not aware of an issue with Chrome.

@starbasessd
Copy link

starbasessd commented Aug 16, 2019 via email

@ccrisan
Copy link
Collaborator

ccrisan commented Aug 19, 2019

Fixed in recent nightly builds.

@ccrisan ccrisan closed this as completed Aug 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests