Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doorbell keeps going offline #338

Closed
dnestico opened this issue Apr 7, 2022 · 37 comments
Closed

Doorbell keeps going offline #338

dnestico opened this issue Apr 7, 2022 · 37 comments

Comments

@dnestico
Copy link

dnestico commented Apr 7, 2022

On the latest update with the following config settings and the doorbell, stream keeps going offline, I have an automation that notifies me when it happens, and it's basically spamming me about the doorbell, but not the other cameras.

Not sure why this is happening as it's not offline in the Wyze app, but offline on HA a lot of the time, Based off the offline notification it seems to come online then go back offline like every 5 mins or so. On the latest firmware for both wyze docker and doorbell.

My config:

WYZE_EMAIL: -
WYZE_PASSWORD: -
NET_MODE: ANY
SNAPSHOT: RTSP15
MQTT_DTOPIC: homeassistant
ROTATE_DOOR: true
WEBRTC: false
FRESH_DATA: true
RTSP_READTIMEOUT: 10s
KEEP_BAD_FRAMES: true
FPS_FIX: true
FILTER_BLOCK: true
FILTER_MACS: -

The log:

2022/04/07 00:56:09 [RTSP][FRONT-DOORBELL] 📕 Client stopped reading
2022/04/07 00:56:11 [RTSP][FRONT-DOORBELL] ❌ '/front-doorbell' stream is down
2022/04/07 00:56:11 [RTSP][FRONT-DOORBELL] 📕 Client stopped reading

@mrlt8
Copy link
Owner

mrlt8 commented Apr 7, 2022

Anything in the log before that?

You might want to increase RTSP_READTIMEOUT to give it a few extra seconds to recover.

@dnestico
Copy link
Author

dnestico commented Apr 7, 2022

That's everything that related to the doorbell, and true, this did start happening after I made it 10s rather than the 30s I had before, will see if that fixes it soon.

Also, not sure if this has anything to do with the addon or if it may be just the camera card for HA but every time the doorbell image refreshes it seems to show that offline picture before showing the updated thumbnail seconds later, only the doorbell does this.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 7, 2022

Try to keep an eye on the logs next time it happens to see if you see any other messages around the time:

2022/04/05 20:50:52 [RTSP][BACK] ❌ '/back' stream is down
2022/04/05 20:50:52 [RTSP][BACK] 📕 Client stopped reading
2022/04/05 20:50:53 [back] Stream did not receive a frame for over 20s

@dnestico
Copy link
Author

dnestico commented Apr 7, 2022

I will definitely do so but gonna have to wait for 2022.4.1 because it seems 2022.4.0 has broken still images since I only have the "stream-source" in the camera configs, (home-assistant/core#69128) which is also taking forever to finish the database upgrade too atm, been over an hour so far.

But I have made the 'RTSP_READTIMEOUT' 30 seconds now, so will have to see if that helps.

@dnestico
Copy link
Author

dnestico commented Apr 8, 2022

I managed to get these from the log too, it seems to be affecting all of my cameras and I have no idea why they keep showing as unavailable in HA when they are not offline, because of this my notification snapshots are never accurate, and always hours behind.

CPU usage also jumps insanely high sometimes all the way from 32% to 88%


Here is my config:

WYZE_EMAIL: -
WYZE_PASSWORD: -
NET_MODE: ANY
SNAPSHOT: RTSP15
MQTT_DTOPIC: homeassistant
ROTATE_DOOR: true
WEBRTC: false
FRESH_DATA: false
RTSP_READTIMEOUT: 30s
KEEP_BAD_FRAMES: false
FPS_FIX: false
FILTER_BLOCK: true
FILTER_MACS: -


2022/04/08 00:13:29 [Front Doorbell] WARNING: Skipping smaller frame at start of stream (frame_size=4)

2022/04/08 00:14:31 [Front Doorbell] WARNING: FPS param mismatch (framerate=20)

2022/04/08 00:13:29 [Driveway] WARNING: Skipping smaller frame at start of stream (frame_size=1)

Screen Shot 2022-04-08 at 12 16 55 AM


And finally, when I was trying to test why the snapshots are old and it wasn't even making a new one when I called the service, this is what the HA log showed:

Error from stream worker: Error opening stream (ERRORTYPE_111, Connection refused) rtsp://:@127.0.0.1:8554/driveway
12:20:33 AM – (ERROR) Stream - message first occurred at April 7, 2022, 7:14:18 PM and shows up 7 times
Error from stream worker: Error opening stream (ERRORTYPE_111, Connection refused) rtsp://:@127.0.0.1:8554/porch
12:20:03 AM – (ERROR) Stream - message first occurred at April 7, 2022, 7:14:18 PM and shows up 25 times
Error from stream worker: Error opening stream (ERRORTYPE_111, Connection refused) rtsp://:@127.0.0.1:8554/garage-cam
12:20:03 AM – (ERROR) Stream - message first occurred at April 7, 2022, 7:14:18 PM and shows up 25 times
Error from stream worker: Error opening stream (ERRORTYPE_111, Connection refused) rtsp://:@127.0.0.1:8554/front-doorbell
12:20:03 AM – (ERROR) Stream - message first occurred at April 7, 2022, 7:14:18 PM and shows up 64 times

@dnestico
Copy link
Author

dnestico commented Apr 9, 2022

@mrlt8 Yea so this now seems to be affecting more than the doorbell, no idea why the cameras go offline randomly when they work fine in the wyze app.

I wonder if it's because they are constantly streaming? Maybe on-demand streaming could solve this... it seems to be really affecting my snapshot automations too.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 9, 2022

How's the WiFi coverage around those two cameras?
Does limiting the number of cameras improve stability?

@dnestico
Copy link
Author

dnestico commented Apr 9, 2022

So I restarted the bridge, and like usual, that fixed it and they are back online.

Also, all of my cameras are near my modem, especially the 2 of them are basically beside it. I actually just added a 5th camera to my setup so not sure about that one 😂... The cameras are back online on HA right now but I guess I'll let you know if they go offline again.

But the two cameras I was talking about were the "Porch" (98% wifi) and the "Garage Cam" (89% wifi)

I just restarted the addon again to get you the wifi info and it really doesn't seem like it should be an issue as you can see here:

2022/04/08 23:41:55 [Luna Cam] 📡 Getting 120kb/s HD stream (20fps) via LAN mode (WiFi: 67%) FW: 4.36.3.19 (2/3)

2022/04/08 23:41:55 [Front Doorbell] 📡 Getting 120kb/s HD stream (20fps) via LAN mode (WiFi: 98%) FW: 4.25.1.042 🔒

2022/04/08 23:41:55 [Porch] 📡 Getting 120kb/s HD stream (15fps) via LAN mode (WiFi: 98%) FW: 4.9.8.1002 🔒 (DTLS) (2/3)

2022/04/08 23:41:55 [Driveway] 📡 Getting 120kb/s HD stream (20fps) via LAN mode (WiFi: 95%) FW: 4.36.9.117 🔒 (DTLS) (2/3)

2022/04/08 23:41:55 [Garage Cam] 📡 Getting 120kb/s HD stream (15fps) via LAN mode (WiFi: 89%) FW: 4.9.8.1002 🔒 (DTLS) (2/3)

@mrlt8
Copy link
Owner

mrlt8 commented Apr 9, 2022

I think we're getting a couple of overlapping issues here..

  1. Are the cams actually offline or is it just reporting as offline in HA?
  2. Do the cameras stay offline or do they eventually come back?
  3. Anything in the add-on logs around the stream is down messages?
  4. Does limiting the number of cameras improve stability?

The streams do go down every now and then (If you watch closely, you'll notice something similar in the app), but they should be able start back up on their own.

@dnestico
Copy link
Author

dnestico commented Apr 9, 2022

Ok sorry for the confusion, I just didn't want to open 100 issues.

  1. No the cameras are not offline, they are just reporting "unavailable" in HA
  2. The cameras seem to stay offline until I restart the bridge
  3. The log says "Camera is up" but it's reporting "unavailable" in HA and normally I haven't gotten the "camera" is down message but I managed to just now:
    "2022/04/09 13:44:09 [RTSP][PORCH] ❌ '/porch' stream is down"
    "2022/04/09 13:44:09 [RTSP][PORCH] 📕 Client stopped reading"
    "2022/04/09 13:44:09 [Porch] Stream did not receive a frame for over 20s"
    "2022/04/09 13:44:12 [RTSP][PORCH] ✅ '/porch' stream is UP! (3/3)" (It isn't though in HA)
  4. I haven't tried that yet but maybe will soon, I just restarted the hub again from last night since 2 cameras went unavailable again, 1 came back after 5 minutes, other didn't

Here are some errors listed in the logs though, note that not all the cameras go "unavailable" even with these errors, (Garage Cam) came bac online even with this:

2022/04/09 13:43:48 [Garage Cam] WARNING: Skipping smaller frame at start of stream (frame_size=1)
2022/04/09 13:43:48 [Garage Cam] WARNING: Waiting for keyframe
2022/04/09 13:43:49 [Porch] WARNING: Skipping smaller frame at start of stream (frame_size=1)
2022/04/09 13:43:50 [Porch] WARNING: Frame not available yet
2022/04/09 13:44:53 [Porch] WARNING: Wrong resolution (frame_size=1)
2022/04/09 13:44:53 [Porch] WARNING: Requesting frame_size=0 and bitrate=120

@dnestico
Copy link
Author

dnestico commented Apr 9, 2022

Just an update: So the "Porch cam" is staying offline for whatever reason so I am going to filter that out from the bridge, so now it will be 4 cameras instead of 5.

🚀 STARTING DOCKER-WYZE-BRIDGE v1.3.2
🏠 Home Assistant Mode
2022/04/09 14:11:19 [WyzeBridge] ♻️ FORCED REFRESH - Ignoring local 'user' data
2022/04/09 14:11:19 [WyzeBridge] ♻️ FORCED REFRESH - Ignoring local 'auth' data
2022/04/09 14:11:19 [WyzeBridge] ☁️ Fetching 'auth' from the Wyze API...
2022/04/09 14:11:20 [WyzeBridge] 💾 Saving 'auth' to local cache...
2022/04/09 14:11:20 [WyzeBridge] ☁️ Fetching 'user' from the Wyze API...
2022/04/09 14:11:20 [WyzeBridge] 💾 Saving 'user' to local cache...
2022/04/09 14:11:20 [WyzeBridge] ♻️ FORCED REFRESH - Ignoring local 'cameras' data
2022/04/09 14:11:20 [WyzeBridge] ☁️ Fetching 'cameras' from the Wyze API...
2022/04/09 14:11:21 [WyzeBridge] 💾 Saving 'cameras' to local cache...

One question I have though does "2022/04/09 14:11:20 [WyzeBridge] ☁️ Fetching 'user' from the Wyze API..." mean it's not streaming the cameras locally but instead through the API? Would hope not...

One more question does SNAPSHOT=RTSP15 have anything to do with it, you think?
I get TONS of "client stopped reading" / "new client reading"

@pjkix
Copy link

pjkix commented Apr 9, 2022

I don't know if this is related or not but i was having issues with safari and Home assistant saying "network stream error" cams not updated live, constantly needing to refresh the page, then i switched browsers to firefox and all my issues magically disappeared, cams are all live now and no error messages. I suspect it has something to do with safari trying to be power efficient or something. Anyway just thought I'd add a note here. Cheers

@mrlt8
Copy link
Owner

mrlt8 commented Apr 10, 2022

Interesting.. this seems like it could be more of a HA issue..? how are you checking the status of the cams, is it from MQTT?

  • Skipping smaller frame at start of stream is pretty much what it says, the camera is probably set to use the 360p stream, so the bridge requests the HD "frame size" and is waiting for the camera to switch before starting the stream. This will keep looping until the HD frame size is received (which may be the CPU spike you see sometimes).

  • Wrong resolution (frame_size=1) same as above, but this is happening midway through the stream. Can happen if you open the app and it's set to pull the 360p stream, so the bridge attempts change it back to HD: Requesting frame_size=0 and bitrate=120

  • Frame not available yet this means that the bridge hasn't received a frame from the camera in over a second.

  • Fetching data from wyze api - you probably have FRESH_DATA set.

  • yes, SNAPSHOT=RTSP will be pulling the images from the stream itself, so it will show as a client reading.

@dnestico
Copy link
Author

dnestico commented Apr 10, 2022

Ok so with this info I'm taking that client reading is not an issue, which is good I guess because I have it set to SNAPSHOT=RTSP15 and soon SNAPSHOT=RTSP5 when that's released.

The whole resolution thing HAS to be a wyze issue because even in the God damn wyze app I always set the camera resolutions to "HD" but they always load in "SD" for some reason! So I really have no idea how to solve that tbh unless the app stream is separate and doesn't matter. I am perfectly fine with HA receiving the cameras in 360p if that helps with everything tbh, would just hope it would switch to 720p/HD when I actually click the thumbnails to view the live video though.

And I am not using MQTT, no and yes correct I enabled FRESH_DATA this week because I added another camera to my setup and I wanted it to be detected by the wyze hub. (I have 4 cameras exposed to it atm)

Also, the thumbnails are sometimes behind the actual stream yes but like I said it fixes itself after a couple of hours sometimes, but as of right now my doorbell is still about 30 mins behind the other cameras.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 10, 2022

The resolution thing shouldn't be an issue as the bridge should be able to correct for that within a couple of frames.

If not via mqtt, where is HA listing the cameras as offline? Can you stream from the RTSP stream via VLC when it is reporting as offline?

@dnestico
Copy link
Author

dnestico commented Apr 10, 2022

So for whatever reason, 1 camera is once again reporting as "unavailable" in HA, on the Wyze app they are perfectly fine and up and running.

Also nope, cannot access it via VLC, something's up with the bridge for sure. I removed 2 cameras fully from the Hub and HA and am only using 4 cameras right now.

Screen Shot 2022-04-10 at 2 22 07 PM

@mrlt8
Copy link
Owner

mrlt8 commented Apr 11, 2022

Unless you have auth setup in rtsp-simple-server, the stream uri to that in the readme, in your case, assuming HA running on the same machine as VLC:

rtsp://homeassistant.local:8554/garage-cam

@dnestico
Copy link
Author

I will have to try this again if another camera goes offline 👍

@dnestico
Copy link
Author

dnestico commented Apr 11, 2022

@mrlt8

So I've just been able to test this as the garage cam is showing as "unavailable" in HA, but not the app.... And it isn't accessible through both URLs on VLC.

But I'm watching it live on the wyze app as I'm typing this! haha. I will be restarting HA a couple times today anyways since I am adding some stuff to it so it'll probably come back after.

One thing I haven't mentioned is I have my HA on a 2nd router that the cameras are not on, this never seemed like an issue as because the router is plugged in to the main one (it's basically a repeater) it still is able to stream the cameras, maybe in the future I may try moving my RIP4 to the main router which is closer to all the cameras, will just have to see if this will mess anything else up though but it shouldn't.

Screen Shot 2022-04-11 at 1 23 43 PM

Screen Shot 2022-04-11 at 1 23 32 PM

@mrlt8
Copy link
Owner

mrlt8 commented Apr 12, 2022

I don't think the router is an issue since it starts up correctly.

Unavailable in HA does not mean that the camera is offline.

Again, please view the bridge's logs when it is showing as unavailable, otherwise we have no idea what is going on.

The bridge, just like the app, will lose connection to the camera from time to time, but is supposed to restart the process if the connection is lost. The bridge will show and adjust the connection attempts if the camera is detected as being offline.

@dnestico
Copy link
Author

dnestico commented Apr 12, 2022

Well in other news my "garage cam" has not come back in HA, but checked in the logs it said the camera came back online... so I checked and the thumbnail updated, but the stream was not online so it looks like it came back for like 1 second enough to update thumbnail then went back as unavailable. weird. It actually did this a couple times.

Now regarding the doorbell, I just checked and ever since turning off the doorbell rotation it's only fell back once and by 5 minutes, I will check again tomorrow too see if this stays like that but that seems like positive news.

QUICK UPDATE:

So I've checked multiple times and now have my dashboard open in the background and the thumbnail seems to be updating on the garage cam but the stream is not working?... damn that is weird. The logs seem to think the camera is online though but the stream definitely isn't so I wonder what's going on here.

ANOTHER UPDATE:

And now suddenly the garage cam stream is also up lol. Gonna keep the developer page and logs open in the background to monitor it a bit.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 12, 2022

Thumbnails from RTSP15? If those are updating then it means the rtsp stream is most likely still up and it's whatever HA is doing to read the stream that is having issues..?

@dnestico
Copy link
Author

Yep, RSTP15 for now then RTSP5 when that comes out, and I assumed it was HA with the problem.

I just wish it wouldn't randomly go unavailable when the wyze app stream isn't down.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 12, 2022

Have you tried the ffmpeg platform instead? I believe that one does it's own snapshots as well.

@dnestico
Copy link
Author

dnestico commented Apr 12, 2022

I moved away from the ffmpeg and switched to generic a couple weeks ago as I was told it was better, I may try the doorbell on it soon as I don't remember these problems on ffmpeg.. I could be wrong though.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 16, 2022

I've been running some tests, and believe this is an MQTT bug that's just reporting the camera as offline even though the camera is still up.

Will need to look into it, but you should be able to ignore the status and continue to use the cams as normal.

mrlt8 added a commit that referenced this issue Apr 17, 2022
@dnestico
Copy link
Author

I'm not even using MQTT but I hope this fixes it because I have one camera that is constantly unavailable in home assistant so no video feed, but it never is offline in the app, getting annoying.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 17, 2022

When you say unavailable, do you mean the RTSP stream is not responding or is it just saying "unavailable" somewhere in home assistant..?

MQTT is what is sending the entity info in #340 and I'm assuming the camera status that you're referring to. MQTT should set the status to "offline" if the stream is down or "disconnected" if the bridge stop updating the status for some reason.

@dnestico
Copy link
Author

When I mean "unavailable" I mean the stream doesn't play, sorry.

Here is more info here in this issue. #340

mrlt8 added a commit that referenced this issue Apr 18, 2022
@mrlt8
Copy link
Owner

mrlt8 commented Apr 18, 2022

This is definitely Home assistant related. I was able to replicate the "unavailable" with generic and ffmpeg even though the actual rtsp stream was still up in VLC. Not sure why HA stop reading the stream though...

@dnestico
Copy link
Author

Just checked again today and the doorbell is 1 minute behind which is the best it's been so will have to report back if it stays this way.

This is still with all cameras filtered out.

And I am for sure starting to think this is a HA issue too, wonder what I would name the issue to report to them.

@mrlt8
Copy link
Owner

mrlt8 commented Apr 19, 2022

take a look at https://community.home-assistant.io/t/i-tried-all-the-camera-platforms-so-you-dont-have-to/222999

They all seem to have some sort of lag and/or processing issue.

As to why HA stops reading the stream, we could try making some tweaks to rtsp-simple-server to see if we might be able to improve things. Is it mostly just the doorbell that goes unavailable or do the other cameras go unavailable at a similar rate?

@dnestico
Copy link
Author

dnestico commented Apr 19, 2022

The other cameras also go unavailable, I would say the doorbell goes unavailable the least, but is always behind when it is available.

And yea i read that before and may make some changes as I plan to test the V3 RTSP firmware alongside the docker.

@krystiancharubin
Copy link

try removing FPS_FIX and setting QUALITY to HD180, this made by doorbell stay up for days with a few short timeout but it always reconnects successfully.

@dnestico
Copy link
Author

dnestico commented Apr 20, 2022

I'm on the dev branch and don't see an option for FPS_FIX but I added the HD180 quality.

mrlt8 added a commit that referenced this issue Apr 20, 2022
@dnestico
Copy link
Author

dnestico commented Apr 20, 2022

I just switched back to the public version and the doorbell started with a 3 min delay.

This is the setup as recommended:

WYZE_EMAIL: -
WYZE_PASSWORD: -
NET_MODE: ANY
SNAPSHOT: RTSP5
MQTT_DTOPIC: homeassistant
ROTATE_DOOR: true
WEBRTC: false
FRESH_DATA: true
RTSP_READTIMEOUT: 25s
KEEP_BAD_FRAMES: false
FPS_FIX: false
FILTER_BLOCK: true
FILTER_MACS: 2CAA8EB10CD8, 2CAA8E71FC03

@mrlt8
Copy link
Owner

mrlt8 commented Apr 21, 2022

I noticed that the rotation could cause the bitrate to spike, so I limited the bitrate to 3,000 kb/s in dev 4 to see if that might help with the drops.

mrlt8 added a commit that referenced this issue Apr 24, 2022
* Clear MQTT topics on exit #338

* Time based keyframes #338

* Speed up snapshots #340

* Limit bitrate on rotation #338 #340

* Don't block on FPS change #340

* Match cases when checking model names #346

* APP_VERSION bump to 2.30.0

* Option to save snapshots as PNG

* tutk_ioctl_mux log level #330

* send_ioctl log level

* Reduce rtsp snap logging #340
@mrlt8 mrlt8 closed this as completed May 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants