Timer Watch Dog, to prevent system lockups

[Cruzxia]

Overview
The Tasmota has two system watchdogs, but they rely on the ESP8266 firmware and hardware.
If the Wi-Fi becomes unstable or if there is a power failure I find that some of my Tasmota devices stop responding, and the inbuilt watchdogs will not reset the device, as they do not detect any issue.
I can only get them to start working again via a reboot or if I reboot the Wi-Fi.
Restarting the Wi-Fi is not ideal, and it will sometimes cause another Tasmota device to stop responding.

I have implemented a timer based watchdog on other ESP8266 projects, with good results.
It is simple to do, and it will prevent the system getting stuck in the ESP core firmware without triggering a reset.
You could make it selectable if not all users want the feature.

Implementation....
Interrupt timer
Setup a timer interrupt with a counter.
In the interrupt test for a timer overflow.
If the timer overflows, call Reboot of the Tasmota device.

In the main loop
Clear the counter once per loop.

OTA updates
Usually I setup my systems for a 60 second timer. That is enough time to perform an OTA update without triggering a reset. It may be necessary to increase amount of time allowed for the OTA update.

Summing up
This will prevent the system locking up in an ESP core loop, that is beyond the control of the Tasmota firmware.
You will get the occasional reboot when something goes wrong, on my systems it can be weeks apart, but eventually they will reboot.

[ascillato]

Hi,

Thanks. The software and hardware watchdogs are already in place. Your issue seems to be another thing. You should attach a serial connection to your device so as to check the logs.

[Cruzxia]

The hardware and software watchdogs do not activate if it is caught in a system loop. Once I noticed it was not responding, I have left it like that for 24hrs to see if it would reset by itself, and the inbuilt watchdogs do not trigger. Also the input switch will not operate the relay. (SonOff Mini).
This glitch can go for days or weeks before it occurs again, so it is hard to monitor with a PC via the serial.

I suspected it may be Wi-Fi 6 that causes the problem, but I tried it turned off, and it still stopped responding.

I have been looking to find a solution, and I have come across a lot of posts of people having similar issues.

My solution is a simple last resort backup. I would like to find and fix the actual issue, but that is not always possible due to device limitations. I don't think that this is a Tasmota software problem, as I have had it occur on other ESP8266 devices.
I think Tasmota is being let down by the Espressif's libraries and subroutines.

[rapi3]

4 days ago experienced a problem with my Tasmota devices in a remote location after I did some work on my old rpi running MQTT server and It rebooted unexpectedly few time.
That remote rpi I have it is configured to use Bond wifi+LAN for safety and this from my experience cause some problem with Cisco AP's ( cisco it is the problem here and need to be configured as Root Bridge with Wireless Clients to support Bond devices and still is not working properly with Bond devices ).
The problem was that all Tasmota connected to AP1 was unable to connect to MQTT server, WIFI was connected; all other Tasmota devices connected to AP2 worked ok in the same lan.
The only solution to fix after I rebooted all tasmota devices and that did not fix them was to restart AP1.
I already have a rule in every Tasmota device that check every hour for wifi connection - ping to switch and if fail will restart Tasmota but in this case did not help.
It look like I need to make another rule to restart Cisco AP if MQTT it is disconnected to long and rpi it is online - wifi work.

My point is that you have 25% chances that fault it is in Tasmota sw or somewhere in the chain in AP-switch-MQTT server.... so you need to think at a solution to test all devices involved.
Implementing a timer that can interfere with OTA upgrade it is a disaster waiting to happen.

[arendst]

Tasmota on esp8266 already contains a watchdog mechanism which will trigger on blocked loops. This mechanism has been disabled on esp32.

See the code looking for OSWatch.

[Cruzxia]

Follow up.
I setup a serial connection to monitor one of the Tasmota SonOff units and it eventually hung up. The Wi-Fi goes into a limbo state, where it is connected to the router, but wont transfer data. Because it is still connected to the Wi-Fi it wont try and reconnect so the Tasmota unit becomes inaccessible, pending a reboot or a router reboot. Adding a Timer Watchdog would keep it out of this limbo state, limiting the outage to 60 seconds rather than until a manual reboot. I have spent some time looking at my router and testing various settings to try and resolve this problem.

Resolved - Wi-Fi router
I have a Asus mesh router, initially I thought it was the Wi-Fi 6 (ax) that was causing issues, but turning that on or off made no difference.
The issue turned out to be IGMP Snooping. Once I disabled the IGMP all my Tasmota remained connected and operational.

The Asus IGMP setting can be found in you router as follows: Wireless/Professional - Enable IGMP Snooping DISABLE.

This may also be a problem on other brands of routers, so if you are having issues it would be worth checking for this setting.

[sfromis]

ASUS routers are a not uncommon source of trouble.

[zakdma]

Hello, @arendst !
I have the same issue with wifi that 'connected' but no data transferred. And this repeated meny times. OpenBeken firmware has watchdog that ping particular IP, usually router IP, and reconnect if there is no answer for particular amount of time. And its working like a charm. So now I have no any connection issues with OpenBeken devices but have with Tasmota ones. And it's annoying.
Could you please implement such a feature for Tasmota?

Regards.

[sfromis]

Using rules or whatever, you could create a ping-watcher without needing core code changes.

[Cruzxia]

I am connecting to an MQTT server.
I added the Rule 'ON Mqtt#Disconnected DO Restart 1 ENDON'
This seems to work quite well by rebooting Tasmota if the device disconnects from the server.
I have had hardly any issues with the units not responding since I added this rule.
Note: if you are running a timer, the time will be reset on rebooting. If the server cannot be contacted the time will not be updated and your timer may not trigger.

[sfromis]

A lot depends on what type of failure you want to try to handle, but the most robust approach would be to have the server send "heartbeat" messages at intervals, enabling you to "do something" in case of those messages no longer arriving. Very simplest type of heartbeat could be to send RuleTimer commands to keep a on Rules#Timer trigger from firing.

The rules cookbook does also have examples of creating watchdogs based on Ping (optional feature not included by default), or WebQuery.
https://tasmota.github.io/docs/Rules/#watchdog-for-wi-fi-router-or-modem