Repository navigation
Watchdog
A watchdog is a monitoring mechanism designed to ensure that a system continues to operate correctly. In the context of a real-time robotic system like our rover, the watchdog acts as a supervisor that detects when part of the system becomes unresponsive or stuck in an unexpected state.
If a fault occurs (e.g., software hangs, sensor stops publishing, or control messages stop being received), the watchdog can automatically take corrective action – such as restarting a process, resetting a subsystem, or triggering an emergency stop (E-stop).
A Hardware Watchdog is a dedicated timer circuit, usually external to the main processor or microcontroller (MCU). It requires the main system to periodically send it a heartbeat signal to ensure that things are running smoothly.
If the watchdog doesn’t receive this signal within a set time interval (the timeout), it assumes the system has failed and triggers a reset or some other recovery action defined in the Watchdog behavior.
Characteristics:
- Independent from the main CPU – continues running even if the CPU crashes.
- Typically implemented using GPIO toggling or dedicated watchdog pins.
- Provides strong fault tolerance and reliability.
- Commonly used in systems where they are placed in harsh environments away from operators and are running autonomously (e.g., rovers, drones).
A software watchdog performs the same conceptual function as the hardware version, but it’s implemented in software, using internal timers or monitoring processes. It can be part of the operating system, a ROS node, or a supervisory application that watches for expected activity – like incoming topics, node heartbeats, or data updates.
Characteristics:
- Easier to implement and customize within the software stack.
- Can monitor specific subsystems.
- Less reliable than hardware watchdogs, since a total system freeze could prevent the software watchdog from executing (dependency).
- Best used as a higher-level monitor, complementing a hardware watchdog.
A watchdog does not fix bugs or eliminate the need for robust programming practices. However, it adds resilience against unexpected failures, such as:
- Software deadlocks
- Node crashes or hangs
- Communication dropouts (from base station to rover)
- Environment factors causing transient faults
By detecting these conditions and automatically recovering (via rest or safe shutdown), the system can continue operating (or at least fail safely) without human intervention.
Note: For remote and autonomous systems like the rover, where manual recovery isn’t possible, watchdogs are essential for maintaining safety and reliability.