Merge #1887

1887: sched: Add a watchdog r=bradjc a=alistair23

### Pull Request Overview

As part of #1882 we started talking about safe waits (#1552).

I have a PR with some initial safe wait support (#1886) in that we can be interrupted while looping on a hardware condition. The idea here is that while entering low power mode we can skip the infinite loop if we get an interrupt.

#1886 doesn't address a large number of loops in the driver capsules as they don't have access to `Chip` so it's more difficult to wait on interrupts. The harder part is what to do when a wait "fails". What happens when returning to full power fails to occur in a specific time, what happens when a bus fails to respond in a reasonable amount of time? How do we gracefully handle this and continue running in some capacity.

This PR starts to implement a watchdog functionality. This is not instead of a safe wait (in lots of cases a safe wait would be more graceful) but is generally to help avoid infinite loops and help debugging. It is also something that has come up to help make Tock more reliable.

It is up to `Chips` and boards to handle the watchdog interrupt and decided how to act.

Currently we tickle the watch dog when iterating on each process and at the start of each kernel loop.

### Testing Strategy

None

### TODO or Help Wanted

Feedback

### Documentation Updated

- [X] Updated the relevant files in `/docs`, or no updates are required.

### Formatting

- [X] Ran `make format`.
- [X] Fixed errors surfaced by `make clippy`.


Co-authored-by: Alistair Francis <alistair.francis@wdc.com>
Co-authored-by: Brad Campbell <bradjc5@gmail.com>

chips/apollo3/src/chip.rs

@@ -31,6 +31,7 @@ impl Chip for Apollo3 {
     type MPU = cortexm4::mpu::MPU;
     type UserspaceKernelBoundary = cortexm4::syscall::SysCall;
     type SchedulerTimer = cortexm4::systick::SysTick;
+    type WatchDog = ();
 
     fn service_pending_interrupts(&self) {
         unsafe {
@@ -75,6 +76,10 @@ impl Chip for Apollo3 {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &cortexm4::syscall::SysCall {
         &self.userspace_kernel_boundary
     }

chips/arty_e21_chip/src/chip.rs

@@ -108,6 +108,7 @@ impl kernel::Chip for ArtyExx {
     type MPU = rv32i::pmp::PMPConfig;
     type UserspaceKernelBoundary = rv32i::syscall::SysCall;
     type SchedulerTimer = ();
+    type WatchDog = ();
 
     fn mpu(&self) -> &Self::MPU {
         &self.pmp
@@ -117,6 +118,10 @@ impl kernel::Chip for ArtyExx {
         &()
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &rv32i::syscall::SysCall {
         &self.userspace_kernel_boundary
     }

chips/e310x/src/chip.rs

@@ -54,6 +54,7 @@ impl<A: 'static + Alarm<'static>> kernel::Chip for E310x<A> {
     type MPU = rv32i::pmp::PMPConfig;
     type UserspaceKernelBoundary = rv32i::syscall::SysCall;
     type SchedulerTimer = kernel::VirtualSchedulerTimer<A>;
+    type WatchDog = ();
 
     fn mpu(&self) -> &Self::MPU {
         &self.pmp
@@ -63,6 +64,10 @@ impl<A: 'static + Alarm<'static>> kernel::Chip for E310x<A> {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &rv32i::syscall::SysCall {
         &self.userspace_kernel_boundary
     }

chips/earlgrey/src/chip.rs

@@ -110,6 +110,7 @@ impl<A: 'static + Alarm<'static>> kernel::Chip for EarlGrey<A> {
     type MPU = rv32i::pmp::PMPConfig;
     type UserspaceKernelBoundary = SysCall;
     type SchedulerTimer = kernel::VirtualSchedulerTimer<A>;
+    type WatchDog = ();
 
     fn mpu(&self) -> &Self::MPU {
         &self.pmp
@@ -119,6 +120,10 @@ impl<A: 'static + Alarm<'static>> kernel::Chip for EarlGrey<A> {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &SysCall {
         &self.userspace_kernel_boundary
     }

chips/nrf52/src/chip.rs

@@ -30,6 +30,7 @@ impl<I: InterruptService> kernel::Chip for NRF52<I> {
     type MPU = cortexm4::mpu::MPU;
     type UserspaceKernelBoundary = cortexm4::syscall::SysCall;
     type SchedulerTimer = cortexm4::systick::SysTick;
+    type WatchDog = ();
 
     fn mpu(&self) -> &Self::MPU {
         &self.mpu
@@ -39,6 +40,10 @@ impl<I: InterruptService> kernel::Chip for NRF52<I> {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &Self::UserspaceKernelBoundary {
         &self.userspace_kernel_boundary
     }

chips/sam4l/src/chip.rs

@@ -76,6 +76,7 @@ impl Chip for Sam4l {
     type MPU = cortexm4::mpu::MPU;
     type UserspaceKernelBoundary = cortexm4::syscall::SysCall;
     type SchedulerTimer = cortexm4::systick::SysTick;
+    type WatchDog = ();
 
     fn service_pending_interrupts(&self) {
         unsafe {
@@ -179,6 +180,10 @@ impl Chip for Sam4l {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &cortexm4::syscall::SysCall {
         &self.userspace_kernel_boundary
     }

chips/sam4l/src/wdt.rs

@@ -6,7 +6,6 @@ use cortexm4::support;
 use kernel::common::math::log_base_two_u64;
 use kernel::common::registers::{register_bitfields, FieldValue, ReadOnly, ReadWrite, WriteOnly};
 use kernel::common::StaticRef;
-use kernel::hil;
 
 #[repr(C)]
 pub struct WdtRegisters {
@@ -220,16 +219,17 @@ impl Wdt {
     }
 }
 
-impl hil::watchdog::Watchdog for Wdt {
-    fn start(&self, period: usize) {
-        self.start(period);
-    }
-
-    fn stop(&self) {
-        self.stop();
+impl kernel::watchdog::WatchDog for Wdt {
+    fn setup(&self) {
+        // Setup the WatchDog with a 100ms period.
+        self.start(100);
     }
 
     fn tickle(&self) {
         self.tickle();
     }
+
+    fn suspend(&self) {
+        self.stop();
+    }
 }

chips/stm32f303xc/src/chip.rs

@@ -32,6 +32,7 @@ impl Chip for Stm32f3xx {
     type MPU = cortexm4::mpu::MPU;
     type UserspaceKernelBoundary = cortexm4::syscall::SysCall;
     type SchedulerTimer = cortexm4::systick::SysTick;
+    type WatchDog = ();
 
     fn service_pending_interrupts(&self) {
         unsafe {
@@ -82,6 +83,10 @@ impl Chip for Stm32f3xx {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &cortexm4::syscall::SysCall {
         &self.userspace_kernel_boundary
     }

chips/stm32f4xx/src/chip.rs

@@ -32,6 +32,7 @@ impl Chip for Stm32f4xx {
     type MPU = cortexm4::mpu::MPU;
     type UserspaceKernelBoundary = cortexm4::syscall::SysCall;
     type SchedulerTimer = cortexm4::systick::SysTick;
+    type WatchDog = ();
 
     fn service_pending_interrupts(&self) {
         unsafe {
@@ -102,6 +103,10 @@ impl Chip for Stm32f4xx {
         &self.scheduler_timer
     }
 
+    fn watchdog(&self) -> &Self::WatchDog {
+        &()
+    }
+
     fn userspace_kernel_boundary(&self) -> &cortexm4::syscall::SysCall {
         &self.userspace_kernel_boundary
     }

kernel/src/hil/mod.rs

@@ -25,7 +25,6 @@ pub mod symmetric_encryption;
 pub mod time;
 pub mod uart;
 pub mod usb;
-pub mod watchdog;
 
 /// Shared interface for configuring components.
 pub trait Controller {

kernel/src/lib.rs

@@ -36,6 +36,7 @@ pub use crate::driver::Driver;
 pub use crate::grant::Grant;
 pub use crate::mem::{AppSlice, Private, Shared};
 pub use crate::platform::scheduler_timer::{SchedulerTimer, VirtualSchedulerTimer};
+pub use crate::platform::watchdog;
 pub use crate::platform::{mpu, Chip, Platform};
 pub use crate::platform::{ClockInterface, NoClockControl, NO_CLOCK_CONTROL};
 pub use crate::returncode::ReturnCode;

kernel/src/platform/mod.rs

@@ -8,6 +8,7 @@ use core::fmt::Write;
 
 pub mod mpu;
 pub(crate) mod scheduler_timer;
+pub mod watchdog;
 
 /// Interface for individual boards.
 ///
@@ -82,6 +83,10 @@ pub trait Chip {
     /// to applications.
     type SchedulerTimer: scheduler_timer::SchedulerTimer;
 
+    /// The implementation of the WatchDog timer used to monitor the running
+    /// of the kernel.
+    type WatchDog: watchdog::WatchDog;
+
     /// The kernel calls this function to tell the chip to check for all pending
     /// interrupts and to correctly dispatch them to the peripheral drivers for
     /// the chip.
@@ -101,6 +106,9 @@ pub trait Chip {
     /// chip.
     fn scheduler_timer(&self) -> &Self::SchedulerTimer;
 
+    /// Returns a reference to the implementation for the WatchDog on this chip.
+    fn watchdog(&self) -> &Self::WatchDog;
+
     /// Returns a reference to the implementation for the interface between
     /// userspace and kernelspace.
     fn userspace_kernel_boundary(&self) -> &Self::UserspaceKernelBoundary;

kernel/src/platform/watchdog.rs

@@ -0,0 +1,35 @@
+//! Interface for configuring a watchdog
+
+/// A trait for implementing a watchdog in the kernel.
+/// This trait is called from the `kernel_loop()` code to setup
+/// and maintain the watchdog timer.
+/// It is up to the specific `Chip` how it will handle watchdog interrupts.
+pub trait WatchDog {
+    /// This function must enable the watchdog timer and configure it to
+    /// trigger regulary. The period of the timer is left to the implementation
+    /// to decide. The implementation must ensure that it doesn't trigger too
+    /// early (when we haven't hung for example) or too late as to not catch
+    /// faults.
+    /// After calling this function the watchdog must be running.
+    fn setup(&self) {}
+
+    /// This function must tickle the watchdog to reset the timer.
+    /// If the watchdog was previously suspended then this should also
+    /// resume the timer.
+    fn tickle(&self) {}
+
+    /// Suspends the watchdog timer. After calling this the timer should not
+    /// fire until after `tickle()` has been called. This function is called
+    /// before sleeping.
+    fn suspend(&self) {}
+
+    /// Resumes the watchdog timer. After calling this the timer should be
+    /// running again. This is called after returning from sleep, after
+    /// `suspend()` was called.
+    fn resume(&self) {
+        self.tickle();
+    }
+}
+
+/// Implement default WatchDog trait for unit.
+impl WatchDog for () {}

kernel/src/sched.rs

@@ -14,6 +14,7 @@ use crate::ipc;
 use crate::memop;
 use crate::platform::mpu::MPU;
 use crate::platform::scheduler_timer::SchedulerTimer;
+use crate::platform::watchdog::WatchDog;
 use crate::platform::{Chip, Platform};
 use crate::process::{self, Task};
 use crate::returncode::ReturnCode;
@@ -292,12 +293,15 @@ impl Kernel {
         ipc: Option<&ipc::IPC>,
         _capability: &dyn capabilities::MainLoopCapability,
     ) {
+        chip.watchdog().setup();
         loop {
+            chip.watchdog().tickle();
             unsafe {
                 chip.service_pending_interrupts();
                 DynamicDeferredCall::call_global_instance_while(|| !chip.has_pending_interrupts());
 
                 for p in self.processes.iter() {
+                    chip.watchdog().tickle();
                     p.map(|process| {
                         self.do_process(platform, chip, process, ipc);
                     });
@@ -313,7 +317,9 @@ impl Kernel {
                         && !DynamicDeferredCall::global_instance_calls_pending().unwrap_or(false)
                         && self.processes_blocked()
                     {
+                        chip.watchdog().suspend();
                         chip.sleep();
+                        chip.watchdog().resume();
                     }
                 });
             };