Timer implementation fails on early wakes

[fko-kuptec]

I stumbled across this issue while working on our device's ESP32S3 firmware based on ESP-IDF using Rust. To handle delays within asynchronous workloads, I employ the timer implementation from async-io-mini. This way, I can just swap out async-io-mini with async-io and theoretically run the firmware on the development machine. That even works on macOS, where async-io-mini does not compile.

[fko-kuptec]

(Early Ctrl+Enter again 🙄)

[fko-kuptec]

The concrete issue now is, that under certain circumstances, awaiting an async_io_mini::Timer might never finish, because the timer is not woken up. It took me some time and a lot of debugging to find the root cause, but I am now quite sure to have found it:

The circumstances triggering the issue are an overrunning timer queue. If the embassy-time Generic Timer Queue is already full and another timer added, the implementation here will pop one timer from the queue and wake it too early. This is not be a problem in and of itself, since any future getting woken up too early should just return Pending and schedule another wake.

embassy_time_queue_utils::queue_generic::ConstGenericQueue

/// Schedules a task to run at a specific time, and returns whether any changes were made.
///
/// If this function returns `true`, the called should find the next expiration time and set
/// a new alarm for that time.
pub fn schedule_wake(&mut self, at: u64, waker: &Waker) -> bool {
    self.queue
        .iter_mut()
        .find(|timer| timer.waker.will_wake(waker))
        .map(|timer| {
            if timer.at > at {
                timer.at = at;
                true
            } else {
                false
            }
        })
        .unwrap_or_else(|| {
            let mut timer = Timer {
                waker: waker.clone(),
                at,
            };

            loop {
                match self.queue.push(timer) {
                    Ok(()) => break,
                    Err(e) => timer = e,
                }

                self.queue.pop().unwrap().waker.wake();
            }

            true
        })
}

The implementation of async_io_mini::Timer here, however, only ever schedules another wake, iff the previously scheduled waker has gotten invalid in the meantime (would not wake anymore). If that is not the case, meaning the previous waker is still valid and would wake, then the timer does not reschedule it and thus will never get woken up again.

async_io_mini::timer::Timer

fn fired_at(&mut self, cx: &mut Context<'_>) -> Option<Instant> {
    let when = self.when?;

    if when > Instant::now() {
        let ticks = Self::ticks(&when);

        if let Some(ticks) = ticks {
            if self
                .waker
                .as_ref()
                .map(|waker| !waker.will_wake(cx.waker()))
                .unwrap_or(true)
            {
                self.waker = Some(cx.waker().clone());
                embassy_time_driver::schedule_wake(ticks, cx.waker());
            }
        } else {
            self.set_never();
        }

        None
    } else {
        Some(when)
    }
}

[fko-kuptec]

I assume, in the special case of my firmware I am running into the timer queue limit, because I am sending out camera frames via WIFI, and the sending of each frame is limited by a timeout.¹ With lets say 12 frames per second, each frame with a timeout of 3s, that's something like 3x12=36 timer schedules per client in just 3 seconds. It seems like that I am able to work around it by just using the largest available queue with 128 timer slots. However, I still think that it would be better to adjust the timer implementation, so that it is able to handle early wakeups.

¹ This timeout is unfortunately needed, because if the WIFI client did silently disconnect without the ESP noticing, the camera thread would happily continue to stuff data into the TCP buffers until they are full. And if they are full, the asynchronous call to write_all will never return. At least that's the explanation if I recall it correctly. 😅