Skip to content

Multicore Architecture and Design Decisions

Jump to bottom
Zakaria Madaoui edited this page Aug 17, 2026 · 6 revisions

Multicore Architecture and Design Decisions

This document details the mechanisms that make RTICX multicore-safe on the RP2040 (dual-core Cortex-M0+) distribution. It covers memory sharing, the SPSC queue, cross-core interrupt pending, async/await task integration, and the design choices behind them.

Memory model

The RP2040 has two Cortex-M0+ cores sharing the same physical address space (264 KiB of SRAM). RTICX application compiles into a single binary (not separate per-core firmware). Both cores execute from the same code and share the same .data/.bss segments.

This means:

  • There are no aliasing concerns across cores. any core can read or write any statically allocated memory.
  • Data races are prevented by the framework's design: each data structure is either (a) accessed exclusively through critical sections, (b) split into per-core exclusive regions, or (c) guarded by atomics and the interrupt hierarchy.

SPSC queue (rticx-spsc)

Why no atomics

The SPSC queue used for software-task message inputs (rticx-spsc) uses plain usize read/write indices: no AtomicUsize, no memory barriers, no critical sections. This is safe because:

  1. Producer-Consumer split: Queue::split() creates a Producer and a Consumer that each hold an exclusive &mut reference to the queue. The producer only mutates buffer[...] and write_idx; the consumer only mutates read_idx.

  2. Single-core dispatch: Under normal use, a task's queue is only accessed from a single core, the core that owns the task's dispatcher ISR.

  3. Cross-core ordering via SIO FIFO ISR: When cross_spawn writes to a remote core's queue, the sequence is:

    core-1 writes to core-0's queue (producer side)
core-1 calls cross_core::pend_irq(n)   → writes to SIO FIFO
                                        → triggers SIO_IRQ_PROC0 ISR on core 0
core-0's SIO_IRQ_PROC0 ISR runs        → NVIC::pend(Interrupt(n)) on core 0
core-0's dispatcher ISR drains queue   (consumer side)

    The SIO FIFO ISR runs at maximum NVIC priority on the receiving core, so it acts as a sequencing barrier: by the time the dispatcher ISR runs (lower priority), the SIO ISR has already completed, ensuring the write to the queue is visible.

Caveat

This ordering argument relies on the SIO FIFO interrupt being pended and handled before any lower-priority interrupt on the receiving core. This is guaranteed by NVIC's priority-based preemption. If the SIO FIFO ISR priority were lowered, a race between cross_spawn writing the queue and the dispatcher draining it could occur. The RP2040 distribution configures SIO_IRQ_PROC{0,1} at priority 0 (max) to prevent this.

Cross-core interrupt pending

The RP2040's NVIC (Nested Vectored Interrupt Controller) is private per core. NVIC::pend(interrupt) on core 0 only sets the pending bit in core 0's NVIC; core 1 is unaffected.

To deliver an interrupt to another core, we use the SIO FIFO, two hardware FIFOs (one per direction) shared between the cores. The protocol:

  1. cross_core::pend_irq(irq: u16) writes the interrupt number to the SIO FIFO TX register:

    let sio = unsafe { &(*rp2040_hal::pac::SIO::PTR) };
cortex_m::interrupt::free(|_| {
    if sio.fifo_st().read().rdy().bit() {
        sio.fifo_wr().write(|wr| unsafe { wr.bits(irq as u32) });
        Ok(())
    } else {
        Err(FullFifoErr) // FIFO is full; caller must retry
    }
})

  2. The SIO FIFO write triggers an interrupt on the receiving core (SIO_IRQ_PROC0 or SIO_IRQ_PROC1). The handler reads the interrupt number from the RX FIFO and pends it locally:

    fn SIO_IRQ_PROC0() {
    if let Some(signal) = cross_core::get_pended_irq() {
        NVIC::pend(signal);  // now pending on core 0
    }
}

  3. The receiving core's dispatcher ISR (lower priority) eventually runs and drains the task queue.

FIFO size limitation

The SIO FIFO is 8 words deep. If it fills up, pend_irq returns Err(FullFifoErr). The framework currently does not handle this: a full FIFO indicates a pathological condition where the receiving core is saturated with cross-core pends. In practice, cross-core task spawning is limited by the SPSC queue depth, which bounds the rate of cross-core pends.

Async task integration on multicore

The waker cross-core problem

When an async task is polled, the generated code creates a waker whose wake() calls a pend function. The pend function needs to trigger the dispatcher ISR on the task's owning core so the task future gets re-polled.

The problem: a waker can be invoked from any core. Consider:

Core 1 calls Sender::send() on a channel whose Receiver lives on Core 0.
  → send_footer() wakes the receiver_waker
  → waker calls wake_pend_fn(DMA_IRQ_0)   // wants to pend DMA_IRQ_0 on Core 0
  → but we're executing on Core 1!
  → NVIC::pend(DMA_IRQ_0) fires on Core 1's NVIC → Core 0 never sees it

How the waker knows the right interrupt

At codegen time (rticx-async-pass/src/codegen/mod.rs), each async task gets a dedicated wake function that captures the exact dispatcher interrupt and per-core pend function:

// Generated (conceptual):
fn __rticx_internal__MyAsyncTask__wake() {
    let exec = unsafe { recover_slot(__rticx_async_MyAsyncTask, &PTR) };
    exec.set_pending();                                           // AtomicBool, Release
    __rticx_wake_irq_pend_core0(pac::Interrupt::DMA_IRQ_0); // the issue
}

The interrupt number (DMA_IRQ_0) is known at compile time from the dispatcher assignment in rticx-async-pass/src/analyze.rs. The pend function (__rticx_wake_irq_pend_core0) is per-core. The only unknown is which core the function is being called from at runtime.

Solution: runtime core-ID check in generate_wake_pend_fn

The AsyncPassBackend::generate_wake_pend_fn override for RP2040 checks the current core via the SIO CPUID register and selects the appropriate pend mechanism:

fn generate_wake_pend_fn(&self, core: u32, mut empty_body_fn: ItemFn) -> ItemFn {
    let body = parse_quote!({
        let current_core = unsafe {
            (*rp2040_hal::pac::SIO::PTR).cpuid().read().bits()
        };
        if current_core == #core {
            // Same core: direct NVIC pend (fast path)
            rticx_rp2040::export::NVIC::pend(irq_nbr);
        } else {
            // Different core: go through SIO FIFO
            use rticx_rp2040::export::InterruptNumber;
            let _ = rticx_rp2040::export::cross_core::pend_irq(
                irq_nbr.number()
            );
        }
    });
    empty_body_fn.block = Box::new(body);
    empty_body_fn
}

The set_pending() on ExecSlot uses AtomicBool with Release ordering, so the flag is visible across cores regardless of pend path.

Design choice: why not always use the FIFO

Using cross_core::pend_irq for everything would make all waker pends go through the FIFO (even same-core ones), adding overhead and FIFO contention. The runtime core-ID check adds a single SIO register read (~2 cycles) to avoid this in the common case (same-core wake).

Channel (rticx-async): why send() returns a future

The Channel<T, N> in rticx-async is a bounded MPSC queue with N slots. Two API variants are provided:

Method Type Behavior
try_send(&mut self, val: T) Result<(), TrySendError<T>> Non-blocking. Returns Err(Full) if no free slot.
send(&mut self, val: T) impl Future<Output = Result<(), NoReceiver<T>>> Async. If no free slot, registers the sender in a wait queue and yields.

The send() async variant exists because the channel may be full. When the Receiver reads a value, it first checks the wait queue: if a sender is waiting, the freed slot is handed directly to it and the sender's waker is woken, avoiding a trip through the free queue. This is the standard pattern for bounded async channels (analogous to tokio::sync::mpsc::Sender::send).

Design choice: critical-section-based over lock-free

The channel uses critical_section::with() (disabling interrupts) rather than lock-free atomics. This keeps the implementation simpler and still meets embedded latency requirements because critical sections are short (a few pointer swaps). On the RP2040, the critical-section implementation uses the RP2040 HAL's hardware spinlocks for multicore safety.

Cross-core task spawning (cross_spawn)

The flow for spawning a software or async task from core A to core B:

Core A: Task::cross_spawn(input)
  1. Runtime core check: is the caller really on `spawn_by`? (see below)
  2. Write input into core B's SPSC queue (direct memory access)
  3. Call cross_core::pend_irq(dispatcher_irq_for_core_B)
     → writes irq number to SIO FIFO

Core B: SIO_IRQ_PROC handler fires (max priority)
  4. Read irq number from SIO FIFO RX
  5. NVIC::pend(read_irq) on Core B

Core B: Dispatcher ISR fires (lower priority)
  6. Drain queue, execute task

Runtime core check in spawn/cross_spawn

Cross-core spawning is enforced exclusively at runtime: the software/async task passes inject a core check into every generated spawn/cross_spawn using the distribution backend's current_core_id expression:

// generated at the top of cross_spawn:
if <current_core_id> != <spawn_by> {
    return Err(Some(input));
}

On the RP2040 the expression reads the cpuid register of the executing core ((*rp2040_hal::pac::SIO::PTR).cpuid().read()), which no user code can forge. A spawn therefore only succeeds when the caller genuinely executes on the configured core (spawn_by for cross_spawn, the task's own core for spawn). Applications with cross-core tasks fail to compile when the distribution backend does not provide current_core_id, so multicore distributions cannot silently skip the guard. Single-core targets have no cross-core tasks and simply return None.

Design choice: why not use the SIO FIFO for data

The SIO FIFO could theoretically carry data payloads, but this would require the framework to manage FIFO depth, handle fragmentation if payloads exceed the 32-bit word size, and serialize/deserialize all inputs. Using shared memory for the data and the FIFO only for signalling (interrupt number) is simpler, faster, and keeps the SPSC queue as the single source of truth.

Summary of cross-core safety mechanisms

Component Mechanism Rationale
rticx-spsc Queue SIO FIFO ISR acts as sequencing barrier Writes happen before interrupt delivery; no CPU-level atomics needed
cross_core::pend_irq SIO FIFO + priority-0 SIO ISR forwards to NVIC NVIC is per-core; FIFO is the only cross-core doorbell on RP2040
ExecSlot (async runtime) AtomicBool via portable-atomic (AcqRel/Release) Emulated inside a critical section on M0+ (no native atomics); set_pending called from any core, poll called from owning core
Channel (async runtime) critical_section::with() (spinlock on RP2040) + SeqCst fences Simpler than lock-free; critical sections are short (pointer swaps)
Waker pend Runtime core-ID check → local NVIC or SIO FIFO Avoids FIFO contention for same-core wake; correct for cross-core wake
spawn/cross_spawn Runtime core-ID check against core/spawn_by Only code genuinely executing on the configured core can spawn; no forgeable compile-time token

Clone this wiki locally