core(actor): rebuild from kameo reference — Actor trait, lifecycle hooks, run-loop

[joeldsouzax]

Part of the core-rebuild epic — follows the shared rigor contract there.

Component

src/actor.rs, src/actor/spawn.rs, src/actor/kind.rs (~1,983 LOC). The Actor trait, lifecycle hooks (on_start/on_panic/on_stop), spawn, and the receive loop.

Learning & discussion

The loop is identity-agnostic — it survives the identity-first inversion unchanged in spirit (it just runs whatever bears the identity). A panic in a handle is actor-internal: caught and routed to on_panic; it is NOT observable via should_panic (see core-wiring latent-defects note) — probe via an on_panic observer.

Existing crates

tokio task/JoinHandle, tokio-util CancellationToken for cooperative abort (#55), futures.

Research — best algorithm/approach

Actor run-loop patterns; panic isolation (catch_unwind / task boundary); guaranteed ordering of lifecycle hooks; cooperative cancellation vs abort.

Testing — hard as fuck

Lifecycle (start → panic → restart → stop ordering; panic in on_start), sequence, defensive boundary. DST over scheduling + abort races. loom where atomics touch the run-state. Mutation: cargo-mutants, zero survivors.

Absorbs

#55 (tokio-util CancellationToken for task abort).

[joeldsouzax]

Research — run-loop / panic / cancellation (per the epic rigor contract, point 3)

Targeted survey of how other actor runtimes solve the four problems this card flags, benchmarked against the kameo oracle. Primary sources cited inline; every non-obvious claim is tied to a repo file/doc + version.

Landscape (7 systems × 4 axes)

	State model	Panic in handler	Run-loop / control plane	Startup msgs
kameo (oracle)	&mut self	catch_unwind in-loop → on_panic; default Break(Panicked), opt-in Continue (resume)	one task; startup_buffer VecDeque	buffered
ractor 0.15	framework-owned State (&mut State into hooks; struct is stateless &self)	catch_unwind → clean fail → supervisor rebuilds via pre_start	one task; separate priority channels signal>stop>supervision>mailbox; explicit Drain	buffered
xtra 0.6	&mut self	none — task dies, state lost, no hook	one task; merged priority mailbox; in-band Shutdown	buffered
actix 0.13	&mut self	none — task dies, stopping/stopped skipped	hand-written Future on !Send arbiter; bounded-16 mailbox; Running::{Stop,Continue} veto	ctx.wait gates mailbox
coerce 0.8	&mut self	none — task dies; Drop still notifies parent	plain while recv().await { handle().await }; unbounded mpsc; Stop is a FIFO msg	awaited before loop
OTP gen_server	returned NewState	let-it-crash → supervisor restart from init; mailbox discarded	sync init, mailbox buffers; terminate best-effort	buffered
Akka Typed	returned next Behavior	default stop; opt-in supervise(restart); drops offending msg, keeps rest of mailbox	behavior fn; PostStop vs Terminated split	buffered

Convergent finding — don't resume on a torn &mut self:

• std: &mut T is not UnwindSafe; a handler that panics mid-mutation leaves an invariant nothing re-checks — memory-safe but logically corrupt (kameo asserts past this with AssertUnwindSafe). — https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html
• catch_unwind is silently defeated by panic = "abort" — any restart guarantee built on it is profile-dependent. — https://doc.rust-lang.org/std/panic/fn.catch_unwind.html
• OTP/Akka get crash-consistency free because state only advances by returning a value from a completed handler — the thing in-place &mut self can't do.
• Only ractor + kameo catch in-loop; ractor uses it to fail-clean-then-rebuild, not resume. (The claim that ractor "preserves state across a panic" was investigated and found false.)

---

The four recommendations

1. Panic model → catch to observe, but recover by discard-and-rehydrate (not resume).
Keep catch_unwind so on_panic runs in-context and the panic is routed cleanly, but treat recovery as discard torn state + rebuild from the AID's event log / Args (OTP let-it-crash; stronger here because bombay is event-sourced + identity-first). Default stays Break. Either drop Continue (resume) or gate it behind an explicit "handler was transactional" contract and document it as a footgun. Pin panic = "unwind" and add a test that fails under abort.
Rationale: single-writer aggregate (CLAUDE.md rule #1 atomicity) must never keep serving a half-mutated self.

2. Restart lives in the supervisor (#120), not in the hook.
ControlFlow<StopReason> (continue/stop) cannot express OTP/Akka restart — that's a third outcome. Keep #116's hook signature continue/stop; reach restart via on_panic→stop→supervisor-rebuild. Restart strategies + intensity limits (one_for_one, MaxR/MaxT) belong in the supervision layer, not smuggled into the hook signature.

3. Cancellation (absorbs #55) → CancellationToken for graceful, JoinHandle::abort for hard kill.
Idiomatic split: cooperative graceful stop via CancellationToken (child-token-per-actor + DropGuard), hard kill via JoinHandle::abort; escalation = cancel → bounded grace → abort. Nobody surveyed uses a token for actor stop (coerce uses it only for timers) — this is a clean improvement over kameo's stop path. — https://tokio.rs/tokio/topics/shutdown, https://docs.rs/tokio-util/latest/tokio_util/sync/struct.CancellationToken.html

4. Drain-on-stop is unspecified everywhere — decide it explicitly.
Almost all frameworks abandon queued messages on stop; only ractor has an explicit Drain. For a single-writer aggregate this is correctness-visible (does a graceful stop finish enqueued commands or drop them?). Must be answered; may be co-decided with the mailbox/request cards.

Validated as-is (keep): kameo's async on_start + startup_buffer is the faithful port of OTP's "sync init, mailbox buffers, nothing dropped." coerce's while recv().await { handle().await } is the closest structural template — not actix's !Send Future state machine.

---

Sources

ractor 0.15 (ractor/src/actor.rs, actor_cell.rs) · xtra 0.6 (xtra/src/lib.rs, chan.rs) · actix 0.13.5 (actix/src/context_impl.rs, mailbox.rs, supervisor.rs) · coerce 0.8.12 @a2d3f19 (coerce/src/actor/lifecycle.rs, context.rs, refs/mod.rs) · OTP gen_server + supervisor principles · Akka fault-tolerance + actor-lifecycle · Pony actors (anti-pattern reference) · std catch_unwind/UnwindSafe · tokio JoinHandle/graceful shutdown · tokio-util CancellationToken

[joeldsouzax]

Resolved design decisions — panic / state / stop (from the run-loop research)

Follow-up to the run-loop research comment above. The four-framework survey + the OTP/Akka contrast + the durability-tier discussion resolve #116's three open decisions into one consistent package. Grounded in: OTP gen_server (crash→restart-from-init, no resume), Akka Typed (behavior-as-state; resume/restart/stop opt-in), and the tokio primitives brief (catch_unwind footguns, CancellationToken vs abort).

The package (all three decisions)

1. Panic recovery = catch to observe, then discard — no resume.
   Keep catch_unwind so a handler panic is observed and routed to on_panic; default Break (stop). On stop, the in-memory &mut self is discarded. bombay does not resume on a panicked &mut self (kameo's opt-in Continue). Rationale: &mut T is not UnwindSafe; a panic mid-mutation leaves torn state, and resuming witnesses it (std docs). Also pin panic = "unwind" and add a test that fails under abort (else catch_unwind is silently defeated).

2. State ownership = keep &mut self (NOT ractor's framework-owned State).
   The torn-state hazard that would justify the inversion is neutralised by decision 1 (discard, never resume). So the core stays idiomatic Rust and kameo-parity-clean. The single-writer boundary is enforced by the loop (one task, one mailbox, sequential handle), not by a type split.

3. Stop = finish current message, then stop (default); optional drain as a graceful mode; hard-kill (abort) never drains.
   Matches OTP external-shutdown / Akka context.stop (neither drains business messages). Cancellation is out-of-band via CancellationToken (cooperative graceful); JoinHandle::abort is the hard kill; escalation cancel → bounded grace → abort is the supervisor's (core(supervision): rebuild from kameo reference — links, death-watch, restart strategies #120). Non-drain is safe because delivery is nexus's guarantee (at-least-once + replay), so a dropped in-memory message is re-drivable — drain is a latency optimisation, not correctness.

Why this is the OTP posture, reached differently

bombay lands on OTP's posture — crash → rebuild, no resume, no drain — but reaches it through nexus event-sourcing at the layer above, not through a returned-state core. Akka keeps behavior-as-state + resume/restart because it is general-purpose and must serve the non-event-sourced majority of actors; recovery-by-replay is only its opt-in EventSourcedBehavior. bombay's core stays idiomatic &mut self and lets the durability layer own recovery (see #12 for the durability dial). That durability layer is nexus as a feature-gated in-crate dependency — not a separate adapter (decision on #66).

The core loop is deliberately TIER-AGNOSTIC

The loop runs "an actor holding some state" and is blind to how durable that state is (throwaway / in-memory aggregate / snapshot / full event-log — see #12). Discard-on-panic is correct for all tiers because the recoverable truth (the committed event-list) is held above the actor (by the supervisor / durability module), never on the discarded &mut self. So #116 does not learn about persistence, aggregates, or tiers — it only: catch panic → stop cleanly → surface the reason. What to do next (rebuild-from-Args, restore-from-snapshot, replay-from-log) is the supervisor (#120) + durability layer's decision (nexus, feature-gated in-crate per #66 — not a separate adapter).

The one deliberate OPEN item

Hard-gated Continue (resume) escape hatch — the only legitimate use is an ephemeral, stateful actor with expensive in-memory state and no log, where a provably-transactional handler failed (Akka keeps resume for exactly this replay-avoidance case). Leaning forbid it in the default path, add back only at a real second concrete need (house style: no speculative API). Recorded as open, not closed.

Cross-links

Durability dial / in-mem aggregate / which-failure-you-survive: #12. Supervision/restart policy + escalation: #120. Activation/passivation (uses this loop as the per-active-entity body): #20. Cancellation primitive absorbs #55.

[joeldsouzax]

Resolving the two open items — the unified model closes both

Both open items from the resolved-decisions comment above now have a forced answer once "every actor is a store-less-capable nexus aggregate" (#12/#66) is assumed.

1. on_panic resume (Continue) → FORBID it. Stop-only.

The only niche resume ever had was "an ephemeral actor with valuable in-memory state and no log, whose handler failed transactionally" — save the state by skipping the bad message. The store-less aggregate model removes that niche:

• On a handler panic, discard the torn &mut self, then re-fold the committed event-list (held above the actor, feat: queryable + persistent actor state via Zenoh storages #12) → back to the last committed state.
• That recovers exactly what resume would (all prior state, minus the offending command) with zero torn-state risk — so re-fold strictly dominates resume.
• The residual case (valuable in-memory state that is not modeled as folded events) doesn't exist by construction — every actor is an aggregate; unmodelled in-mem state is the thing we've designed away.

Decision: on_panic returns stop-only (ControlFlow::Break semantics); no Continue/resume variant in the surface. Matches OTP (no resume). Re-introduce only if a real non-aggregate case ever appears (house style: add at second concrete use).

2. Drain-on-stop → finish-current-then-stop is the default and the only mode in v1.

No drain mode built. Rationale: non-drain is safe because nexus owns delivery (at-least-once + replay; the subscription cursor re-reads the log), so an abandoned in-memory mailbox message is re-drivable — draining is a latency nicety, not correctness. Building a drain path adds an unbounded-shutdown failure mode for no correctness gain (YAGNI).

• Cooperative graceful stop (CancellationToken): finish the in-flight message, then exit. Queued messages abandoned (re-driven by nexus).
• Hard kill (JoinHandle::abort): no finish, no drain.
• Revisit a drain mode only at a concrete need.

Net — #116's panic/stop surface is now fully specified

on_panic = observe (catch_unwind) → stop-only; recovery is the durability layer's re-fold/rehydrate (not the loop's). Stop = finish-current-then-stop (cooperative) or abort (hard). panic = "unwind" pinned + tested. Keep &mut self. No open items remain on the panic/stop axis.

[joeldsouzax]

FINALIZED — Actor trait, hooks, run-loop (supersedes the open-items note above)

Three decisions researched across frameworks (OTP, Akka Typed, ractor, actix, kameo), the failure primitives (tokio/futures/std), and the primary literature (Armstrong 2003 thesis; Agha 1986; Pony / Deny Capabilities AGERE 2015; crash-stop session types ECOOP 2023). All three land off kameo, toward the OTP / let-it-crash model. The unifying principle: "the decider should not be the corpse" — a panicked actor's state is the suspect thing (Agha: a failed atomic step yields no valid successor behavior), so it gets a total "which way did I die" and a different process decides recovery.

Decision 1 — on_panic is infallible, stop-only

fn on_panic(&mut self, actor_ref: WeakActorRef<Self>, err: PanicError)
    -> impl Future<Output = ActorStopReason> + Send;

No Result, no ControlFlow::Continue. Resume was already forbidden; an infallible-total return bakes it in — the hook must always yield a terminal decision, can't "fail to decide." Cleanup-failure / panic-cause are StopReason data, not an error channel.
Grounding: OTP ignores terminate/2's return; Pony forbids partial (fallible) async behaviours — "too late to report this to the caller" once across the async boundary; Armstrong §4.3 "Let some other process do the error recovery."

Decision 2 — supervision + links leave the base Actor trait (→ #120)

supervision_strategy() and on_link_died() move to a separate Supervisor: Actor supertrait (see #120).
Grounding: OTP separate supervisor behaviour; Akka supervise wrapper; actix separate Supervised trait (3 of 4 separate; ractor/kameo the minority). Armstrong §4.3.2 / §5.2 names worker vs supervisor as distinct roles (supervisor code is generic/reusable, worker code app-specific). POLA: watch/link/kill is more authority than send → a separately-held capability. Also object-safety: on_link_died's impl Future + generic Self makes the base trait non-dyn-erasable; quarantining it keeps base Actor lean and erasable.

Decision 3 — on_stop Err is logged, never unwrapped

on_stop(..) -> Result<(), Error> keeps its shape, but a returned Err is logged + surfaced (diagnostic/observer), not unwrapped to a panic. The original stop reason is preserved for death-watch.
Grounding: OTP terminate/2 return ignored; Akka PostStop terminal cleanup; Rust Drop docs — a double-panic "will likely abort the program" (live on the shutdown path); and unwrapping replaces the real cause with a second panic, blinding watchers.

The locked trait

pub trait Actor: Sized + Send + 'static {
    type Args: Send;
    type Error: ReplyError;
    fn name() -> &'static str { type_name::<Self>() }

    fn on_start(args: Self::Args, actor_ref: ActorRef<Self>)          // construct/hydrate
        -> impl Future<Output = Result<Self, Self::Error>> + Send;

    fn on_message(&mut self, msg: BoxMessage<Self>, actor_ref: ActorRef<Self>,
                  tx: Option<BoxReplySender>, stop: &mut bool)
        -> impl Future<Output = Result<(), Box<dyn ReplyError>>> + Send { .. }

    fn on_panic(&mut self, actor_ref: WeakActorRef<Self>, err: PanicError)  // infallible, stop-only
        -> impl Future<Output = ActorStopReason> + Send
        { async move { ActorStopReason::Panicked(err) } }

    fn on_stop(&mut self, actor_ref: WeakActorRef<Self>, reason: ActorStopReason)
        -> impl Future<Output = Result<(), Self::Error>> + Send { async { Ok(()) } }
        // Err logged + surfaced; NEVER unwrapped; original `reason` preserved
}

Loop (kind.rs, not the trait)

startup_buffer (async on_start buffering — OTP sync-init semantics; nothing dropped) · catch_unwind per handle/hook (observe → on_panic) · CancellationToken in select! (graceful = finish-current-then-stop, no drain) + JoinHandle::abort (hard kill) · panic = "unwind" pinned + a test that fails under abort · keep &mut self (discard-on-panic neutralizes torn state; nexus event-fold supplies OTP crash-consistency at the durability layer, not the core — see #12/#66).

OPEN — message/reply erasure & allocation

The on_message signature carries BoxMessage<Self> / Box<dyn ReplyError> / BoxReplySender, inherited from the mailbox/message/reply cards (#112/#114/#115) — kameo's heterogeneous-mailbox model, which boxes every message. Whether to keep that or move to a homogeneous one-message-type-per-actor model (ractor's type Msg / Akka's Behavior[T], which has zero per-message allocation — the mailbox is a typed mpsc<Msg> and dispatch is a match) is under active review (allocate-last, rule #6). Decisions 1–3 above are independent of this and stand either way. Tracked against #114.

ActorStopReason (owned by #113)

#116 uses Normal | Killed | Panicked. SupervisorRestart | LinkDied are added by #120; PeerDisconnected by the zenoh/remote layer.

Sources

OTP gen_server/supervisor · Akka fault-tolerance · actix Supervised · ractor Actor · Armstrong 2003 thesis (§4.3, §4.3.2, §5.2, §8) · Agha 1986 Actors · Clebsch et al. Deny Capabilities AGERE 2015 (DOI 10.1145/2824815.2824816) · Barwell et al. ECOOP 2023 (arXiv:2305.06238) · std::ops::Drop · POLA (arXiv:1907.07154).

[joeldsouzax]

TIGHTENED — message erasure removed from trait + loop (resolves the "OPEN erasure" flag)

The OPEN erasure flag on the finalized note above is now resolved by #114's closed-enum message model. Because the mailbox carries a concrete Signal<A> (never a trait object), all per-message Box/dyn leaves the trait and the loop.

Trait — the tightening

• Add type Msg (the actor's closed message enum; = the nexus Command enum for aggregates).
• on_message collapses to handle:

  // BEFORE (kameo, boxed):
  fn on_message(&mut self, msg: BoxMessage<Self>, actor_ref: ActorRef<Self>,
                tx: Option<BoxReplySender>, stop: &mut bool)
      -> impl Future<Output = Result<(), Box<dyn ReplyError>>> + Send;
  // AFTER (bombay, typed):
  fn handle(&mut self, msg: Self::Msg, actor_ref: ActorRef<Self>, stop: &mut bool)
      -> impl Future<Output = Result<(), Self::Error>> + Send;

  Gone: BoxMessage<Self>, Option<BoxReplySender>, Box<dyn ReplyError>. A request's reply is sent by handle through the typed oneshot::Sender<Reply> embedded in its Msg variant (ractor RpcReplyPort pattern) — not returned, not boxed.

Loop (kind.rs) — the tightening

• Signal<A> = concrete envelope enum { Message(A::Msg), Stop, LinkDied{..}, /* system arms */ } — not Box<dyn DynMessage>.
• next() awaits over mpsc<Signal<A>>; dispatch is a match (no vtable, no downcast).
• catch_unwind wraps the typed handle(msg) call; startup_buffer: VecDeque<Signal<A>> holds typed signals.

Net

All per-message dyn/Box is removed from the core trait and loop. The only erasure left in the system is ref-level (Recipient/registry, #117/#119), which is not per-message and is built to hold a typed sender so even an erased ref sends without boxing the message. The zero-box property holds as long as ActorRef<A> stays monomorphic (mpsc::Sender<Signal<A>> inside) — this is precisely where ractor gives it up (type-erased ActorCell) and where bombay does not.

The finalized trait (updated)

pub trait Actor: Sized + Send + 'static {
    type Args: Send;
    type Msg: Send;                 // NEW — closed message/command enum
    type Error: ReplyError;
    fn name() -> &'static str { type_name::<Self>() }

    fn on_start(args: Self::Args, actor_ref: ActorRef<Self>)
        -> impl Future<Output = Result<Self, Self::Error>> + Send;

    fn handle(&mut self, msg: Self::Msg, actor_ref: ActorRef<Self>, stop: &mut bool)
        -> impl Future<Output = Result<(), Self::Error>> + Send;              // typed, unboxed

    fn on_panic(&mut self, actor_ref: WeakActorRef<Self>, err: PanicError)
        -> impl Future<Output = ActorStopReason> + Send                      // infallible, stop-only
        { async move { ActorStopReason::Panicked(err) } }

    fn on_stop(&mut self, actor_ref: WeakActorRef<Self>, reason: ActorStopReason)
        -> impl Future<Output = Result<(), Self::Error>> + Send { async { Ok(()) } }
        // Err logged + surfaced; never unwrapped; original reason preserved
}

Cross-links: message model #114 · mailbox #112 · reply #115 · supervisor trait #120 · ref erasure #117/#119.