[Feature] Grain Reminders (persistent timers that survive deactivation)

[pathosDev]

Size / Priority

• Size: L
• Reference: Orleans Grain Reminders.

Rationale

actor-ts has volatile timers (scheduler.scheduleOnce, scheduler.scheduleAtFixedRate) and per-actor TimerScheduler. These cancel automatically when:

• The actor stops.
• The node restarts.
• The cluster member is downed.

Orleans Reminders are durable: they persist across grain deactivation, node restart, and cluster topology changes. A reminder scheduled for "5 minutes from now" survives a full cluster restart in the middle and still fires on schedule.

Use cases:

• Subscription renewal — actor scheduled to fire on day-30 of a subscription.
• Workflow timeouts — "approve within 24h or escalate" — must survive deployments.
• Scheduled batch jobs — fire every Monday 09:00, even if the cluster was down all weekend.
• Cleanup reminders — "delete this temporary entity in 30 days".

Without durable reminders, every long-lived scheduled action requires user code that:

1. Persists "I want X to happen at time T" to the journal.
2. On actor recovery, computes "what's still pending?" and re-schedules volatile timers.

That's boilerplate. A first-class Reminders extension absorbs it.

Reference: what Orleans does

public class OrderGrain : Grain, IRemindable
{
    private IGrainReminder reminder;

    public async Task ScheduleCleanup()
    {
        reminder = await this.RegisterOrUpdateReminder(
            reminderName: "cleanup",
            dueTime: TimeSpan.FromDays(30),
            period: TimeSpan.MaxValue);   // one-shot
    }

    public async Task ReceiveReminder(string reminderName, TickStatus status)
    {
        if (reminderName == "cleanup") await this.Cleanup();
    }

    public async Task CancelCleanup()
    {
        await this.UnregisterReminder(reminder);
    }
}

Orleans stores reminders in a per-cluster IReminderTable (typically a SQL or Azure-Tables backend); a per-cluster "reminder service" polls the table for due reminders + dispatches them to the relevant grain (activating it if necessary).

Design sketch — actor-ts equivalent

// src/reminders/Reminder.ts (new)

export interface ReminderEntry {
  /** Unique within the (persistenceId, reminderName) pair. */
  readonly persistenceId: string;
  readonly reminderName: string;
  readonly dueAt: number;             // wall-clock ms
  readonly period: number | null;     // periodic interval; null for one-shot
  readonly payload?: unknown;         // optional JSON-safe payload
}

export interface ReminderStore {
  /** Persist or update. */
  save(entry: ReminderEntry): Promise<void>;
  /** Remove by (persistenceId, reminderName). */
  remove(persistenceId: string, reminderName: string): Promise<void>;
  /** All reminders due at or before `now`. */
  loadDueAt(now: number, limit?: number): Promise<ReadonlyArray<ReminderEntry>>;
  /** Single lookup. */
  load(persistenceId: string, reminderName: string): Promise<ReminderEntry | undefined>;
}

// src/reminders/RemindersExtension.ts

export class RemindersExtension implements Extension {
  constructor(
    private readonly system: ActorSystem,
    private readonly store: ReminderStore,
    private readonly options?: { readonly tickIntervalMs?: number },
  ) {}

  /**
   * Schedule a reminder; survives deactivation and restart.  Re-scheduling
   * the same (persistenceId, name) updates the existing reminder.
   */
  async schedule(persistenceId: string, name: string, dueIn: number, options?: {
    readonly period?: number;
    readonly payload?: unknown;
  }): Promise<void>;

  async cancel(persistenceId: string, name: string): Promise<void>;
}

Tick loop (runs on the cluster's reminder-service singleton):

private async tick(): Promise<void> {
  const due = await this.store.loadDueAt(Date.now(), this.batchSize);
  for (const r of due) {
    // Resolve the target actor (via PersistentActor's pid → sharding/lookup)
    const target = await this.resolveActorByPid(r.persistenceId);
    if (target) {
      target.tell(new ReminderFired(r.reminderName, r.payload));
    }
    // Reschedule or remove
    if (r.period !== null) {
      await this.store.save({ ...r, dueAt: r.dueAt + r.period });
    } else {
      await this.store.remove(r.persistenceId, r.reminderName);
    }
  }
}

export class ReminderFired {
  constructor(public readonly name: string, public readonly payload?: unknown) {}
}

User code:

class OrderActor extends PersistentActor<...> {
  // ... existing ...
  override async onRecoveryComplete(state: State) {
    if (state.status === 'pending' && state.createdAt < Date.now() - 30 * 86400e3) {
      // Schedule the cleanup reminder if not already
      await this.system.extension(RemindersId).schedule(
        this.persistenceId, 'cleanup', 30 * 86400e3,
      );
    }
  }

  override async onCommand(state: State, cmd: Cmd | ReminderFired) {
    if (cmd instanceof ReminderFired && cmd.name === 'cleanup') {
      await this.persistAll([{ kind: 'cleaned-up' }], () => {});
    }
  }
}

Integration with existing actor-ts subsystems

• Backends: ReminderStore impls for SQLite (SqliteReminderStore), Cassandra (CassandraReminderStore), InMemory (test only). Mirror the journal-store pattern.
• Cluster: the reminder-service runs as a ClusterSingleton (one per cluster — coordinator-style). On singleton handoff, the new leader takes over the tick loop.
• Sharding/lookup: reminders fire by persistenceId. The dispatcher uses cluster.actorAt(persistenceId) or shard-routing to deliver to the target actor.
• Failure semantics: at-least-once delivery. Reminders that fail to deliver (target actor unreachable) are retried on the next tick.
• Metrics: reminders_total{outcome}, reminders_due_total, reminders_delivery_latency_seconds.

Out of scope / non-goals

• Cron-style schedules — Orleans uses simple (dueAt, period). Cron is sugar; user code can compute the next dueAt.
• Exactly-once delivery — at-least-once is honest; targets handle dedup if needed (idempotency keys).
• Per-grain reminders limit — out of scope for phase 1; defer to ops-level monitoring.
• Cross-cluster reminders — only within one cluster.

Open design questions

1. Storage backend mandatory? Yes — that's the point. InMemory variant is test-only.
2. Reminder-service placement: cluster singleton (one per cluster) vs. partitioned (sharded by hash(persistenceId)). Singleton is simpler; partitioned scales better at >100K reminders. Recommend: singleton phase 1; partitioned phase 2.
3. Tick interval: default 5s? 1s? Trade-off: lower = sharper timing, higher load. Recommend: 5s default, configurable.
4. Reminder name uniqueness scope: per-actor (sketch) or per-cluster? Per-actor is more intuitive (analogous to per-grain in Orleans). Recommend: per-actor.
5. Delivery target type: any ActorRef (sketch) or only PersistentActors (since they have stable persistenceIds)? Recommend: any ref with stable address — PersistentActor or sharded actor.

Test plan

1. One-shot reminder — schedule for 100ms ahead; verify ReminderFired arrives within ~5s + tick interval.
2. Periodic reminder — schedule with period 200ms; verify multiple fires.
3. Survives actor deactivation — schedule; deactivate actor; tick fires; actor reactivated + receives ReminderFired.
4. Survives node restart — schedule on node A; kill node A; reminder still fires on node B's singleton-takeover.
5. Cancel — schedule + cancel; tick does NOT fire.
6. Update — schedule with name X dueAt 100; re-schedule X dueAt 50; only the 50 fires.
7. At-least-once on delivery failure — target actor down; reminder retries on next tick.
8. Metrics — counters + histograms emit.
9. Singleton handoff — kill the singleton-holding node; verify new singleton continues firing pending reminders.
10. Backend parity — same tests against SQLite + Cassandra stores.

Acceptance criteria

• RemindersExtension + ReminderStore interface.
• SqliteReminderStore + CassandraReminderStore + InMemoryReminderStore (test).
• ReminderFired message class.
• Reminder-service singleton with tick loop.
• At-least-once delivery semantics.
• Metrics emitted.
• Documentation: "Durable reminders for long-running actors".
• Test suite covers all 10 cases.
• CHANGELOG entry under "New: Grain Reminders (Orleans-style durable timers)".

Pre-implementation checklist

• Joint review of the design sketch.
• Resolve the 5 open design questions.
• Choose storage-backend priority (likely SQLite first — broadest applicability).
• Implementation order: ReminderStore interface + InMemory + SQLite; singleton tick service; PersistentActor integration; Cassandra backend last.