Adaptive Reminder Service - Proposal

[KSemenenko]

Adaptive Reminder Service for Orleans (related to #7573 )

Status: Draft — open for feedback

---

1  Problem Statement

Limitation	Impact in production
Fixed‑interval only	Cannot schedule “every weekday 09:00”, “last day of month”, etc.
Partition‑wide scan	A silo with 100 000+ reminders loads them all into RAM at startup → long boot & GC churn (issues #947, #457).
Missed tick is lost	A cluster outage or GC pause during the due window silently drops that tick.

---

2  Solution Overview

Adaptive Reminder Service replaces LocalReminderService.
It scales to hundreds of thousands of reminders, supports cron calendars, and guarantees at‑least‑once delivery while bounding CPU & memory.

Highlights

• Cron scheduling — six‑field UTC cron expressions (second precision).
• Adaptive buckets — schedule only reminders due within a look‑ahead window; bucket size auto‑scales with CPU, memory and active‑grain count.
• Three priorities — Critical, Normal, Background let the runtime guarantee timeliness for critical work while deferring background tasks under load.
• Per‑reminder missed‑tick action — choose FireImmediately, Skip (default) or Notify (stream event); a background repair job fixes severely overdue rows.
• Management grain — IReminderManagementGrain lists, counts, edits and repairs reminders.
• Startup helper — [RegisterReminder] attribute recreates required reminders on silo boot if missing.
• Backward compatible — existing APIs compile unchanged; storage migration is additive.

---

3  Data Model

public enum ReminderPriority : byte
{
    Critical   = 0,
    Normal     = 1,
    Background = 2
}

public enum MissedReminderAction : byte
{
    FireImmediately = 0,
    Skip            = 1,
    Notify          = 2
}

public sealed class ReminderEntry
{
    // identifiers
    public string   ServiceId    { get; init; }
    public GrainId  GrainId      { get; init; }
    public string   ReminderName { get; init; }

    // schedule
    public DateTime  StartUtc   { get; init; }
    public TimeSpan? Period     { get; init; } // null ⇒ cron
    public string?   Cron       { get; init; }

    // runtime
    public DateTime             NextDueUtc  { get; set; }
    public DateTime?            LastFireUtc { get; set; }
    public ReminderPriority     Priority    { get; set; } = ReminderPriority.Normal;
    public MissedReminderAction Action      { get; set; } = MissedReminderAction.Skip;
}

Storage: add columns Cron, NextDueUtc, LastFireUtc, Priority, Action; index (NextDueUtc, Priority).

---

4  Configuration (ReminderOptions)

public class ReminderOptions
{
    public TimeSpan LookAheadWindow { get; set; } = TimeSpan.FromMinutes(5);
    public int      BaseBucketSize  { get; set; } = 1_024;
    public TimeSpan PollInterval    { get; set; } = TimeSpan.FromSeconds(30);
    public bool     EnablePriority  { get; set; } = true;
}

Adaptive Bucket Formula

BucketSize = BaseBucketSize × CPU × MemoryFactor × GrainFactor

CPU          = max(1, Environment.ProcessorCount / 4)
MemoryFactor = max(0.25, 1 − MemoryLoad%)
GrainFactor  = min(1.0, 50_000 / ActiveGrainCount)

Example (16 cores, 60 % memory load, 40 000 grains) → 1 024 × 4 × 0.4 × 1 ≈ 1 640 concurrent timers.

---

5  API Additions

public interface IReminderService
{
    // interval‑based
    Task<IGrainReminder> RegisterOrUpdateReminder(
        GrainReference grain, string name,
        TimeSpan due, TimeSpan period,
        ReminderPriority     priority = ReminderPriority.Normal,
        MissedReminderAction action   = MissedReminderAction.Skip);

    // cron‑based
    Task<IGrainReminder> RegisterOrUpdateReminder(
        GrainReference grain, string name,
        string cron,
        ReminderPriority     priority = ReminderPriority.Normal,
        MissedReminderAction action   = MissedReminderAction.Skip);
}

IGrainReminder now exposes Cron, Priority, Action.

---

6  Scheduling Algorithm

1. Poll storage every PollInterval; stream rows with NextDueUtc ≤ now + LookAheadWindow.
2. Sort & bucket by Priority, then NextDueUtc; bucket size uses adaptive formula.
3. Queue buckets to a Channel<ReminderEntry>.
4. Workers consume channel under a RateLimiter (permits = CPU × 4).
5. Execute • overdue → apply Action; • otherwise delay until due, then deliver.
6. Persist new NextDueUtc & LastFireUtc.
7. Repair job (every minute) fixes rows with NextDueUtc < now − LookAheadWindow.

---

7  Management Grain

public interface IReminderManagementGrain : IGrainWithGuidKey
{
    Task<IEnumerable<ReminderEntry>> UpcomingAsync(TimeSpan horizon);
    Task<IEnumerable<ReminderEntry>> ListForGrainAsync(GrainId grainId);
    Task<int>                        CountAllAsync();

    Task SetPriorityAsync(GrainId grainId, string name, ReminderPriority     p);
    Task SetActionAsync  (GrainId grainId, string name, MissedReminderAction a);
    Task RepairAsync     (GrainId grainId, string name);
    Task DeleteAsync     (GrainId grainId, string name);
}

---

8  [RegisterReminder] Attribute (BAD IDEA or not so bad 🙃)

[AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = false)]
public sealed class RegisterReminderAttribute : Attribute
{
    public string               Name     { get; }
    public TimeSpan?            Due      { get; }
    public TimeSpan?            Period   { get; }
    public string?              Cron     { get; }
    public ReminderPriority     Priority { get; }
    public MissedReminderAction Action   { get; }

    // interval‑based    public RegisterReminderAttribute(
        string name,
        double dueSeconds,
        double periodSeconds,
        ReminderPriority     priority = ReminderPriority.Normal,
        MissedReminderAction action   = MissedReminderAction.Skip)
    {
        Name     = name;
        Due      = TimeSpan.FromSeconds(dueSeconds);
        Period   = TimeSpan.FromSeconds(periodSeconds);
        Priority = priority;
        Action   = action;
    }

    // cron‑based
    public RegisterReminderAttribute(
        string name,
        string cron,
        ReminderPriority     priority = ReminderPriority.Normal,
        MissedReminderAction action   = MissedReminderAction.Skip)
    {
        Name     = name;
        Cron     = cron;
        Priority = priority;
        Action   = action;
    }
}

At startup Orleans invokes IReminderService.RegisterOrUpdateReminder for each attribute — no temporary grain activation.

[RegisterReminder("MonthlyReport", "0 0 0 1 * *", priority: ReminderPriority.Critical)]
[RegisterReminder("QuickPing",     30, 30)]
public sealed class StatsGrain : Grain, IRemindable
{
    public Task ReceiveReminder(string name, TickStatus _) => Task.CompletedTask;
}

---

9  Integration Steps

1. Schema upgrade — add columns & index.
2. Register AddAdaptiveReminderService() in ISiloBuilder.
3. Update reminder providers (ADO, Azure Table, etc.) for new fields.
4. Document cron syntax, priority semantics, missed‑action matrix.

---

10  Open Questions

Topic	Decision
Cron parser	Bundle NCrontab (MIT) vs. internal lightweight parser
Stream Id for Notify	Fixed GUID vs. configurable via ReminderOptions
Bucket constants	Fine‑tune via performance benchmarks

[rkargMsft]

Overall, it may be a good start to add and maintain the optional NextDueUtc that is only used for:

• Filtering out all Reminders with far out dates to limit memory usage and load times
• For Reminders with NextDueUtc < UtcNow, calculate and store the next future tick (this would match the existing behavior for now)
  while continuing the existing behavior for existing Reminders without NextDueUtc.

Further feedback

Some of this can be built on top of the existing Reminder systems (cron, missed tick recovery) with code and not requiring any changes to the existing providers.

The partial loading of only Reminders that will be due soon does seem to required some additional information like the NextDueUtc to be added to the persisted data model. That's an additional persistence operation per Tick, but it does seem necessary for this functionality. Additionally, it could be kept backward compatible by always pulling Reminders that have a null NextDueUtc which would maintain the current behavior.

Similarly for back-compat, having the priority be an arbitrary byte value but with helper values at Critical = 100, Normal = 0, Background = -100 would allow for an existing ReminderEntry to default to Normal = 0 to maintain existing functionality, as well as provider finer priority control if more tiers of priority are required for exceptional scenarios.

For look-ahead interval, it seems like only the polling frequency would be needed to be considered. If it's looking every 5 minutes then looking 2 or 3 times that amount to the future would allow for 2 or 3 polling failures, respectively, before it would miss a far future reminder. Allowing that multiplier to be configured could be sufficient.

One clarification, the Channel for backpressure doesn't actually limit the local Silo's ReceiveReminder processing as the "responsible" silo is just responsible for sending a message to the specific grain instance which could be already activated on another Silo or will get activated somewhere according to the configured Placement strategy.

[KSemenenko]

@rkargMsft thanks for feedback!

I also want to have backward compatibility, I love your idea about enum, to make normal as 0 which is smart!

For channels - I mean we have to limit number of notifications per time frame, to manage load. And prevent suituation when 100k activations happens simultaneously. Anyway we will use existiee implantation and table will be sparred between silos. And in this case I except each silo anyway will create limited number of events.

And second - do not load all remainders in memory. So for now I come up with precalculated next time. Which can be flag for batch or flag for missing event

[benjaminpetit]

HI @KSemenenko, @ReubenBond and I have a design for RemindersV2, and I am a bit late posting the proposal on GH.

I really like the idea of the attribute and the cron scheduling pattern. I think we are in the same timezone, maybe we could sync tomorrow or later this week to see ideas that we could merge into the new design.

[KSemenenko]

Hi @benjaminpetit of course! I would love to discuss deeper remainders!