Notifier runtime: central routing, delivery state, settings, retries, and AO-app desktop handoff

[whoisasx]

Summary

Build the central notifier runtime that sits on top of the durable notification foundation from #54. This issue owns routing, settings, delivery state, retry/lease mechanics, and the backend handoff model for AO-app desktop delivery.

This issue does not implement Slack, webhook, Discord, or any external notifier sink. It also does not shell out to legacy desktop fallback tools. Desktop delivery means the AO Electron app consumes backend delivery records and shows native notifications itself.

Depends On

• Notifier foundation: durable notification domain, storage, CDC, and lifecycle enqueue #54

Context

After #54, lifecycle reactions should create durable rows in notifications. This issue adds the central runtime that decides which built-in delivery surfaces should receive each notification and tracks delivery status durably.

The target model is:

notifications row
  -> central notifier route resolver
  -> notification_deliveries rows
  -> AO app / dashboard consumes through API + SSE
  -> delivery status updated in SQLite

Important boundary:

• Dashboard is a read model over notifications, not a plugin.
• Desktop is AO-app delivery only. The backend does not call OS notification APIs or shell commands.
• Future external sinks can later use this delivery table and dispatcher model without changing lifecycle.

Goals

• Add notification settings with safe defaults.
• Add route resolution for built-in surfaces only: dashboard and desktop.
• Add delivery-state storage for AO-app desktop and future sinks.
• Add retry/lease/status mechanics in the central notifier runtime.
• Add a background router/dispatcher that converts eligible notification rows into delivery rows.
• Make delivery state durable across daemon restarts.
• Keep delivery failures isolated from lifecycle.

Non-Goals

• No Slack sink.
• No webhook sink.
• No Discord sink.
• No external notifier plugin execution.
• No terminal-notifier.
• No osascript.
• No Linux notify-send.
• No Windows PowerShell toast.
• No direct OS notification from Go backend.
• No dashboard notification center UI.

Proposed Files

Add:

backend/internal/notification/settings.go
backend/internal/notification/routing.go
backend/internal/notification/manager.go
backend/internal/notification/dispatcher.go
backend/internal/notification/retry.go
backend/internal/notification/delivery.go
backend/internal/notification/store.go
backend/internal/notification/settings_test.go
backend/internal/notification/routing_test.go
backend/internal/notification/dispatcher_test.go
backend/internal/notification/retry_test.go
backend/internal/storage/sqlite/migrations/0003_notification_deliveries.sql
backend/internal/storage/sqlite/queries/notification_deliveries.sql
backend/internal/storage/sqlite/notification_delivery_store.go
backend/internal/storage/sqlite/notification_delivery_store_test.go
backend/notifier_wiring.go

Modify:

backend/internal/config/config.go
backend/internal/cdc/event.go
backend/main.go
backend/lifecycle_wiring.go, only if #54 does not already pass the notifier through cleanly
backend/internal/storage/sqlite/gen/*, regenerated by sqlc

Settings Model

Add notification settings to config/runtime defaults.

Suggested Go shape:

type NotificationConfig struct {
    Enabled bool
    Dashboard DashboardNotificationConfig
    Desktop DesktopNotificationConfig
    Routing NotificationRoutingConfig
    Retry NotificationRetryConfig
}

type DashboardNotificationConfig struct {
    Enabled bool
    Limit int
}

type DesktopNotificationConfig struct {
    Enabled bool
    Priorities []ports.Priority
    SoundPriorities []ports.Priority
}

type NotificationRoutingConfig struct {
    // priority -> built-in surfaces; initially dashboard and desktop only
    Priorities map[ports.Priority][]string
}

type NotificationRetryConfig struct {
    MaxAttempts int
    BaseDelay time.Duration
    MaxDelay time.Duration
    LeaseTTL time.Duration
    BatchSize int
}

Defaults:

notifications.enabled = true
dashboard.enabled = true
dashboard.limit = 50
desktop.enabled = true
desktop.priorities = [urgent, action]
desktop.soundPriorities = [urgent]
routing.urgent = [dashboard, desktop]
routing.action = [dashboard, desktop]
routing.warning = [dashboard]
routing.info = [dashboard]
retry.maxAttempts = 5
retry.baseDelay = 1s
retry.maxDelay = 5m
retry.leaseTTL = 30s
retry.batchSize = 50

Routing Rules

Initial built-in route names:

dashboard
desktop

Rules:

• Dashboard is always represented by the notification row itself. It does not require a delivery row unless we decide to track dashboard-specific read/dismiss delivery later.
• Desktop route creates a notification_deliveries row with sink = 'ao-app'.
• If notifications are globally disabled, do not create delivery rows, but do not delete existing notification rows.
• If desktop is disabled, create no new desktop delivery rows.
• If a priority is not desktop-eligible, create no desktop delivery row.
• Unknown routes are recorded as skipped only if they are explicitly configured. Do not implement their delivery.
• No provider-specific data should enter lifecycle events.

Delivery Schema

Add backend/internal/storage/sqlite/migrations/0003_notification_deliveries.sql.

Suggested schema:

CREATE TABLE notification_deliveries (
    id                  TEXT PRIMARY KEY,
    notification_id     TEXT NOT NULL REFERENCES notifications(id) ON DELETE CASCADE,
    notification_seq    INTEGER NOT NULL,
    project_id          TEXT NOT NULL REFERENCES projects(id),
    session_id          TEXT NOT NULL REFERENCES sessions(id),

    route_name          TEXT NOT NULL,
    sink                TEXT NOT NULL,
    destination_key     TEXT NOT NULL DEFAULT '',
    request_json        TEXT NOT NULL DEFAULT '{}' CHECK (json_valid(request_json)),

    status              TEXT NOT NULL CHECK (status IN ('queued','leased','sent','retry_wait','failed','skipped','cancelled')),
    attempts            INTEGER NOT NULL DEFAULT 0,
    max_attempts        INTEGER NOT NULL DEFAULT 5,
    next_attempt_at     TIMESTAMP NOT NULL,
    lease_owner         TEXT NOT NULL DEFAULT '',
    lease_expires_at    TIMESTAMP,

    last_error_code     TEXT NOT NULL DEFAULT '',
    last_error          TEXT NOT NULL DEFAULT '',
    external_id         TEXT NOT NULL DEFAULT '',

    created_at          TIMESTAMP NOT NULL DEFAULT (datetime('now')),
    updated_at          TIMESTAMP NOT NULL DEFAULT (datetime('now')),
    delivered_at        TIMESTAMP,

    UNIQUE(notification_id, route_name, destination_key)
);

CREATE INDEX idx_notification_deliveries_due
    ON notification_deliveries(status, next_attempt_at, lease_expires_at, created_at);

CREATE INDEX idx_notification_deliveries_notification
    ON notification_deliveries(notification_id, status);

CREATE INDEX idx_notification_deliveries_project
    ON notification_deliveries(project_id, created_at DESC);

Add optional attempt audit table if implementation cost is acceptable in this issue:

CREATE TABLE notification_delivery_attempts (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    delivery_id TEXT NOT NULL REFERENCES notification_deliveries(id) ON DELETE CASCADE,
    attempt_no INTEGER NOT NULL,
    status TEXT NOT NULL CHECK (status IN ('started','sent','retryable_failed','failed')),
    started_at TIMESTAMP NOT NULL,
    finished_at TIMESTAMP,
    error_code TEXT NOT NULL DEFAULT '',
    error TEXT NOT NULL DEFAULT '',
    response_json TEXT NOT NULL DEFAULT '{}' CHECK (json_valid(response_json)),
    UNIQUE(delivery_id, attempt_no)
);

CDC Events

Add delivery CDC events:

notification_delivery_created
notification_delivery_updated

Update backend/internal/cdc/event.go.

Trigger payloads should be compact and useful for clients:

{
  "id": "del_...",
  "notificationId": "ntf_...",
  "routeName": "desktop",
  "sink": "ao-app",
  "status": "queued",
  "attempts": 0,
  "lastErrorCode": "",
  "lastError": ""
}

Runtime Components

Manager

notification.Manager should own high-level runtime operations:

type Manager struct {
    store Store
    settings SettingsProvider
    clock func() time.Time
    logger *slog.Logger
}

Responsibilities:

• route newly created notification rows
• create desktop delivery rows when eligible
• expose store methods needed by API/Electron later
• keep lifecycle independent from route details

Dispatcher / Router

The dispatcher should run as a daemon goroutine.

Responsibilities:

• periodically route un-routed notifications into delivery rows
• release expired leases on startup and on tick
• claim due desktop delivery rows if backend-side processing is needed
• for AO-app desktop, leave actual OS delivery to Electron; backend should expose claim/mark APIs and update statuses

Important: because AO app is the actual desktop sink, the backend dispatcher should not attempt native desktop delivery. It should only manage route/delivery state.

Retry / Lease

State transitions:

queued -> leased
leased -> sent
leased -> retry_wait
retry_wait -> leased
leased -> queued       when lease expires
leased -> failed       after max attempts or permanent error
queued -> skipped      if route disabled/unavailable
queued -> cancelled    if notification is archived/cancelled before delivery

Backoff:

base = 1s
max = 5m
jitter = +/-20%
max attempts = 5
lease TTL = 30s

For AO-app handoff, retries apply when Electron reports transient delivery failure or never completes a lease before expiry.

Store Interface

type Store interface {
    ListUnroutedNotifications(ctx context.Context, limit int) ([]domain.Notification, error)
    MarkNotificationRouted(ctx context.Context, id domain.NotificationID, at time.Time) error

    EnqueueDelivery(ctx context.Context, row DeliveryRow) (DeliveryRow, bool, error)
    ClaimDueDeliveries(ctx context.Context, sink string, owner string, now time.Time, limit int, lease time.Duration) ([]DeliveryRow, error)
    ReleaseExpiredDeliveryLeases(ctx context.Context, now time.Time) (int, error)
    MarkDeliverySent(ctx context.Context, id string, externalID string, at time.Time) error
    MarkDeliveryRetry(ctx context.Context, id string, errCode string, errMessage string, next time.Time) error
    MarkDeliveryFailed(ctx context.Context, id string, errCode string, errMessage string, at time.Time) error
    MarkDeliverySkipped(ctx context.Context, id string, reason string, at time.Time) error
}

If #54 does not include routed_at on notifications, add either:

notifications.routed_at

or use existence of delivery rows plus idempotent EnqueueDelivery to avoid double routing. Prefer routed_at for efficient scans.

AO-App Desktop Handoff

Backend responsibilities in this issue:

• create notification_deliveries rows for eligible desktop notifications
• track status as queued, leased, sent, retry_wait, failed
• expose store methods for later API endpoints
• do not call Electron APIs directly
• do not shell out to desktop notifier tools

Electron responsibilities are tracked in the API/dashboard/AO-app issue.

Wiring

Add backend/notifier_wiring.go.

Suggested shape:

type notifierStack struct {
    Manager *notification.Manager
    done <-chan struct{}
}

func startNotifier(ctx context.Context, cfg config.Config, store *sqlite.Store, log *slog.Logger) (*notifierStack, error) {
    mgr := notification.NewManager(store, notification.SettingsFromConfig(cfg), log)
    done := mgr.Start(ctx)
    return &notifierStack{Manager: mgr, done: done}, nil
}

func (s *notifierStack) Stop() {
    <-s.done
}

Startup order in backend/main.go:

open store
start CDC
start notifier runtime
start lifecycle with notification manager/enqueuer
start HTTP

Shutdown order:

cancel context
stop lifecycle/reaper
stop notifier runtime
stop CDC
close store

Tests

Routing tests:

• default routes: dashboard all priorities, desktop urgent/action
• desktop disabled creates no desktop deliveries
• info priority creates no desktop delivery by default
• explicit empty route suppresses route
• unknown route creates skipped record only when explicitly configured

Store tests:

• delivery enqueue idempotency
• claim due rows in stable order
• lease expiry releases rows
• mark sent
• retry increments attempts and sets next attempt
• max attempts moves to failed
• skipped rows are terminal
• CDC events for create/update

Retry tests:

• exponential backoff capped at max
• jitter stays inside bounds
• permanent error classification does not retry
• transient error classification retries

Runtime tests:

• startup releases expired leases
• dispatcher routes existing unrouted notifications after restart
• dispatcher stops cleanly on context cancel
• one failed delivery does not block routing other notifications

Integration tests:

• notification row from Notifier foundation: durable notification domain, storage, CDC, and lifecycle enqueue #54 becomes desktop delivery when priority is urgent
• action priority also creates desktop delivery
• info priority only remains dashboard/read-model
• archived notification cancels queued desktop delivery if implemented in this issue

Acceptance Criteria

• Central notifier runtime starts and stops cleanly with the daemon.
• Notification rows are routed according to built-in settings.
• Desktop-eligible notifications create durable ao-app delivery rows.
• Delivery leases and retries survive daemon restart.
• Dashboard remains storage-backed and does not require a delivery row.
• No Slack, webhook, Discord, or external sink code is added.
• No legacy desktop fallback shell commands are used anywhere.
• cd backend && go test ./... passes.
• cd backend && go vet ./... passes.

Risks / Follow-Ups

• Exactly-once desktop delivery is impossible if Electron shows an OS notification and crashes before marking sent. Delivery idempotency and notification id should limit duplicate impact.
• Notification retention is still out of scope and should be a later issue.
• Per-project notification settings may be needed later; start global and keep schema extensible.
• Future external sinks should plug into this delivery model without changing lifecycle or notification storage.