Proposal: Consolidate Redis, RabbitMQ, and Prefect onto NATS JetStream as the Unified Event Backbone

[sdktr]

Summary

This proposal suggests replacing the current combination of:

• Redis (cache / ephemeral coordination state)
• RabbitMQ (messaging / pub-sub)
• Prefect (task orchestration / workflow state)

with a unified event backbone based on NATS JetStream.

The goal is to simplify infrastructure, reduce operational complexity, and establish a consistent foundation for event-driven execution, task coordination, and system state management.

⸻

Current Architecture Problems

Today, responsibilities are split across multiple systems:

Redis

• Used for caching and ephemeral coordination state
• Also used for lightweight runtime flags and deduplication patterns
• Introduces multiple inconsistent state semantics across the system

RabbitMQ

• Handles pub/sub and async task dispatch
• Separate routing and delivery model from other systems
• Operational overhead of queue management and topology

Prefect

• Provides task orchestration and workflow state management
• Requires PostgreSQL for persistence
• Introduces a central orchestration dependency outside the event system

⸻

Core Issue

These systems overlap significantly:

Concern Redis RabbitMQ Prefect
Messaging partial primary indirect
Task execution no partial yes
State tracking partial no yes

This results in:

• duplicated coordination mechanisms
• fragmented system semantics
• multiple sources of truth for state and execution
• increased operational overhead

⸻

Proposed Architecture: NATS JetStream as Unified Event Backbone

We propose consolidating messaging, task execution, and lightweight state coordination into NATS JetStream.

⸻

NATS Responsibilities

1. Event Bus (RabbitMQ replacement)

• Subject-based pub/sub (device.updated, task.completed.*)
• Durable event streams via JetStream
• Consumer-based queue semantics via durable subscriptions

⸻

2. Task Queueing + Execution (Prefect replacement)

Tasks are modeled explicitly as events:

• task.requested.*
• task.execute.*
• task.completed.*
• task.failed.*

Workers:

• consume tasks via JetStream consumers
• operate statelessly
• rely on ACK/NAK semantics for reliability and retry

⸻

3. Lightweight State Coordination (Redis replacement)

JetStream Key-Value store is used for:

• idempotency tracking
• execution guards (deduplication)
• ephemeral coordination state
• runtime flags and small shared state

This replaces Redis in coordination and state-tracking roles only, not general-purpose caching patterns.

⸻

Event Router (Control Plane Component)

A lightweight deterministic service is introduced:

Responsibilities:

• Translate domain events into task intents
• Define system behavior in a centralized, inspectable layer
• Emit task events based on rules or configuration

Example:

device.updated →
task.requested.recalculate_fields
task.requested.sync_external_systems

This replaces implicit orchestration logic previously embedded in Prefect flows or worker chaining.

⸻

Optional Future Extension: Object Storage via JetStream

As a separate and optional evolution path, JetStream can also be extended to support artifact storage using its Object Store capabilities.

This is not part of the core Redis/RabbitMQ/Prefect replacement, but a potential future consolidation step.

Potential use cases:

• Generated configuration artifacts
• Rendered templates
• Task execution outputs
• Intermediate pipeline results
• Logs or serialized payloads from workflows

Benefits (future consideration only):

• Artifacts become event-system-native
• No external object storage dependency for workflow outputs
• Tight coupling between task lifecycle and produced artifacts

This capability is treated as:

a future enhancement layer, not a requirement for initial migration

⸻

New Execution Model

domain event (device.updated)
↓
JetStream event stream
↓
event-router (policy layer)
↓
task.requested.*
↓
workers (stateless execution)
↓
task.completed.*
↓
(optional future) GraphDB dependency resolver
↓
(optional future) JetStream Object Store for artifacts

⸻

Key Design Shift

From orchestration → event expansion

Instead of:

Prefect defines and executes workflows

We move to:

Events define intent, the router expands intent into tasks, and workers execute atomic units of work

⸻

Why NATS JetStream

1. Unified event backbone

NATS replaces:

• RabbitMQ (messaging)
• Prefect (task coordination layer)
• Redis (lightweight state coordination)

Using consistent primitives:

• subjects (routing)
• streams (durability)
• consumers (execution control)
• KV store (state coordination)

⸻

2. Reduced system complexity

• fewer moving parts
• fewer integration boundaries
• single operational model for event-driven behavior

⸻

3. Native event-driven semantics

JetStream provides:

• at-least-once delivery
• replayability
• backpressure handling
• horizontal scaling via consumers

⸻

Future Extension: DAG Execution (GraphDB-backed)

Once task execution stabilizes, dependency management can be introduced via GraphDB.

Dependency model:

Task A → depends_on → Task B

Dependency resolver service:

• listens to task.completed.*
• evaluates readiness conditions
• emits task.ready.*

This enables:

• deterministic DAG execution
• fan-in / fan-out workflows
• replayable dependency graphs

without introducing a traditional workflow engine.

⸻

Migration Plan

Phase 1 — Replace Redis + RabbitMQ

• Migrate pub/sub to JetStream
• Replace Redis coordination usage with JetStream KV where applicable

⸻

Phase 2 — Replace Prefect execution layer

• Convert workflows into event-driven tasks
• Introduce task.requested.*
• Workers become stateless executors

⸻

Phase 3 — Introduce event-router

• Centralize event → task mapping logic

⸻

Phase 4 — Introduce GraphDB dependency model (optional)

• Enable DAG semantics via dependency resolution service

⸻

Phase 5 (optional future) — Artifact storage integration

• Evaluate JetStream Object Store for:
  • task outputs
  • generated artifacts
  • workflow results

⸻

Expected Benefits

Infrastructure simplification

• 3 systems → 1 unified event backbone

Operational reduction

• fewer failure domains
• fewer coordination layers
• reduced system complexity

Architectural consistency

• events, tasks, and state share a single semantic model

Future extensibility

• clean path toward DAG execution via GraphDB
• optional artifact lifecycle integration via JetStream Object Store

⸻

Risks / Considerations

1. At-least-once delivery semantics

Requires:

• idempotent task design
• deduplication via task IDs or KV guards

⸻

2. Loss of built-in workflow orchestration (Prefect)

Mitigated via:

• event-router abstraction
• optional DAG layer in GraphDB

⸻

3. Object storage lifecycle (future only)

If adopted later:

• retention policies must be defined
• artifact cleanup strategy required
• versioning conventions needed

⸻

Conclusion

Adopting NATS JetStream as the unified event backbone enables:

• elimination of Redis, RabbitMQ, and Prefect as separate systems
• consolidation of eventing and task execution into a single model
• a cleaner, event-native architecture
• a controlled future path toward DAG execution and optional artifact storage integration

This results in a simplified, consistent system where:

events, tasks, and state coordination share one unified substrate, with optional extension into artifact storage and DAG semantics over time.