Skip to content

v0.1.0 release

Choose a tag to compare

View all tags
@FDK0901 FDK0901 released this 24 Feb 07:48
· 41 commits to main since this release
v0.1.0
a4f7cd6

dureq v0.1.0 — Initial Release

dureq (Durable Redis Execution Queue) is a distributed, actor-based job scheduling and workflow orchestration system built in Go. It replaces the traditional pull-based polling model with a push-based actor architecture powered by Hollywood, delivering low-latency dispatch and seamless horizontal scaling.

Highlights

  • Push-based actor model — no polling overhead; jobs are dispatched instantly via Redis Pub/Sub + actor messaging
  • Cluster-native — add nodes dynamically; workload is automatically distributed based on capacity and task-type affinity
  • Batteries included — jobs, recurring schedules, DAG workflows, batch processing, middleware, priority queues, retry policies, DLQ, and real-time monitoring all in one package

Core Features

Job Scheduling

Schedule Type Description
Immediate Fire-and-forget
One-Time Execute once at a specific time
Duration Repeat at fixed intervals
Cron Standard cron expressions
Daily / Weekly / Monthly Calendar-based with timezone support
  • Overlap policies — ALLOW_ALL, SKIP, BUFFER_ONE, BUFFER_ALL
  • Catchup / Backfill — recover missed executions within a configurable window
  • Priority queues — Low / Normal / High / Critical with per-job override

Retry & Error Handling

  • Exponential backoff with jitter (configurable max attempts, initial delay, max delay, multiplier)
  • Error classification: Retryable, Non-Retryable, Rate-Limited (client-specified backoff)
  • Dead Letter Queue (DLQ) — auto-move after exhausting retries
  • Unique keys — prevent duplicate enqueuing

Batch Processing

  • Onetime preprocessing — shared setup step before per-item execution
  • Chunk-based dispatch with sliding-window backpressure (ChunkSize items in flight)
  • Failure policies — ContinueOnError (default) or FailFast
  • Per-item retry and batch-level retry
  • Real-time progress tracking with per-item result storage

Workflow Orchestration

  • DAG-based task dependencies — define workflows as directed acyclic graphs
  • Automatic cycle detection and validation
  • OrchestratorActor advances tasks in dependency order
  • Workflow-level retry and execution timeout

Handler System

  • Pattern-based routing (mux) — "email.send" (exact), "image.*" (wildcard)
  • Global middleware via srv.Use() + per-handler middleware
  • HandlerFunc (error only) and HandlerFuncWithResult (output + error)
  • Context utilities — GetJobID, GetTaskType, GetAttempt, GetHeaders, GetNodeID, ReportProgress, etc.
  • Automatic panic recovery

Architecture

Actor System (Hollywood)

Singleton actors (Redis-based leader election with automatic failover):

Actor Role
SchedulerActor Scans due schedules, dispatches jobs, detects orphaned runs, backfills missed executions
DispatcherActor Routes jobs to nodes based on idle capacity and task-type affinity
OrchestratorActor Advances workflow DAG and batch state machines

Per-node actors:

Actor Role
WorkerSupervisorActor Manages worker pool, enforces MaxConcurrency
WorkerActor Executes a single job handler with heartbeat, lock, and progress reporting
EventBridgeActor Persists results, forwards domain events
NotifierActor Subscribes to Redis Pub/Sub for instant job notifications
HeartbeatActor Maintains node liveness
ClusterGuardActor Monitors singleton health, triggers re-election

Distributed Coordination

  • Per-run distributed locks (Redis + Lua) with auto-extension
  • Heartbeat-based stale detection — orphaned runs are auto-recovered
  • Capacity-based routing — Dispatcher selects nodes with the most idle workers

Storage — Redis

  • Keys — job metadata, schedules, run state
  • Streams — persistent event log
  • Pub/Sub — real-time dispatch notifications
  • Lua scripts — atomic lock/leader operations

Monitoring & Observability

REST API (HTTP)

Full CRUD for jobs, workflows, batches, runs, schedules, nodes, queues, and DLQ. Pagination, filtering by status/task-type/tags, and bulk cancel/retry/delete operations.

gRPC API

37+ RPC methods with full feature parity to the HTTP API. Protocol reflection enabled.

WebSocket

Live event streaming at /api/ws with automatic polling fallback and connection status indication.

Domain Events (29+ types)

Job, workflow, batch, schedule, node, and leader lifecycle events — broadcast via Redis Pub/Sub and persisted to Redis Streams.

Tools

  • dureqd — Server daemon with HTTP (:8080) + gRPC (:9090) + WebSocket monitoring. YAML config via Koanf.
  • dureqctl — Interactive terminal UI (Bubble Tea) with 9 pages: Dashboard, Jobs, Workflows, Batches, Runs, Nodes, Schedules, Queues, DLQ.
  • durequi — React-based web dashboard with real-time WebSocket updates.

Client SDKs

Go Client

cli, _ := client.New(client.WithRedisURL("redis://localhost:6379"))

// Immediate job
job, _ := cli.Enqueue(ctx, "email.send", payload)

// Scheduled / recurring
job, _ := cli.EnqueueScheduled(ctx, "report.generate", payload, types.EnqueueOptions{
    Schedule: types.Schedule{Type: types.ScheduleCron, Cron: "0 */6 * * *"},
})

// Synchronous execution
result, _ := cli.EnqueueAndWait(ctx, "task.run", payload, 30*time.Second)

// Batch
batch, _ := cli.EnqueueBatch(ctx, types.BatchDefinition{...})

// Workflow
wf, _ := cli.EnqueueWorkflow(ctx, types.WorkflowDefinition{...})

Supports Redis standalone, Sentinel, and Cluster modes. Multi-tenancy via WithKeyPrefix().

Python Client

Type-safe bindings for all job, workflow, and batch operations.

Server Quick Start

srv, _ := server.New(
    server.WithRedisURL("redis://localhost:6379"),
    server.WithNodeID("node-1"),
    server.WithMaxConcurrency(10),
)

srv.Use(loggingMiddleware)

srv.RegisterHandler(types.HandlerDefinition{
    TaskType: "image.*",
    Timeout:  30 * time.Second,
    HandlerWithResult: imageHandler,
    Middlewares: []types.MiddlewareFunc{timingMiddleware},
})

srv.Start(ctx)

Configuration

Server Options

WithRedisURL, WithRedisPassword, WithRedisDB, WithRedisPoolSize, WithNodeID, WithListenAddr, WithClusterRegion, WithMaxConcurrency (default 100), WithShutdownTimeout (default 30s), WithKeyPrefix, WithLogger, WithLockTTL, WithLockAutoExtend, WithRedisSentinel, WithRedisCluster

dureqd YAML

dureqd:
  nodeId: ""
  apiAddress: ":8080"
  grpcAddress: ":9090"
  concurrency: 100
  prefix: ""

redis:
  url: "redis://localhost:6379"
  password: ""
  db: 0
  poolSize: 100

Examples

Example Demonstrates
festival/ Recurring scheduled jobs
festival_with_mux/ Pattern routing + global/per-handler middleware
onetimeat/ One-time delayed execution
batch/ Batch processing with onetime preprocessing
batch_with_mux/ Batch with mux handler pattern
workflow/ DAG workflow execution
workflow_with_mux/ Workflow with mux handler pattern
heartbeat_progress/ Progress reporting + custom heartbeat timeout
overlap_policy/ Overlap policy demonstrations

Requirements

  • Go 1.25+
  • Redis 7+

Key Dependencies

hollywood v1.0.5, rueidis v1.0.72, robfig/cron v3, sonic v1.15, bubbletea v1.3, koanf v2.3, grpc-go v1.60, nhooyr.io/websocket v1.8, ants v2.11, xid v1.6, vtprotobuf v0.6

Assets 15
Loading