RELAY

A self-hosted, laptop-scale event pipeline engine — think Zapier, but local, inspectable, and replayable.

Events come in from watched folders, cron schedules, HTTP posts, and RSS feeds. YAML-defined pipelines filter, transform, branch, delay, and call out — then write files, append logs, or hit local webhooks. Every event is journaled to disk, every failure retries with exponential backoff into a dead-letter inbox, and any moment in history can be replayed through an edited pipeline to see what would have happened.

Zero cloud. Zero paid services. One jar + one browser tab.

Live flow graphs: per-node throughput, retries in amber, events animating along the edges.

---

The signature feature: Time Travel

Pick any historical event from the journal. Edit the pipeline YAML in a side-by-side editor. Hit dry-run replay: the event's original payload runs through your edited definition, and the UI diffs the old execution against the new one, node by node — payload diffs, routing changes, and what each sink would have written.

Dry-run guarantees:

• Sinks never fire. They return "would write data/out/… (312 B)" descriptions instead.
• The network is never touched. http-call steps replay the response captured in the original run's journal; if there is none, they mark themselves skipped.
• History is never rewritten. No journal entries, no idempotency marks, no files.

When the dry-run looks right, one click applies the edited YAML to the live pipeline (hot-swap, persisted to disk).

A historical event replayed through an edited definition — toUpperCase changed to toLowerCase, and the diff shows exactly what would change, including the sink's would-be output.

Everything else

Sources	file-watch (with write-settle detection), cron (5-field expressions or every: 30s), http ingest endpoint, rss poller
Steps	filter, transform, branch, delay, http-call (loopback-guarded), image-resize — plus per-node when: guards
Sinks	write-file (content or file copy), append-log, webhook
Expressions	SpEL in a sandboxed context (no type refs, no constructors) with payload.*, meta.*, event.*, #now, #nowIso, #date, and ${…} templating
Reliability	per-step retries with exponential backoff + jitter, dead-letter queue with resume-at-failed-node replay, persistent idempotency keys, append-only JSONL journal
UI	animated SVG flow graphs (hand-rolled, no chart libs), live per-node throughput + green/amber/red health over SSE, event journal browser, dead-letter inbox with one-click replay, time travel tab
Queue	Redis (RPUSH/BLPOP) when a local Redis answers PING, bounded in-memory queue otherwise — same interface, chosen at boot

More screenshots — dead-letter inbox and the event journal

Every dead letter carries the exact payload that entered the failing node; replay resumes there, not at the start.

The journal: every event's step-by-step timeline with payload snapshots after each node.

Architecture

flowchart LR
    subgraph Sources
        FW[file-watch] --> ING
        CR[cron] --> ING
        HT[HTTP ingest] --> ING
        RS[rss poller] --> ING
    end

    ING{{"ingest<br/>(idempotency check)"}} --> Q[["queue<br/>redis | in-memory<br/>(bounded = backpressure)"]]
    Q --> W["workers<br/>(virtual threads)"]

    W -->|"node loop:<br/>filter → transform → branch → … → sinks"| OUT[(files / logs / webhooks)]
    W -->|every attempt + payload snapshot| J[("journal<br/>JSONL on disk")]
    W -->|"failure → backoff × N"| Q
    W -->|retries exhausted| DLQ[("dead letters<br/>payload-at-failure")]
    DLQ -->|"replay resumes<br/>AT failed node"| Q

    J --> TT["time travel<br/>dry-run replayer"]
    TT -.->|never touches| OUT

    W --> SSE[/SSE stream/] --> UI["React UI<br/>flow graphs · journal · DLQ · time travel"]
    J --> API[/REST API/] --> UI

Loading

Quickstart

Requirements: JDK 21+, Node 18+. (Maven comes via the wrapper; Redis is optional.)

# 1. build the UI into the server's static resources, then the jar
cd ui && npm install && npm run build && cd ..
./mvnw clean install          # mvnw.cmd on Windows

# 2. run — loads every YAML in ./pipelines
java -jar relay-server/target/relay-server-1.0.0.jar

# 3. make it move
./scripts/demo.sh             # scripts\demo.ps1 on Windows

Open http://localhost:8080. The demo script drops sample images into the watched folder and fires HTTP pings whose downstream call is deliberately flaky (~35%) — so within a minute you'll see dots flowing through the graphs, amber retry states, and usually a dead letter or two waiting in the inbox for a one-click replay.

For UI development: cd ui && npm run dev → http://localhost:5173 (proxies /api to :8080).

The three example pipelines

1. image-sorter — drop an image into data/inbox/images; it's resized to ≤640px and filed into large/ medium/ small/ by original width (branch step), with an audit log line per image.
2. rss-digest — polls an RSS feed every 20s (RELAY's own synthetic feed by default, so it works offline), keyword-filters titles, appends matches to data/out/digest/digest-<date>.md.
3. http-ping — POST /api/ingest/http-ping with any JSON; it's enriched, sent through the flaky demo endpoint (the retry/DLQ showcase), and logged.

Plus a bonus heartbeat cron pipeline, mostly so the cron source has something to do on the graph page.

Pipeline YAML reference

id: my-pipeline                # required, [a-zA-Z0-9_-]+
name: "Readable Name"          # optional
source:
  type: file-watch             # file-watch | cron | http | rss
  dir: data/inbox              # …source-specific config inline
steps:
  - id: keep-big               # optional (auto-assigned), must be unique
    type: filter               # filter | transform | branch | delay | http-call | image-resize
    expr: "payload.sizeBytes > 10000"
    when: "meta.route != 'small'"          # optional guard on ANY node
    retry:                                  # optional, per node
      maxAttempts: 5           # default 3
      backoffMs: 250           # default 250
      multiplier: 2.0          # default 2.0 (exponential) + 0–10% jitter
      maxBackoffMs: 15000
sinks:
  - type: append-log           # write-file | append-log | webhook
    path: "data/out/${#date}/events.log"
    line: "${event.ts} ${payload.title}"

Expressions see payload (mutable event data), meta (routing metadata, e.g. set by branch), event (id, ts, source, pipeline, key, replay) and time variables #now / #nowIso / #date. Any string config value supports ${…} interpolation.

Adding a custom node type is one call: registry.registerStep("my-step", MyStep::new) with a one-method interface — image-resize is implemented exactly this way.

Design decisions

Backpressure — block, don't drop. The queue between sources and workers is bounded (10k items by default). When it fills, push blocks, which stalls the source threads themselves: the file watcher stops emitting, the HTTP ingest request waits. For a laptop-scale tool this is the honest failure mode — slow down the producer rather than silently shedding events or growing an unbounded buffer until the heap dies. Workers are virtual threads, so a delay step sleeping or an HTTP call blocking costs a few KB, not a pinned OS thread.

Idempotency — natural keys, persisted, bypassable. Every source derives a natural key (file name+size+mtime, RSS guid, cron fire-time, SHA-256 of an HTTP body or an explicit X-Idempotency-Key header) rather than a random UUID, so re-delivery of the same fact is recognized even across restarts — the seen-set is an append-only file per pipeline. Duplicates are journaled as DUPLICATE (visible, not silent) and dropped. Explicit replays set a flag that bypasses the check, because "run it again on purpose" is the whole point of a replayable engine.

Retries resume at the failed node, not at the start. A WorkItem carries resumeAt + the payload snapshot taken entering the failed node. Retrying node 4 never re-executes nodes 1–3 — which matters enormously once sinks are involved: a flaky webhook after a write-file sink must not rewrite the file on every attempt. Dead letters store the same snapshot, so DLQ replay is surgical: fix the downstream, click replay, and only the broken tail of the pipeline runs.

Journal — append-only JSONL, the single source of truth. Three record types (received, step, terminal) per event, one file per day, flushed per line. Crash-safe by construction (a torn final line is skipped on load), greppable with standard tools, and sufficient to reconstruct any execution — which is precisely what time travel does. The in-memory index is just a cache of the newest 2000 events; cold lookups scan the files.

Dry-run is a first-class execution mode, not a mock. Steps and sinks receive a dryRun flag plus a lookup into the original run's recorded results. Each component decides what fidelity it can offer: transforms run for real (pure), http-call replays its recorded response, sinks describe their action. That keeps the diff honest — the parts that can be exact are exact.

Redis is an optimization, not a dependency. The engine is written against a four-method EventQueue interface. At boot RELAY pings localhost:6379 and uses Redis if it answers (durable queue, restart-surviving), else the in-memory implementation. No config required for either path; relay.queue=memory|redis forces one.

Security posture of a local tool. Expressions run in SpEL's sandboxed SimpleEvaluationContext — no type references, no constructors, no reflection. Outbound http-call/webhook targets are loopback-only unless -Drelay.allow-external-http=true is set. The RSS poller (read-only) is exempt. XML parsing has doctypes disabled (no XXE).

Project layout

relay-core/     engine: sources, steps, sinks, queue, journal, DLQ,
                idempotency, retry scheduler, time travel — plain Java 21,
                no Spring except spring-expression
relay-server/   Spring Boot host: REST API, SSE hub, Redis queue impl,
                demo endpoints, serves the built UI
ui/             React + TypeScript + Vite; hand-rolled SVG flow graphs,
                blueprint aesthetic, no component/chart libraries
pipelines/      the example pipeline YAMLs (loaded at boot)
scripts/        demo.sh / demo.ps1 + sample images
data/           runtime state (journal, DLQ, keys, in/out boxes) — gitignored

Tests

./mvnw test

Twenty tests cover the interesting failure machinery end-to-end, not just happy paths: retry sequencing and attempt counts, dead-lettering with payload-at-failure capture, DLQ replay resuming at the failed node, idempotency across engine restarts, journal reconstruction from disk, time-travel dry runs verified side-effect-free, the cron parser, the expression sandbox, and a full HTTP → engine → sink → time-travel integration test against a running server.

API sketch

Method	Path
POST	/api/ingest/{pipelineId}	HTTP source ingest (202 + event id)
GET	/api/events?limit=100	journal summaries
GET	/api/events/{id}	full execution record
POST	/api/events/{id}/replay	full re-run through the live pipeline
GET	/api/dlq	dead-letter inbox
POST	/api/dlq/{id}/replay	resume at the failed node
POST	/api/timetravel/replay	{eventId, yaml} → original + dry-run executions
GET/PUT	/api/pipelines[/{id}]	list / validate + hot-swap + persist
GET	/api/stream	SSE: transits, terminals, dead letters, 1 Hz stats
GET	/api/demo/feed.xml, POST /api/demo/flaky	self-contained demo endpoints

License

MIT