BeamTrail: an OTP-native durable step runner using PostgreSQL event logs

[sherry255]

Hi,

I am building BeamTrail, an Erlang/OTP library for durable business process
execution. I would like design feedback from people who have operated OTP
systems, PostgreSQL-backed queues, workflow engines, or event-sourced systems.

Repository: https://github.com/sherry255/BeamTrail

BeamTrail is not a Temporal clone and not a job queue. The current goal is
smaller: an embedded OTP runtime for durable step/process execution in an
Erlang application that already runs PostgreSQL.

The basic model:

• active runs are supervised OTP processes;
• business progress is stored as an append-only PostgreSQL event stream;
• state is rebuilt by reducing events;
• snapshots and indexed projections are optimizations only;
• leases, fencing tokens, and expected sequence checks protect concurrent
  writers;
• a scanner recovers unfinished runs whose lease is missing or expired.

The split is intentional:

• OTP owns live execution, timers, process isolation, and cleanup;
• PostgreSQL owns durable history, append ordering, leases, fencing, and
  cross-process recovery.

The project started as a linear durable step runner. It now also has an optional
deterministic decider callback for one-command-at-a-time orchestration. The
decider path supports explicit step inputs, durable signals, durable timers, and
{wait, Reason}. There is an executable approval-deadline pattern built from
signals, timers, and waits.

Current implemented pieces:

• PostgreSQL durable adapter and memory test adapter;
• append-only event log and reducer;
• expected-sequence append checks;
• per-run PostgreSQL append lock;
• lease renewal and fencing tokens;
• supervised per-run gen_statem active runner;
• scanner recovery with indexed recoverable-run projections;
• retries, timeouts, crash-atomic failure decisions;
• cancel, park/resume, and manual requeue;
• version mismatch gates for in-flight attempt replay and decider workflows;
• durable signals and scanner-driven durable timers;
• crash recovery and PostgreSQL stress examples;
• EUnit, PostgreSQL integration tests, xref, Dialyzer, and secret scan in CI.

Current limits:

• no DAG, fan-out/fan-in, child workflows, or parallel command batches yet;
• no first-class human task assignment UI;
• no HTTP API or browser UI;
• no SQL-native JSON payload inspection;
• no built-in external side-effect deduplication;
• no exactly-once execution of user callbacks.

The at-least-once boundary is explicit. BeamTrail provides stable idempotency
keys, but workflow code must use them when calling external systems. If a VM
dies after a callback performs an external side effect but before BeamTrail
records the outcome, the same attempt can be re-entered with the same
idempotency key.

What I would especially like reviewed:

1. Is the OTP/PostgreSQL split reasonable?
   OTP supervision handles live process failure, but durable business progress
   is stored in PostgreSQL. Does this boundary feel natural to Erlang/OTP users,
   or does it fight the platform?

2. Is the lease/fencing model strong enough?
   Non-creation appends require a valid fencing token. PostgreSQL serializes one
   run's append path with a per-run row lock, and stale expected_seq writes are
   rejected. Are there takeover or zombie-writer races I am missing?

3. Is the active runner/scanner recovery shape idiomatic enough?
   A live run has a supervised gen_statem owner. If it disappears, scanner
   recovery acquires a lease and dispatches a new runner. Does that fit OTP
   expectations, or should a different process topology be preferred?

4. Is scanner-driven timer materialization acceptable?
   Durable timers are stored as events and projected as next_wake_at. When due,
   recovery dispatch materializes timer.fired before the decider continues.
   Live runners can use local timers for retry, but durable timers still rely on
   the scanner path. Is this a reasonable MVP boundary?

5. Is the public API shaped naturally for Erlang?
   Current APIs include start_workflow/3, await_terminal/2, signal_run/3,
   cancel_run/2, park_run/2, resume_run/1, requeue_run/2, and
   beamtrail_query:describe/1. I would like feedback on naming and expected
   return semantics.

6. Is the MVP scope honest enough?
   The project intentionally does not claim to be a full workflow platform yet.
   The near-term target is a credible embedded durable step/process runtime
   before adding richer orchestration.

Useful docs:

• Design rationale:
  https://github.com/sherry255/BeamTrail/blob/main/design-rationale.md
• Architecture and failure model:
  https://github.com/sherry255/BeamTrail/blob/main/ARCHITECTURE.md
• Decider and dataflow:
  https://github.com/sherry255/BeamTrail/blob/main/DECIDER.md
• Durable timers:
  https://github.com/sherry255/BeamTrail/blob/main/TIMER.md
• Human approval deadline pattern:
  https://github.com/sherry255/BeamTrail/blob/main/APPROVAL.md
• Roadmap:
  https://github.com/sherry255/BeamTrail/blob/main/ROADMAP.md

I am most interested in failure-model criticism, OTP design criticism, and
whether the current scope would be useful to real Erlang/Elixir teams.

Thanks.