Releases: parallelworks/hopper
Release list
v0.5.0
v0.4.0
v0.3.1
v0.3.0
v0.2.0
v0.1.2
v0.1.1
The first soak on the reference hardware found three defects in the live
table's upkeep that only show under sustained load; all three are fixed here
and verified with a ten-minute loaded run at 43,000 jobs/s (p50 9.6 ms, p99
52 ms in every window). Upgrading applies schema v7 (one ALTER TABLE).
Fixed
- The leader re-analyzes the live table on churn again (rate-limited to
every 30 s, with a small sample): with statistics refreshed only on a
tenfold size change, the planner's picture of the state mix went stale
and the finalize drifted back onto its slow join order within minutes at
43,000 jobs/s. - Vacuums of the live table no longer truncate its heap (schema v7 sets
vacuum_truncate = false; the leader passesTRUNCATE false): the
truncation retried an exclusive lock for five seconds per pass and
stalled every claim behind it whenever it got one, which showed as
pickup-latency spikes of one to two seconds. - The claim's
LIMITis a constant (one statement per bucket, 1 to 512,
with the count applied through row numbers) instead of a bind parameter,
whose generic plan scanned the whole table per claim; a claim returns at
most 512 jobs. hopperbench loaded -report Nprints per-window latency and rate lines
for soaks.
Modules: github.com/parallelworks/hopper, github.com/parallelworks/hopper/hopperotel, github.com/parallelworks/hopper/hopperui, all at v0.1.1.
hopper v0.1.0
The first release: a job queue and message broker on PostgreSQL, with the
hopperotel and hopperui modules released alongside it at the same
version. The §8.2 performance targets in docs/PLAN.md were
run on the reference hardware before tagging; the results are recorded
there.
Jobs
- Core engine: typed workers,
Insert/InsertTx/InsertMany(one statement,
orCOPYfor large batches), batchedFOR UPDATE SKIP LOCKEDclaims,
batched finalize into time-partitioned history, retries with backoff,
SnoozeandCancel, timeouts, panic recovery, and a three-step graceful
Run/Stop. - Reliability: per-client leases, rescue of jobs from crashed processes,
fencing of paused processes, leader election, partition-drop retention,
LISTEN/NOTIFY wake-ups with per-process coalescing and polling fallback,
unique jobs with skip and replace. - Control: cron and interval periodic jobs with time zones, in-flight
cancellation, retry from the dead-letter queue, TTLs, queue pause and
resume, runtime queues, middleware,SetOutput/Await, theJobs
iterator (paged withJobFilter.After), events andStats. - Flow control: cluster-wide
GlobalLimit,RateLimit/RateBurstand
PartitionLimitper queue (declared inQueueConfigor set at runtime
withQueues().SetLimitsandhopper queues limit),PriorityAging,
andInsertOpts.PartitionKey. - Batches:
NewBatch,Add,Insert/InsertTx,BatchGet, with
OnSuccess,OnFailureandOnCompletecallbacks inserted by the
finalizing statement. - Workflows:
NewWorkflow,AddwithAfter,InsertWorkflow/
InsertWorkflowTx,WorkflowGetandhopper workflows get. Steps with
dependencies wait pending and are promoted by the statement that finalizes
the last of them; a failed step cancels its dependents unless they opt to
DependencyIgnore.
Messaging
- Subscriptions:
Subscribewith AMQP topic patterns, typedMessage[T],
Publish/PublishTxfan-out in one statement, dedup keys, ordering keys
(also on plain jobs throughInsertOpts.OrderingKey), request/reply, and
ReplayDiscarded. - Streams:
Streams().Append/AppendTxwrite to a retained, time-partitioned
log;hopper.Consumeregisters a consumer that delivers matching events as
jobs from a position of its own, starting at the earliest or latest event
and movable withSeek;Readpages the log. Consumers read by snapshot
deltas, so a late-committing transaction is delivered when it commits and
never skipped.Config.StreamRetention,hopper streams consumers|seek
andhopper subscriptions list. - The SQL contract functions
hopper_insertandhopper_publish, for
producers in other languages (docs/sql-contract.md).
Drivers and schema
hopperpgx: the pgx v5 driver, with COPY, LISTEN and pipelined statements.hoppersql: a driver fordatabase/sql(pgx's stdlib adapter or lib/pq)
that passes the same conformance suite; it polls instead of listening and
inserts without COPY. Both drivers share one implementation of the SQL.hoppermigrate: embedded, versioned migrations under a cross-process lock.
Schema versions 1 through 6: the job tables and history partitions,
messaging, flow control and batches, workflows, streams, and the live
table's autovacuum settings.drivertest: the conformance, concurrency and chaos suite every driver
must pass, includingTestUpgradeUnderTraffic, which migrates to the
latest schema while a client works jobs.
Operations
- The leader maintains the live table (
driver.Executor.JobsMaintain):
it vacuumshopper_jobsonce a hundred thousand dead rows have
accumulated and re-analyzes it when the planner's row count is an order
of magnitude off, every leader interval, without waiting for autovacuum's
lock. Every claim walks the claim index past the
entries of finished jobs until a vacuum removes them, and autovacuum
looks only every minute by default: on the reference hardware pickup
latency at 30,000 jobs/s climbed from 10 ms to seconds within a minute. - The claim's candidate subquery is a
MATERIALIZEDCTE. A plan cached
whilehopper_jobswas empty otherwise re-executed the locking subquery
once per row of the table after a burst: a single claim ran for 20-30
seconds holding locks, with every finalizer queued behind it. cmd/hopper: migrations, jobs, queues, clients, workflows, subscriptions,
stream consumers and stats from the shell.cmd/hopperbench: the benchmark harness (includingloaded, pickup
latency under a paced insert load) and the CI performance gate.hoppertest: test helpers for code that uses hopper.hopperotel(separate module): OpenTelemetry tracing and metrics.hopperui(separate module): an embeddable web UI for queues, jobs,
workflows (as a DAG), subscriptions, stream consumers and clients, with
actions behind anAuthorizehook, and a standalonehopperuiserver.
Install
go get github.com/parallelworks/hopper@v0.1.0
go get github.com/parallelworks/hopper/hopperotel@v0.1.0 # OpenTelemetry
go get github.com/parallelworks/hopper/hopperui@v0.1.0 # web UIRequires Go 1.27 and PostgreSQL 14 or later. Start with docs/getting-started.md.