Skip to content

Releases: parallelworks/hopper

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 03 Oct 01:56
93e73e9

0.5.0 (2026-10-03)

Features

  • core: a client can work jobs without standing for leader (#46) (eef321d)

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 02 Oct 15:09
b56fe99

0.4.0 (2026-10-02)

Features

  • core: a configurable client lease, and a job on its last attempt survives a lost lease (#44) (23abd36)

v0.3.1

Choose a tag to compare

@github-actions github-actions released this 02 Oct 00:59
e4e8e5a

0.3.1 (2026-10-02)

Bug fixes

  • streams: events are ordered by transaction ID as a number, not as text (#38) (863c673)

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 01 Oct 23:31
dc8391b

0.3.0 (2026-10-01)

Features

  • driver: make PostgreSQL schema configurable (#31) (5fd71c6)

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 01 Oct 14:34
6b1ee62

0.2.0 (2026-10-01)

Features

  • core: the client running a job is available from its context (#27) (e3e843a)

v0.1.2

Choose a tag to compare

@github-actions github-actions released this 01 Oct 01:35
251e490

0.1.2 (2026-10-01)

Bug fixes

  • core: a deep finalizer buffer, and timed-out flushes retried in halves (#22) (c27b943)

v0.1.1

Choose a tag to compare

@giraffesyo giraffesyo released this 29 Sep 21:26
b2ab68c

The first soak on the reference hardware found three defects in the live
table's upkeep that only show under sustained load; all three are fixed here
and verified with a ten-minute loaded run at 43,000 jobs/s (p50 9.6 ms, p99
52 ms in every window). Upgrading applies schema v7 (one ALTER TABLE).

Fixed

  • The leader re-analyzes the live table on churn again (rate-limited to
    every 30 s, with a small sample): with statistics refreshed only on a
    tenfold size change, the planner's picture of the state mix went stale
    and the finalize drifted back onto its slow join order within minutes at
    43,000 jobs/s.
  • Vacuums of the live table no longer truncate its heap (schema v7 sets
    vacuum_truncate = false; the leader passes TRUNCATE false): the
    truncation retried an exclusive lock for five seconds per pass and
    stalled every claim behind it whenever it got one, which showed as
    pickup-latency spikes of one to two seconds.
  • The claim's LIMIT is a constant (one statement per bucket, 1 to 512,
    with the count applied through row numbers) instead of a bind parameter,
    whose generic plan scanned the whole table per claim; a claim returns at
    most 512 jobs.
  • hopperbench loaded -report N prints per-window latency and rate lines
    for soaks.

Modules: github.com/parallelworks/hopper, github.com/parallelworks/hopper/hopperotel, github.com/parallelworks/hopper/hopperui, all at v0.1.1.

hopper v0.1.0

Choose a tag to compare

@giraffesyo giraffesyo released this 29 Sep 16:26
fc984e0

The first release: a job queue and message broker on PostgreSQL, with the
hopperotel and hopperui modules released alongside it at the same
version. The §8.2 performance targets in docs/PLAN.md were
run on the reference hardware before tagging; the results are recorded
there.

Jobs

  • Core engine: typed workers, Insert/InsertTx/InsertMany (one statement,
    or COPY for large batches), batched FOR UPDATE SKIP LOCKED claims,
    batched finalize into time-partitioned history, retries with backoff,
    Snooze and Cancel, timeouts, panic recovery, and a three-step graceful
    Run/Stop.
  • Reliability: per-client leases, rescue of jobs from crashed processes,
    fencing of paused processes, leader election, partition-drop retention,
    LISTEN/NOTIFY wake-ups with per-process coalescing and polling fallback,
    unique jobs with skip and replace.
  • Control: cron and interval periodic jobs with time zones, in-flight
    cancellation, retry from the dead-letter queue, TTLs, queue pause and
    resume, runtime queues, middleware, SetOutput/Await, the Jobs
    iterator (paged with JobFilter.After), events and Stats.
  • Flow control: cluster-wide GlobalLimit, RateLimit/RateBurst and
    PartitionLimit per queue (declared in QueueConfig or set at runtime
    with Queues().SetLimits and hopper queues limit), PriorityAging,
    and InsertOpts.PartitionKey.
  • Batches: NewBatch, Add, Insert/InsertTx, BatchGet, with
    OnSuccess, OnFailure and OnComplete callbacks inserted by the
    finalizing statement.
  • Workflows: NewWorkflow, Add with After, InsertWorkflow/
    InsertWorkflowTx, WorkflowGet and hopper workflows get. Steps with
    dependencies wait pending and are promoted by the statement that finalizes
    the last of them; a failed step cancels its dependents unless they opt to
    DependencyIgnore.

Messaging

  • Subscriptions: Subscribe with AMQP topic patterns, typed Message[T],
    Publish/PublishTx fan-out in one statement, dedup keys, ordering keys
    (also on plain jobs through InsertOpts.OrderingKey), request/reply, and
    ReplayDiscarded.
  • Streams: Streams().Append/AppendTx write to a retained, time-partitioned
    log; hopper.Consume registers a consumer that delivers matching events as
    jobs from a position of its own, starting at the earliest or latest event
    and movable with Seek; Read pages the log. Consumers read by snapshot
    deltas, so a late-committing transaction is delivered when it commits and
    never skipped. Config.StreamRetention, hopper streams consumers|seek
    and hopper subscriptions list.
  • The SQL contract functions hopper_insert and hopper_publish, for
    producers in other languages (docs/sql-contract.md).

Drivers and schema

  • hopperpgx: the pgx v5 driver, with COPY, LISTEN and pipelined statements.
  • hoppersql: a driver for database/sql (pgx's stdlib adapter or lib/pq)
    that passes the same conformance suite; it polls instead of listening and
    inserts without COPY. Both drivers share one implementation of the SQL.
  • hoppermigrate: embedded, versioned migrations under a cross-process lock.
    Schema versions 1 through 6: the job tables and history partitions,
    messaging, flow control and batches, workflows, streams, and the live
    table's autovacuum settings.
  • drivertest: the conformance, concurrency and chaos suite every driver
    must pass, including TestUpgradeUnderTraffic, which migrates to the
    latest schema while a client works jobs.

Operations

  • The leader maintains the live table (driver.Executor.JobsMaintain):
    it vacuums hopper_jobs once a hundred thousand dead rows have
    accumulated and re-analyzes it when the planner's row count is an order
    of magnitude off, every leader interval, without waiting for autovacuum's
    lock. Every claim walks the claim index past the
    entries of finished jobs until a vacuum removes them, and autovacuum
    looks only every minute by default: on the reference hardware pickup
    latency at 30,000 jobs/s climbed from 10 ms to seconds within a minute.
  • The claim's candidate subquery is a MATERIALIZED CTE. A plan cached
    while hopper_jobs was empty otherwise re-executed the locking subquery
    once per row of the table after a burst: a single claim ran for 20-30
    seconds holding locks, with every finalizer queued behind it.
  • cmd/hopper: migrations, jobs, queues, clients, workflows, subscriptions,
    stream consumers and stats from the shell.
  • cmd/hopperbench: the benchmark harness (including loaded, pickup
    latency under a paced insert load) and the CI performance gate.
  • hoppertest: test helpers for code that uses hopper.
  • hopperotel (separate module): OpenTelemetry tracing and metrics.
  • hopperui (separate module): an embeddable web UI for queues, jobs,
    workflows (as a DAG), subscriptions, stream consumers and clients, with
    actions behind an Authorize hook, and a standalone hopperui server.

Install

go get github.com/parallelworks/hopper@v0.1.0
go get github.com/parallelworks/hopper/hopperotel@v0.1.0   # OpenTelemetry
go get github.com/parallelworks/hopper/hopperui@v0.1.0     # web UI

Requires Go 1.27 and PostgreSQL 14 or later. Start with docs/getting-started.md.