Skip to content

Dwarf v0.16.0 Under the Foreman, Recently Closed Flows, and One Handler per Queue

Latest

Choose a tag to compare

@bw19 bw19 released this 21 Nov 01:46

v1.51.0 moves the foreman to dwarf v0.16.0 and a project's SQL-backed microservices to sequel v1.20.2. The engine's
background work runs as separate jobs on each shard, a wide forEach no longer stalls at its fan-in, and new flows
go around a shard the replica cannot read. The foreman lists the flows that closed but are not archived yet, and
reads one flow's summary by its key. On the short-circuit transport, a request that overlapping subscriptions in one
queue match now reaches one of them, as it does over NATS. The foreman's databases need a schema change, or new
databases.

Highlights

  • dwarf v0.16.0. Per-shard background jobs, wide fan-outs kept in check, and placement around an unreadable
    shard. A database the v1.50.0 foreman created needs dwarf's v0.15.0 and v0.16.0 schema changes by hand.
  • ListRecentlyClosed and Summary on the foreman, and Agent Studio's Closed tab lists the flows that just
    closed.
  • One handler per queue. Subscriptions whose routes overlap in a queue no longer each run a request that the
    short-circuit transport delivers.
  • sequel v1.20.2. db.Conn returns a *sequel.Conn, a migration file of several statements runs in one
    transaction, and MySQL statements are no longer cut off after 8 seconds.

dwarf v0.16.0

The foreman moves to dwarf v0.16.0, by way of v0.15.0, and sequel v1.20.2. The engine's background work (archiving,
deletion, retention, expiration and crash recovery) runs as separate jobs on each shard, so one slow or blocked job
holds back no other, and a tree of any size is archived and deleted. A wide forEach no longer stalls at its fan-in
or takes the replica's workers with it, and new flows go around a shard the replica cannot read. dwarf's release
notes for v0.15.0 and
v0.16.0 have the full list.

The foreman's databases need a schema change. dwarf edits its migrations in place before v1.0, so a database
v1.50.0's foreman created does not pick the changes up, and nothing detects it: the foreman starts, then fails on
its first write of a flow or step. Start each environment's foreman on new, empty databases, or apply the
statements in the Breaking sections of dwarf's v0.15.0 and then v0.16.0 release notes with every replica stopped,
every archive day included. Tests need nothing.

  • ListRecentlyClosed lists the closed flows not archived yet. A flow that closes leaves ListOpen at once but
    reaches ListClosed only when it is archived, about a minute later, so for that minute it was in neither list. To
    list every closed flow, call ListRecentlyClosed and then ListClosed, and drop duplicates by FlowKey.
  • Summary reads one flow's summary by its key, open or closed, archived or not. Every FlowSummary now names
    its RootFlowKey, which Resume, Cancel, Terminate, Delete and Continue take, and its ParentStepKey,
    the step that called a subgraph child.
  • Agent Studio's Closed tab lists the flows that closed in the last minute, beside the archived ones. On a
    subgraph child's page it links to the root flow, and no longer offers the actions that a child's key cannot take.
    Fork is offered on a flow's page and a step's page only once the tree's root is closed, which Fork requires: a
    completed child or step of a root still running or interrupted showed Fork, and the fork failed with a 409.
  • Durations in the foreman's JSON are integer milliseconds. FlowOptions.TimeBudget is timeBudgetMillis
    (was timeBudget, nanoseconds), and ShardSummary.Latency is latencyMillis. Go callers pick the names up with
    the bump; a body written by hand that still sends timeBudget is ignored, and its flow runs on the default budget.
  • Text PostgreSQL cannot store is refused on every database. A NUL in a string, or invalid UTF-8 in JSON passed
    through as given, fails the step that writes it, and is a 400 at Create, Run, Resume, Continue and Fork.
    It used to pass on SQLite. Carry binary data base64-encoded.
  • An expired flow is deleted, not archived. A flow FlowExpiration ends is terminated as before and deleted
    within minutes. The check runs hourly, or every half of the expiration when that is shorter.
  • A subgraph's error in the carrier is a JSON object. Code using the generated clients needs no change, but a
    foreman and a task-bearing microservice on different releases misread a subgraph's error: deploy them together.
  • Metrics. dwarf_operation_seconds is renamed dwarf_shard_operation_seconds. dwarf_archive_days and
    dwarf_reap_pass_seconds are removed: the background work is timed per job by dwarf_janitor_batch_seconds,
    whether each job keeps moving is dwarf_janitor_progress_age_seconds, and the retention kept is
    dwarf_archive_oldest_day_age_seconds. dwarf_archive_lag_seconds and dwarf_archive_failing_seconds gain a
    job label, so aggregate them with max. New: dwarf_ingress_operations, one count per API call by status, and
    dwarf_placement_skipped, the flows placed around a shard the replica cannot read. The Grafana workflow dashboard
    reads the new metrics and adds panels for the API calls, placement, and each background job.

sequel v1.20.2

A project's own SQL-backed microservices move to sequel v1.20.2 with the bump, from v1.18.0.

  • db.Conn returns a *sequel.Conn (v1.19.0), which holds one connection for a sequence of statements and
    transactions and runs them with sequel's handling: placeholders, virtual functions and tracing. The *sql.Conn it
    returned before bypassed sequel. Code that stores the connection under the database/sql type, or calls BeginTx
    or Raw on it, no longer compiles; db.DB.Conn(ctx) returns the raw one.
  • A migration file of two or more statements runs in one transaction on PostgreSQL, SQL Server and SQLite
    (v1.20.0), so a failure applies nothing and the next start runs the file again. A file not applied yet that holds a
    statement that cannot run in a transaction, such as CREATE INDEX CONCURRENTLY or VACUUM, or that begins or
    commits a transaction itself, now fails: move such a statement to a file of its own. sequel.txt states the rule.
  • MySQL and MariaDB statements are no longer cut off after 8 seconds (v1.20.1). sequel set an 8-second read and
    write timeout on every connection, so a lock wait, a large DELETE or a slow migration failed with
    invalid connection while the server went on running it. The driver's timeouts now apply, none by default.
  • A cancelled SQL Server transaction no longer breaks the next statement on its connection (v1.18.1).
  • Driver security fixes (v1.20.2): the MySQL driver v1.10.2, pgx v5.13.0 and go-mssqldb v1.12.0. On PostgreSQL, a
    statement whose connection fails after it is sent now returns an error rather than being run again on another
    connection.
  • The foreman registers a sequel virtual function, SEEK_FROM. sequel's virtual functions are shared by
    everything in the process, so a microservice bundled with the foreman must not register its own under that name.

Overlapping Subscriptions in One Queue

A request to a subject that more than one subscription in a queue matches now reaches one of them, as it does over
NATS. The short-circuit transport picked one subscriber per queue at each subject that matched, so subscriptions
whose routes overlap, such as /{p...} and /orders/{p...} in a microservice's default queue, each got a request
to /orders/1 when the caller was in the same process. Publish returned the first response and dropped the
others, but every handler ran, so its side effects happened once per overlapping route. The short-circuit now groups
a queue's subscribers across every subject that matches and picks one, each equally likely. Neither transport picks
the most specific route, so a catch-all such as /{p...} takes a share of the requests to the more specific routes
it overlaps in its queue.

Other Changes

  • Email parsing moves to github.com/go-letters/letters, the successor of the deprecated
    github.com/mnako/letters, with the same API. smtpingressapi.Email aliases its Email type, so a project that
    imports the old package to name a type of a parsed email renames that import.
  • Dependencies are updated, among them nats-server v2.15.3, nats.go v1.56.0 and grpc v1.85.0.
  • The short-circuit trie no longer reallocates on long subjects. A lookup where two or more subscriptions
    overlap at every segment of a long subject allocated twice per segment past the sixteenth; it now allocates only
    its result.
  • A subject with a literal * or > segment matches a wildcard subscription once. On the short-circuit
    transport, such a segment matched both as the wildcard and as a literal, so the handler ran twice. It now matches
    only through the wildcard, as in NATS.
  • CI runs govulncheck, failing on a known vulnerability the code can reach, scanned on the latest Go release.
  • Task tools fail Chat with a 400. A tool URL that names a task fails Chat, or InitChat and with it the
    ChatLoop flow, with a 400 rather than a 500, the same as a tool URL that is unknown or that the caller is not
    authorized for.

Migration

From inside a Microbus project, ask Claude Code to upgrade Microbus:

Get the latest version of Microbus.

The upgrade-microbus skill first tells you that the foreman's databases need dwarf's schema changes, and asks
whether each environment takes new, empty databases or the statements from dwarf's v0.15.0 and v0.16.0 release
notes. It offers to delete the local SQLite files. It then tells you to deploy the foreman and the microservices
that carry tasks together, renames timeBudget and latency in hand-written foreman JSON, and renames
dwarf_operation_seconds in dashboards and alerts, pointing out the queries that read a removed metric. It offers
to add ListRecentlyClosed where the project lists closed flows, and checks the project's own SQL-backed
microservices for sequel's changes: db.Conn, migration files with a statement that cannot run in a transaction,
and the SEEK_FROM name. It renames the email package import where a project uses it, and tells you about
the change to overlapping subscriptions and the status Chat returns for a task tool.