Skip to content

v2026.09.14

@turbolytics turbolytics tagged this 14 Sep 01:28
Upgrade from v1.2.0. This is the first date-tagged release, and it
changes the pipeline file: the tumbling window `manager` block is gone.
Everything before this tag is the old window format.

Windows (#281, closes #203 and #280):

- A table under `tables.sql` declares a `window`: `time_column`,
  `size_seconds`, `grace_seconds`, `idle_close_seconds`, `late_rows`
  (required, `drop` or `reemit`), an optional `emit_sql` over `closed`,
  and a sink. The engine runs a watermark manager per window. It keeps
  an event-time watermark in `sqlflow_windows` that never moves
  backwards, publishes every bucket the watermark has passed, and
  deletes it. `now()` appears in no SQL the close runs, so an in-memory
  and a stateful pipeline behave the same.

- Late rows have a policy. `drop` deletes a row for a bucket that
  already closed and counts it in `window_late_rows_total`; `reemit`
  publishes the bucket again, for a sink that upserts. `sqlflow
  validate` warns when `reemit` is paired with the Iceberg or Kafka
  sink.

- Each window runs on two DuckDB connections of its own. It reads
  committed rows only, its delete and its watermark commit together, and
  it takes no part in the pipeline's lock or transaction. The poll
  failure #280 reported has no shared connection to happen on.

- `window_watermark_seconds`, `window_closed_total` and
  `window_late_rows_total`.

- Window tables carry no index, and the examples append each batch's
  counts and sum them in `emit_sql` (#277). DuckDB never frees rows
  deleted from an indexed table, and the Bluesky demo grew from 100 MB
  to 450 MB in a day on the old pattern. `validate` warns when a window
  table has an index and the pipeline has no state path.

- Three invariants join the manager's four in the harness, all
  enforced: the watermark never regresses, a close publishes committed
  rows only, and the late policy holds.

Migrating a `manager` block:

    manager.tumbling_window.collect_closed_windows_sql  -> window.emit_sql
        over `closed`, with the WHERE dropped
    manager.tumbling_window.delete_closed_windows_sql   -> gone
    the predicate's grace                               -> window.grace_seconds
    the predicate's idleness clause                     -> window.idle_close_seconds
    the bucket length the predicate assumed             -> window.size_seconds
    the bucket column                                   -> window.time_column
    manager.tumbling_window.poll_interval_seconds       -> window.poll_interval_seconds
    manager.sink                                        -> window.sink
    nothing                                             -> window.late_rows, required

`sqlflow validate` refuses the old block and names these keys. Every
shipped example carries the declaration.

Process contract (#273, closes #161):

- `pipeline.drain_deadline_seconds` bounds the whole shutdown after
  SIGTERM, 30 seconds by default. The batch in flight, the drain, the
  state syncs and every window's final poll share one deadline. When it
  passes the process exits 15 with `system.lifecycle.drain_incomplete`.
  Nothing unwritten was committed, and the next start replays it.

- `/healthz` reports `starting`, `healthy`, `degraded` or `failed` with
  a `reason`. `failed` answers 503 and the rest 200. The previous `ok`
  and `stuck` bodies are `healthy` and `failed`; their HTTP codes did
  not change. `degraded` means a sink's retry ladder is running or the
  loop recorded an error inside the last flush interval.

- A SIGTERM during a flush lets the flush finish inside the deadline.
  A window's final poll always runs, and its failure is the exit code.

Correctness (#279, #270, #269, #283):

- A batch the sink refused no longer has its offsets committed on the
  way out. Positions advanced when a message reached the handler, and
  the shutdown's state syncs made them durable, so a restart resumed
  past rows nothing had written. The loop resets to the last committed
  positions on any error. Proven by a new enforced invariant,
  `pipeline.shutdown.commits_only_delivered`, and a fifth harness
  trigger for a cancel that lands as a batch fills.

- A window poll the sink refuses stops the process with the sink's
  error, after one attempt. Before, it was retried every tick while the
  container reported healthy.

- A value the sink's driver cannot encode fails once, as
  `user.sink.encode_failed`, exit 10. It was retried the full ladder and
  reported as unreachable.

- Every statement on the pipeline's DuckDB connection holds its lock:
  the handler reset, the progress write and the sqlcommand sink took
  none, and a `--with-http-debug` query could close a result the loop
  had in flight. A lock-checked connection in the tests holds the rule.

sqlflow serve (#278):

- `sqlflow serve` serves named SQL over HTTP from a serve config:
  datasets with typed parameters, grains, bearer tokens that name the
  caller, a row cap, a query timeout and CORS. `sqlflow validate` checks
  a serve config, and `sqlflow config example --serve` prints one.

Also: `sqlflow validate` warns when the sinks' retry ladders add up to
more than the drain deadline; `/stats` carries `errors` and
`last_error`; a security disclosure policy (#263); dependency bumps
(#264, #266); the TigerData config follows the site tutorial (#262).

Upgrade note: a v1.2.0 state file opens unchanged; `sqlflow_windows` is
created beside the existing tables on first start. A pipeline file with
a `manager` block does not start until it declares a `window`. A
supervisor that matched on the `ok` or `stuck` health bodies should
match `healthy` and `failed`. Exit code 15 is new and retryable.
Assets 2
Loading