Upgrade from v1.2.0. This is the first date-tagged release, and it
changes the pipeline file: the tumbling window `manager` block is gone.
Everything before this tag is the old window format.
Windows (#281, closes #203 and #280):
- A table under `tables.sql` declares a `window`: `time_column`,
`size_seconds`, `grace_seconds`, `idle_close_seconds`, `late_rows`
(required, `drop` or `reemit`), an optional `emit_sql` over `closed`,
and a sink. The engine runs a watermark manager per window. It keeps
an event-time watermark in `sqlflow_windows` that never moves
backwards, publishes every bucket the watermark has passed, and
deletes it. `now()` appears in no SQL the close runs, so an in-memory
and a stateful pipeline behave the same.
- Late rows have a policy. `drop` deletes a row for a bucket that
already closed and counts it in `window_late_rows_total`; `reemit`
publishes the bucket again, for a sink that upserts. `sqlflow
validate` warns when `reemit` is paired with the Iceberg or Kafka
sink.
- Each window runs on two DuckDB connections of its own. It reads
committed rows only, its delete and its watermark commit together, and
it takes no part in the pipeline's lock or transaction. The poll
failure #280 reported has no shared connection to happen on.
- `window_watermark_seconds`, `window_closed_total` and
`window_late_rows_total`.
- Window tables carry no index, and the examples append each batch's
counts and sum them in `emit_sql` (#277). DuckDB never frees rows
deleted from an indexed table, and the Bluesky demo grew from 100 MB
to 450 MB in a day on the old pattern. `validate` warns when a window
table has an index and the pipeline has no state path.
- Three invariants join the manager's four in the harness, all
enforced: the watermark never regresses, a close publishes committed
rows only, and the late policy holds.
Migrating a `manager` block:
manager.tumbling_window.collect_closed_windows_sql -> window.emit_sql
over `closed`, with the WHERE dropped
manager.tumbling_window.delete_closed_windows_sql -> gone
the predicate's grace -> window.grace_seconds
the predicate's idleness clause -> window.idle_close_seconds
the bucket length the predicate assumed -> window.size_seconds
the bucket column -> window.time_column
manager.tumbling_window.poll_interval_seconds -> window.poll_interval_seconds
manager.sink -> window.sink
nothing -> window.late_rows, required
`sqlflow validate` refuses the old block and names these keys. Every
shipped example carries the declaration.
Process contract (#273, closes #161):
- `pipeline.drain_deadline_seconds` bounds the whole shutdown after
SIGTERM, 30 seconds by default. The batch in flight, the drain, the
state syncs and every window's final poll share one deadline. When it
passes the process exits 15 with `system.lifecycle.drain_incomplete`.
Nothing unwritten was committed, and the next start replays it.
- `/healthz` reports `starting`, `healthy`, `degraded` or `failed` with
a `reason`. `failed` answers 503 and the rest 200. The previous `ok`
and `stuck` bodies are `healthy` and `failed`; their HTTP codes did
not change. `degraded` means a sink's retry ladder is running or the
loop recorded an error inside the last flush interval.
- A SIGTERM during a flush lets the flush finish inside the deadline.
A window's final poll always runs, and its failure is the exit code.
Correctness (#279, #270, #269, #283):
- A batch the sink refused no longer has its offsets committed on the
way out. Positions advanced when a message reached the handler, and
the shutdown's state syncs made them durable, so a restart resumed
past rows nothing had written. The loop resets to the last committed
positions on any error. Proven by a new enforced invariant,
`pipeline.shutdown.commits_only_delivered`, and a fifth harness
trigger for a cancel that lands as a batch fills.
- A window poll the sink refuses stops the process with the sink's
error, after one attempt. Before, it was retried every tick while the
container reported healthy.
- A value the sink's driver cannot encode fails once, as
`user.sink.encode_failed`, exit 10. It was retried the full ladder and
reported as unreachable.
- Every statement on the pipeline's DuckDB connection holds its lock:
the handler reset, the progress write and the sqlcommand sink took
none, and a `--with-http-debug` query could close a result the loop
had in flight. A lock-checked connection in the tests holds the rule.
sqlflow serve (#278):
- `sqlflow serve` serves named SQL over HTTP from a serve config:
datasets with typed parameters, grains, bearer tokens that name the
caller, a row cap, a query timeout and CORS. `sqlflow validate` checks
a serve config, and `sqlflow config example --serve` prints one.
Also: `sqlflow validate` warns when the sinks' retry ladders add up to
more than the drain deadline; `/stats` carries `errors` and
`last_error`; a security disclosure policy (#263); dependency bumps
(#264, #266); the TigerData config follows the site tutorial (#262).
Upgrade note: a v1.2.0 state file opens unchanged; `sqlflow_windows` is
created beside the existing tables on first start. A pipeline file with
a `manager` block does not start until it declares a `window`. A
supervisor that matched on the `ok` or `stuck` health bodies should
match `healthy` and `failed`. Exit code 15 is new and retryable.