v0.18.52

v0.18.52

Parquet writer, scan, and ingest correctness hardening — eight data-corruption gaps, each found by an adversarial probe of a path that reported success while returning or storing wrong data, and each now made right or refused.

• Parquet writer: a float column with a leading or all-NaN run no longer writes NaN min/max statistics (which could mis-prune a row group). (#928)
• Parquet writer: a page whose bytes exceed the Parquet 2GB page-size limit is refused loudly instead of overflowing its size field and finalizing a corrupt file. (#929)
• Parquet: a nested MAP KEY of any type (DECIMAL, DATE, TIMESTAMP, IPv4/IPv6, MAC, UUID, CIDR, BYTES) round-trips exactly, instead of a decimal key being rescaled or a date/network/bytes key being lost. (#883)
• Scan: a decoded-cache hit applies the same DECIMAL precision/overflow check as a cold read, so an out-of-range value raises instead of being served from a warm cache. (#914)
• Scan: dictionary pruning reconciles a DECIMAL predicate and the file's dictionary to a common scale before comparing, so a file declaring a different scale can no longer prune a matching row group. (#916)
• Ingest: a partially-rejected batch no longer duplicates its accepted prefix on retry — Ingest is all-or-nothing. (#917)
• Ingest: buffered rows are deep-copied, so reusing or mutating the maps and byte slices you passed to Ingest no longer corrupts not-yet-flushed rows. (#918)
• Ingest: an ingester whose table was dropped and recreated no longer flushes its buffered rows into the new table; the stale flush is refused. (#919)

v0.18.51

v0.18.51

Parquet hardening — a wave of reader/writer corruption gaps, each found by an
adversarial mutation of a file Wadjet itself wrote and read back through the
public reader, and each now refused instead of returning wrong data.

• Parquet writer: a short write from the underlying io.Writer (fewer bytes
  taken than handed over, with no error) is detected and reported as a failure
  instead of silently finalizing a truncated file; the failure latches, so later
  writes and Close stay failed. (#926)
• Parquet reader: a v1 data page whose level section decodes fewer values than
  it declares is refused, rather than reinterpreting the level bytes as column
  values. (#923)
• Parquet reader: a dictionary page encountered in the data-page walk is refused,
  rather than being skipped after its rows were already charged (which dropped
  the tail of a column). (#924)
• Parquet reader: row-group statistics are keyed by full leaf path, so a nested
  field no longer overwrites a same-named top-level column's bounds — which could
  wrongly prune a row group. (#925)
• Parquet reader: each row-group column chunk is bound to its schema leaf by full
  path; footer metadata whose columns are swapped, duplicated, missing, unknown,
  or contradict the schema is refused, rather than silently returning one
  column's values under another's name. (#927)
• Scan: predicate pruning resolves a column by its full path, so a query on a
  top-level column is never pruned away by a nested field that shares its name.
  (#915)

v0.18.50

v0.18.50

• The benchmark harness no longer runs background compaction on the clusters it
  spawns, and refuses to load a data bucket that contains compaction residue
  (compacted_* objects) from an earlier run — a benchmark harness must never
  mutate the bucket it reads. This prevents the silent row-duplication that made
  one SF10 fixture hold three copies of its data. Set
  WADJET_HARNESS_ALLOW_COMPACTED=1 to load a dataset whose canonical files are
  deliberately compaction-produced.
• A new correctness gate covers a two-key GROUP BY ... SUM under an OUTER join
  whose result feeds a predicate on the sum, on the single-process engine and
  three distributed configurations with the aggregate's spill drain forced,
  including probe rows that match no group — those must come back NULL and must
  not satisfy the predicate.

v0.18.49

v0.18.49

• A value kept by a sorting, joining or collecting operator downstream of a
  column prune is no longer overwritten when the batch pool recycles the
  storage behind it. The pool now refuses to recycle any batch whose storage a
  consumer still holds, including storage reached through a nested field or a
  view. (#897)
• FromRowsChecked refuses an unrepresentable DECIMAL value inside a ROW,
  ARRAY or MAP, the same way it already refused one at the top level. Text
  that names no number, an integer supplied as a raw unscaled carrier, and a
  value too wide for the column's scale are errors carrying PostgreSQL's
  SQLSTATE and naming the field or element that failed, instead of being
  stored as 0.00, as the value divided by 10^scale, or as a saturated
  maximum. (#898)
• ARRAY<VECTOR(N)>, ROW{VECTOR(N)} and MAP<..., VECTOR(N)> values round
  trip, and a NULL vector followed by a present one reads back correctly. Both
  used to fail with an out-of-range panic. (#899)
• A VECTOR(N) write must supply exactly N components. A shorter or longer
  value is refused with SQLSTATE 22000 and expected N dimensions, not M —
  what PostgreSQL's vector extension answers — instead of being silently
  padded with whatever the storage last held. (#900)
• The VECTOR width check above costs nothing outside VECTOR writes: it runs
  only at the row-to-batch write boundary (SetValue's VECTOR arm and the
  embedding provider's output), not on any scalar kernel or on any TPC-H or
  ClickBench query.
• A VECTOR literal of the declared width now round-trips through INSERT, UPDATE, MERGE and COPY (previously every VECTOR literal reached the writer as text and corrupted the page); a literal of the wrong width is refused with SQLSTATE 22000 on every door.

v0.18.48

v0.18.48

• A parquet page that carries a checksum is now verified before any value is
  read from it. A page whose stored bytes do not hash to the checksum its own
  header declares fails the query, naming the file, the column and the page,
  instead of returning corrupted values. Data pages (both format versions) and
  dictionary pages are covered, on every read path.
• A column chunk that ends before the rows its row group declares now fails
  the read, naming the column and both counts. Previously the rows the file
  never supplied were returned as NULLs — including in columns whose schema
  says NULL is impossible.
• A row group that carries no data at all for a column of its own file's
  schema is refused rather than read as a column of NULLs. Reading a column
  the table has and a file predates is unchanged.
• Compaction refuses a file the reader can prove wrong instead of merging it:
  the manifest is left unchanged and every input file stays in place, so
  corruption is never made durable by a rewrite.
• A page of a type the reader cannot decode, sitting inside a column chunk, is
  refused rather than skipped. Skipping it dropped a whole page of values and
  shifted every later page's values into their place, without an error.
• A data page v2 whose declared null count cannot be placed by its definition
  levels, or disagrees with them, is refused. Previously the mismatch appeared
  as holes in a column whose schema says NULL is impossible.
• A data page v2 whose level byte lengths do not fit the page is refused
  rather than crashing the reader.
• The page-checksum check only costs anything on a file that carries
  checksums: none of wadjet's own writer output does, and neither does the
  122 MB ClickBench hits_0.parquet.
• docs/disaster-recovery.md describes the refusals and how to recover the
  file they name.

v0.18.47

v0.18.47

• Security — a policy set now binds to the catalog at every place it is attached or installed, including the embedded API's wadjet.Open (the entry the wadjet mcp command uses), the provider's own update setters, and a configuration hot reload; previously the bind reached only wadjet serve, so a column mask, column deny or row filter whose policy spelled the table's name in a different case than the catalog silently did not apply, on every door, read and write, and a masked column was also readable through a DML predicate. A policy naming a relation or a policed column the catalog does not hold is now refused there — wadjet.Open and wadjet mcp refuse to start, and a caller that discards the error still gets every statement refused (SQLSTATE 42501). A hot reload that cannot bind, or a setter applied to an already-bound provider, keeps the previously running set rather than enforcing an unbound one, and a retired policy set can no longer resurface through a race in the bind's swap. A policy may name a relation either the way the catalog holds it or folded to lower case; any other spelling is a name the catalog does not hold (#882).
• A WHERE predicate on a column whose catalog name carries upper-case letters (WHERE UserAgent = 'agent-A') stood down row-group statistics pruning and dictionary-probe pruning entirely — every row group was read regardless of the predicate. Both now engage on such a table (#731).
• UPDATE on a column whose catalog name has upper-case letters (UPDATE hits SET UserAgent = 'NEW') reported UPDATE 1 and left the row's old value in place; it writes the new value now (#731).
• A MERGE ... ON condition naming a mixed-case column matched no target row and silently inserted a duplicate instead of updating the row it should have matched; it updates the matching row now (#731).
• A MERGE's WHEN MATCHED THEN UPDATE SET clause could report success while leaving the row unchanged, and its WHEN NOT MATCHED THEN INSERT clause could store NULL in every column whose catalog name carries upper-case letters; both actions now write under the schema's own column names (#731).
• A delimited (quoted) column reference in an UPDATE, DELETE, or MERGE that differs only in case from an existing column — UPDATE t SET "USERAGENT" = ..., WHERE "WATCHID" = 1 — previously either silently applied to the wrong column or silently matched zero rows; it is now correctly refused (#731).
• A row-filter or column mask/deny policy scoped to a mixed-case table name applied to SELECT but not to UPDATE, DELETE, or INSERT against the same table, so a denied column was writable and a masked one readable through a write. The write door now checks the same table name the read door does (#731).
• COPY with an explicit column list against a table whose catalog names carry upper-case letters either refused a well-formed file with a spurious type error or silently filled every listed column with NULL; the column list now resolves against the catalog schema (#731).
• COPY's column list is now parsed the way every other identifier is — an unquoted name folds, a delimited one keeps its exact case — so an unquoted upper-case list over a lower-case schema no longer wrongly refuses, a delimited name carrying upper-case letters is no longer wrongly accepted, and a genuinely missing column is reported as such instead of a confusing type-conversion error (#731).
• A nested ROW, ARRAY, or MAP column referenced by its catalog's mixed-case spelling could come back on the wire with its ROW fields in the wrong order, or with an ARRAY and a MAP no longer distinguishable from each other; the value now renders under its declared type either way (#731).
• Two catalog column names that fold to the same reference across a join could make one unchanged query return different wire bytes from run to run; an ambiguous folded reference now resolves consistently to no declared type instead of racing (#731).
• Querying two columns with the same output name through the MCP query tool (wadjet mcp) returned the second column's value for both; each now reports its own value, matching the gRPC and HTTP doors (#513).
• Disabling the WADJET_SCAN_COL_SANITIZE scan-projection switch previously changed query results on a mixed-case schema — silently dropping a GROUP BY, join, or ORDER BY key column; the correctness-critical part of that optimization no longer depends on the switch, which now also participates in the optimization-invariance gate (#731).
• MERGE's SET and INSERT column targets, and a plain INSERT's column list, now read a quoted column name the way every other statement does: the quotes are no longer treated as part of the name, a delimited name matches byte-exact, and an unquoted one folds. A quoted MERGE ... SET "UserAgent" = ... previously refused as an unknown column even when correctly cased, while the equivalent UPDATE succeeded; a quoted wrong-case column in a bare INSERT's column list was previously accepted and written, and is now correctly refused. Two spellings of one column in an INSERT list are still refused as duplicates (#731).
• A regression battery covering joins, outer joins, DISTINCT, grouped aggregates, windows, CTEs, set operations, and a correlated subquery over a mixed-case schema now runs 32 shapes on the single-process path, both distributed execution arms, and a forced group-key spill, and no longer accepts a same-error or same-empty result on both spellings as agreement (#731).
• cmd/unifi-ingest — the UniFi ingest example tool moved out of the core command set; it will return in a separate examples repository.
• wadjet mcp opens the shared catalog the way every other command does; it previously opened a private, empty catalog on every invocation, so it saw no tables and, with a policy configured, could not enforce it (the ABAC fix above now refuses to start rather than run unbound).

v0.18.46

v0.18.46

• Every numeric Go type the ingest API accepts is now stored as its own value.
  int8, int16, uint8, uint16 and uint32 into an INT32 column,
  uint64 into INT64, and integers into FLOAT32/FLOAT64 were previously
  written as 0.
• A VECTOR(N) value must have exactly N components. A wrong width used to be
  written without error and shifted the components of every later row.
• A missing value in a NOT NULL column is 23502 not_null_violation at the
  write. It previously corrupted the column's alignment - a required BOOL
  written as [NULL, true] read back as [true, false].
• A typed slice or map ([]int64, map[string]int64, ...) written into an
  ARRAY, ROW or MAP column is now stored; it previously became NULL
  with no error. A value that is not a container at all is 42804.
• An I/O failure while writing a parquet file is now permanent: every later
  write and Close return it, and the file is never finalized. A failure
  part-way through a row group previously produced a readable file with a
  column missing.
• IPv6 and UUID values must be sixteen bytes (a zero-length value, the
  legacy absence form, is still accepted).
• A table or column name that cannot be part of an object key - containing
  /, \ or a NUL byte, being . or .., or beginning with . - is
  refused at CREATE TABLE with 42602. On storage.type: file such a name
  previously wrote data outside the configured data directory.
• A value that a column cannot hold now fails the WRITE instead of being
  stored as a different number. An integer with no INT32 - through the
  programmatic ingest API, the Go writer, or a PORT/PROTOCOL/DATE cast -
  raises 22003 integer out of range; previously it wrapped (3000000000
  was stored as -1294967296, and 3000000000::DATE answered
  -3543531-12-19).
• The programmatic ingest API now refuses at the door exactly what the
  parquet writer refuses. A malformed IPv4/IPv6/MAC/UUID/DURATION
  literal, a bad TIMESTAMP, or an out-of-range DATE is rejected on the
  call that carried it; previously the door checked only the Go type, so the
  bad literal was admitted and the write failed later at flush, taking an
  entire buffer of otherwise-good rows down with it and reporting against a
  partition rather than the call that caused it.
• Object keys and bucket names are validated on every operation, in every
  object store (MemStore, FileStore, and the S3-compatible store) - not
  only on write. List, BucketExists and MakeBucket also hold the bucket
  name to a single path component.
• A relation or column name longer than 63 bytes is refused at
  CREATE TABLE with 42622 - PostgreSQL's own NAMEDATALEN - 1. PostgreSQL
  truncates an over-length name and emits a notice; wadjet refuses it instead
  (recorded in ADR-0012).
• CI runs with a read-only GITHUB_TOKEN.

v0.18.45

v0.18.45

• Background compaction now publishes a merged file in a single conditional
  catalog transaction: the files it replaces, the replacement, and the delete
  markers it applied all commit together, or none of them do. A compaction whose
  publication fails leaves the table exactly as it was and fully queryable,
  instead of leaving it with no visible rows.
• A DELETE that commits while a compaction or a delete-marker rewrite is in
  progress is no longer undone by it. The compaction is refused, discards its
  output, and redoes itself against the newer state.
• Two compactors working on the same files — a background sweep and a
  wadjet compact, or two workers — can no longer both publish. The losing one
  is refused and cleans up after itself, so a table can no longer end up holding
  two copies of every row.
• A delete-marker GC rewrite now applies all of a file's delete markers or none.
  Markers that previously survived a rewrite pointed at a file that no longer
  existed, and the deleted rows they named came back permanently on the next GC
  sweep.
• Compaction no longer deletes an object that another table in the same catalog
  still references. Object cleanup checks every live table's manifest first, and
  keeps any object it cannot prove is unreferenced.
• A table load that registers a path currently being retired by the deferred
  cleanup queue (AddFiles, used by harness loaders and the Iceberg catalog
  integration) now fails with a retryable error instead of silently losing the
  object's bytes.
• wadjet compact reports a run that lost a publication race to a concurrent compactor (and whether it then replanned) instead of printing the summary of an uneventful run; the CLI prints every line the run's summary carries.

v0.18.44

v0.18.44

• An IPv6 value prints the way PostgreSQL's inet prints it: a v4-mapped
  address is ::ffff:10.0.0.1 and not 10.0.0.1, and a v4-compatible one is
  ::1.2.3.4. LIKE matches that text.
• IPv6 values render about twice as fast (Vector.GetValue over an IPv6
  column: 280.5 → 141.6 µs per 2048 rows, allocations 9043 → 4096).
• PostgreSQL's boolean input grammar (t/f, yes/no, y/n, on/off, 1/0, word
  prefixes, case-insensitive, trimmed) applies to a COMPUTED boolean, not only
  to a boolean column: WHERE (NOT flag) = 'yes' answers, and a string that
  names no boolean raises 22P02 instead of answering no rows. Comparing a
  boolean against '' raises 22P02 too, instead of answering false.
• element_at reads the container it is given at any depth: a nested
  element_at, a COALESCE, a CASE or a GREATEST over MAP or ARRAY columns
  routes and compares by the container's declared element type. A MAP with a
  DECIMAL key is looked up at the key's own scale.
• An abbreviated network literal beside a column follows PostgreSQL's inet
  grammar, which is the one the server itself uses there: '10/8',
  '192.168/16', '10.1/8' and '10/008' are values, and a maskless
  abbreviation ('239', '192.168') is 22P02 exactly as it is on the
  server. A mask may not name a byte the literal did not write ('10/15' is a
  value, '10/16' is an error). Every network literal is classified once,
  when the query is planned, so the same query no longer answers or errors
  depending on the data or the execution path — including a full four-octet
  address spelled with a leading zero or a trailing dot ('010.1.2.3'), and
  an IPv6 prefix mask, which is now checked digit by digit against
  PostgreSQL's own 0-128 rule ('::1/064' is 22P02; '10.0.0.1/031' is a
  value).
• An IPv4 or MAC column now compares as the address it holds, not as its
  raw internal encoding, in IN, NOT IN, BETWEEN, and inside a CASE —
  those sites previously matched a quoted address literal against zero, on
  every execution path.
• COALESCE refuses a quoted literal its own folded type cannot read, the way
  GREATEST, LEAST and NULLIF already did — COALESCE(<int col>, 'text')
  and COALESCE(<network col>, 'zzz') are 22P02, before any row.
• Arithmetic over an aggregate keeps the aggregate's exact type through a
  derived table, a CTE, or a set operation (UNION, INTERSECT, EXCEPT):
  SUM(v * 3000000) + 1 over (SELECT c_i64 AS v FROM ...) answers
  36280278840510000001 under numeric, where it was a float8
  3.6280278840509997e+19.
• A literal compared against a BYTES column is read by PostgreSQL's byteain,
  so WHERE b = '\x6869' finds the row holding those two bytes. A malformed
  bytea literal is refused rather than compared as its own spelling, and
  '\x68 69' — whitespace inside the hex digits — is accepted, as PostgreSQL
  accepts it.
• length over BYTES counts bytes, for a stored column as well as a computed
  value, and char_length agrees with it. substring over BYTES returns
  BYTES indexed by bytes.
• text || bytea is text; bytea || bytea is bytea.
• Arithmetic over a computed aggregate through a set operation (UNION, INTERSECT, EXCEPT) whose arms declare the same type now declares and answers that exact type on every path; a set operation whose arms declare different types is a recorded open boundary and still answers float8.
• The statistics-answered MIN/MAX rewrite engages again on tables whose column names carry upper-case letters; since v0.18.30 an unquoted MIN(EventDate) had been scanning the whole column instead of reading two numbers per row group (#881).
• The ClickBench correctness and optimization-invariance gates compare duplicate-named output columns positionally (three queries were being checked over one cell), and the dialect spells Q28/Q29 with OCTET_LENGTH, the byte count the reference workload's length() means; TestHitsCorrectness is 43/43 against the DuckDB baseline again.

v0.18.43

v0.18.43

• An output column with no AS alias is now named the way PostgreSQL names it:
  ?column? for an expression, a literal or a predicate; the function's own
  name for a call or an aggregate (count, sum, abs, coalesce, case);
  the ARGUMENT's name for a cast (CAST(g AS bigint) is g), and the type's
  name only when the argument has none. SELECT * over a derived table whose
  item had no name publishes ?column? too. Values are unchanged; this is the
  name a client reads out of RowDescription.
• Several output columns may now carry the same name — SELECT g + 1, g + 2 is
  two columns called ?column?, as in PostgreSQL. Read such a result
  positionally (QueryResult.Cells, or any wire client).
• A query that refers to an unnamed derived column still spells it with the
  inner block's own text (SELECT "g + 1" FROM (SELECT g + 1 FROM t) s);
  "?column?" is refused, naming the column that exists.
• An aggregate output aliased like a GROUP BY key — SELECT COUNT(*) AS g, g AS x FROM t GROUP BY g — answers correctly when read through a CTE or
  through two or more derived tables. It previously returned the key's values
  under the aggregate's name on the distributed engine.
• A derived table in a subquery's FROM that references the enclosing query is
  now refused as an unsupported shape (SQLSTATE 0A000) naming two workarounds,
  instead of being reported as a missing FROM-clause entry. The SQL is legal;
  this engine has no lowering for it yet.
• EXTRACT(field FROM x) reports the column name extract, TRIM(x) reports
  btrim, POSITION(x IN y) reports position and arr[1] reports arr,
  matching PostgreSQL.
• A SET OPERATION is the one exception to the naming rule: its output columns
  keep the leftmost arm's own spelling (g + 1), not PostgreSQL's ?column?.
  Alias the arm's items to choose the name.
• The HTTP door sends a positional values array beside rows, and the gRPC
  Row message a values list — on the unary and the streaming RPC alike —
  whenever two output columns share a name, because a JSON object and a
  protobuf map can each carry only one of them. columns is always the full
  ordered list, and both fields are absent for a result whose names are unique,
  so existing clients are unaffected.
• The HTTP door returns SELECT * columns in TABLE order (it was Go
  map-iteration order), and a zero-row result now carries the declared column
  list instead of an empty one.