feat(eql_v2)!: collapse ste_vec ORE terms to single oc field#219
Merged
Conversation
📝 WalkthroughWalkthroughThis PR implements the EQL v2.3 schema consolidation by introducing a typed Changesv2.3 Schema & Operator Consolidation
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60–90 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
This was referenced May 18, 2026
coderdan
force-pushed
the
feat/oc-op-consolidation
branch
from
eb7004e to
f2857d4
Compare
coderdan
changed the title
Mirrors cipherstash-suite#1955's ste_vec consolidation. The pre-2.3
`ocf` (fixed-width, numeric) and `ocv` (variable-width, text) split on
sv elements collapses to a single `oc` field with a leading domain-tag
byte (`0x00` numeric, `0x01` string) on the ciphertext. Walking the
CLLW per-byte protocol over `[tag][ciphertext]` preserves cross-domain
separation and within-domain ordering.
OPE-based ordered comparison is removed entirely from `eql_v2_encrypted`.
It will move to a separate encrypted column type in a future release;
v2.2 didn't emit `opf`/`opv` from production `@cipherstash/protect`, so
this isn't a customer-facing data migration.
SQL side
- `src/ore_cllw_u64_8/` + `src/ore_cllw_var_8/` → `src/ore_cllw/`
- `eql_v2.ore_cllw(jsonb)` extractor: inlinable single-statement SQL.
Returns composite with `bytes IS NULL` rather than raising on a
missing `oc` field — required for inlinability.
- `eql_v2.compare_ore_cllw` stays plpgsql: the CLLW per-byte protocol
(`y + 1 == x` mod 256) can't be expressed as a single SELECT.
- Function families collapse: `ore_cllw_u64_8` / `ore_cllw_var_8` →
`ore_cllw`; same for `has_*` and `compare_*`.
- `eql_v2.compare()` priority list shortens: `ob → oc → hm → literal`.
- OPE removed from `eql_v2_encrypted`: `src/ope_cllw_*/` directories
deleted, `eql_v2.order_by_ope` removed, `src/operators/sort.sql`
drops the entire `'ope'` strategy + `_compare_ope_key` helper +
`ope_keys` plumbing through internal helpers.
- `eql_v2.hash_encrypted` flipped to inlinable single-statement SQL
with data-hash fallback (cherry-picked from dan/inline-hash-encrypted
via #205 follow-up; required for ANALYZE to succeed on sv-shaped
payloads).
- `tasks/pin_search_path.sql` allowlists the new inlinable extractors
(`ore_cllw`, `has_ore_cllw`); the comparator stays pinned.
Schema (`docs/reference/schema/eql-payload-v2.3.schema.json`)
- `EncryptedPayload` + `SteVecElement`: `ocf`/`ocv` removed, single
`oc` added. `op` removed entirely.
- `OrderedExclusion` collapses to a single `not (ob AND oc)`.
- `IndexTerms`: `oc` description documents the domain-tag byte.
Tests
- `tests/test_helpers.sql` + `tests/ste_vec.sql` + the
`tests/sqlx/migrations/{003,004,005}_*.sql` fixtures: bulk rename
`ocf`/`ocv` → `oc` in JSON literals. (Hex bodies preserved — these
are structural tests that don't validate cryptographic correctness;
the wire-format change is exercised by re-ingest in the benches.)
- `tests/sqlx/tests/ope_tests.rs` deleted.
- `payload_schema_tests.rs`: legacy `ocf`/`ocv`/`opf`/`opv` are
rejected by the v2.3 schema; `op` is also rejected (no OPE path);
the `ob` × `oc` mutual exclusion is asserted at both the root
payload and sv-element levels.
- Other Rust tests: function-name references updated (`ore_cllw_u64_8`
→ `ore_cllw`, etc.). Tests pinned to duplicate names from the
var_8/u64_8 collapse given a `_str` suffix on the second occurrence
to disambiguate.
Docs
- `CHANGELOG.md`: entries under `Added`/`Changed`/`Removed` rewritten
for ORE-only consolidation. `Removed` gains a line documenting the
removal of OPE-CLLW from `eql_v2_encrypted`.
- New upgrade note `U-006: ste_vec ORE field consolidation` in
`docs/upgrading/v2.3.md` with the `ocf`/`ocv` → `oc` table,
re-encryption action, and forward-pointer about OPE's future home.
- `docs/reference/database-indexes.md`: ORE-CLLW recipe refreshed;
OPE references removed.
Counterpart: cipherstash-suite#1955. EQL-side issue: #207.
Follow-up: #220 (restore CLLW ORE operator class for functional-index
match on sv-element field_order queries).
coderdan
force-pushed
the
feat/oc-op-consolidation
branch
from
f2857d4 to
c1b2c3e
Compare
This was referenced May 18, 2026
The consolidation in this PR introduces `eql_v2.ore_cllw` and `eql_v2.has_ore_cllw` as inlinable SQL functions (no SET search_path) so the planner can fold extractor calls into the calling query. Supabase's splinter lint flags any function with a mutable search_path; both functions need allowlist entries with the inlining rationale. Two overloads each — (jsonb) and (eql_v2_encrypted) — so four findings total from splinter, covered by two allowlist entries that match by schema + name (the (jsonb)/(eql_v2_encrypted) split is collapsed by `uniq` in the splinter test). Fixes the `Supabase splinter` job in CI for this branch.
coderdan
added a commit
that referenced
this pull request
May 19, 2026
Restores functional-index match for sv-element ordered queries after the consolidation in #219 left the type without an opclass. Closes #220. ## What this adds `src/ore_cllw/operators.sql` — same-type comparison operators (`<`, `<=`, `=`, `>=`, `>`, `<>`) on `eql_v2.ore_cllw`. Each operator is backed by a single-statement `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` wrapper that reduces to `eql_v2.compare_ore_cllw_term(a, b) <op> 0`. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match. `src/ore_cllw/operator_class.sql` — `eql_v2.ore_cllw_ops` btree opclass registered `DEFAULT FOR TYPE eql_v2.ore_cllw`. FUNCTION 1 is `eql_v2.compare_ore_cllw_term` directly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase build variant via the existing `**/*operator_class.sql` glob in `tasks/build.sh` (operator classes require superuser). `tasks/pin_search_path.sql` — allowlists the six operator backing functions (`ore_cllw_eq` / `_neq` / `_lt` / `_lte` / `_gt` / `_gte`). Pinning would break the inlining chain and prevent the planner from structurally matching predicates to functional indexes. ## Design choices - **No `HASHES` / `MERGES` flags** on the operator declarations. HASHES needs a registered hash function on the type (no, and we don't want one — the CLLW protocol is for ordering, not hashing). MERGES needs an equivalent operator family on both sides, which we'd register separately if/when needed. This is the gap that disabled the pre-2025-06-24 opclass; see issue #220's history. - **Equality via `compare_ore_cllw_term = 0`**, not a `bytea_eq` shortcut. Consistent with the rest of the CLLW path; one source of truth for equality semantics; resilient to any future change in the underlying ciphertext encoding. - **The opclass operators are different from the operators on `eql_v2_encrypted`.** Those (per #211) inline to `ore_block_u64_8_256` and raise on non-Block-ORE columns. The new operators here are on the `eql_v2.ore_cllw` composite type itself — what callers reach through the extractor form `WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1)`. No conflict, different scope. ## Tests `tests/sqlx/tests/ore_cllw_opclass_tests.rs` covers: - Operator wiring: `=`, `<>`, `<`, `<=`, `>`, `>=` on hand-crafted byte strings under the CLLW per-byte protocol. - Cross-domain ordering via the leading tag byte (`0x00` numeric, `0x01` string) — numeric < string within the same column. - Opclass registration: `pg_opclass.opcdefault = true` for `eql_v2.ore_cllw_ops`. - Functional-index match: build a functional btree on `eql_v2.ore_cllw(value)`, confirm `EXPLAIN` for `ORDER BY eql_v2.ore_cllw(value) LIMIT n` shows `Index Scan` (or `Index Only Scan`) and no `Sort` node. - Inlinability lint: read `pg_proc` directly, assert each backing function is `LANGUAGE sql`, `IMMUTABLE`, `STRICT`, `PARALLEL SAFE`, and not pinned with `SET search_path`. ## Bench impact From the bench results in cipherstash/benches#14 (post-#219 baseline): json/field_order/functional @ 1M = 20.0 s (no opclass; seq scan + Sort) With this opclass, `EXPLAIN` flips to `Index Scan + Limit` and the same query should land in single-digit ms on the bench rig. End-to-end bench re-run is the next step on the bench-side branch. Docs: CHANGELOG `Added` entry, U-005 action-required note refreshed, database-indexes.md ORE-CLLW recipe entry refreshed.
coderdan
commented
May 19, 2026
…shape Introduces a new `eql_v2.ste_vec_entry` DOMAIN over `jsonb`, constrained to the sv-element shape (an object that carries at minimum a selector field `s`). Per the v2.3 payload schema, sv-element-only fields like `oc` live exclusively inside `sv[i]` and never at the root of an `eql_v2_encrypted` column value — the pre-existing `eql_v2.ore_cllw(eql_v2_encrypted)` / `has_ore_cllw(eql_v2_encrypted)` overloads read `oc` from `eql_v2_encrypted.data` and only "worked" because `->` happened to merge sv-element fields into a fake-root shape. This commit makes the distinction explicit. What's added - `eql_v2.ste_vec_entry` DOMAIN (`src/ste_vec/types.sql`). - Typed sv-element extractors on the domain: `ore_cllw`, `has_ore_cllw`, `hmac_256`, `has_hmac_256`, `selector`. - 3-way comparator `eql_v2.compare(ste_vec_entry, ste_vec_entry)`. - `=`, `<>`, `<`, `<=`, `>`, `>=` operators on `ste_vec_entry × ste_vec_entry`. Inlinable SQL: equality folds to `hmac_256(a) = hmac_256(b)`, ordering folds to `ore_cllw(a) <op> ore_cllw(b)`. What's removed - `eql_v2.ore_cllw(eql_v2_encrypted)` and `has_ore_cllw(eql_v2_encrypted)` — replaced by the typed `(ste_vec_entry)` overloads. - `eql_v2.compare_ore_cllw(eql_v2_encrypted, eql_v2_encrypted)` plus `src/ore_cllw/compare.sql` — dead once the entry-level dispatch path exists. - The CLLW branch from `eql_v2.compare(eql_v2_encrypted, eql_v2_encrypted)` and its `to_ste_vec_value` unwrap — root payloads never carry `oc`, so the branch was dead. The root `compare()` is now strict Block-ORE-only with a directive error pointing callers at the entry- shape recipe. Caller migration: field-level CLLW patterns go from eql_v2.ore_cllw(col -> 'sel') to eql_v2.ore_cllw((col -> 'sel').data::eql_v2.ste_vec_entry) The `->` operator still returns `eql_v2_encrypted` (so the merged-meta-plus-entry shape is unchanged for existing callers) — the explicit `.data::eql_v2.ste_vec_entry` cast is the migration path to the typed extractor. A future PR may flip `->`'s return type to `eql_v2.ste_vec_entry` directly, removing the explicit cast. pin_search_path and the Supabase splinter allowlist updated for the new extractor + operator backing functions; CHANGELOG and the U-006 upgrade note describe the migration.
The 6 `ore_cllw_*_compare_*` tests in `tests/sqlx/tests/index_compare_tests.rs` were calling the removed `eql_v2.compare_ore_cllw(eql_v2_encrypted, eql_v2_encrypted)` function. Migrate to the post-#219 typed API by: - wrapping each `eql_v2.jsonb_path_query(...)` expression with `((expr).data)::eql_v2.ste_vec_entry` to reach the entry shape - calling `eql_v2.compare(eql_v2.ste_vec_entry, eql_v2.ste_vec_entry)` (the new 3-way comparator on the domain type) Assertion semantics preserved: same numeric / text fixtures, same expected -1 / 0 / 1 outcomes.
Two batches of fixes for `operator_compare_tests.rs` after #219's strict separation: 1. `compare_ore_cllw_hello_path` / `compare_ore_cllw_number_path` — wrap each `eql_v2.jsonb_path_query(...)` expression with `((expr).data)::eql_v2.ste_vec_entry` so the planner resolves to `eql_v2.compare(ste_vec_entry, ste_vec_entry)` (the new typed 3-way comparator on the domain) rather than the root-only `eql_v2.compare(eql_v2_encrypted, eql_v2_encrypted)` which now raises on missing `ob`. 2. `compare_raises_on_hmac_only_payloads` / `compare_raises_when_no_index_terms_present` / `compare_raises_when_ore_term_is_json_null` — the strict root `compare()` error now says "requires Block ORE (`ob`)" instead of the previous "requires an ORE term". Update the substring assertion to match the new, more specific wording.
coderdan
added a commit
that referenced
this pull request
May 19, 2026
Restores functional-index match for sv-element ordered queries after the consolidation in #219 left the type without an opclass. Closes #220. `src/ore_cllw/operators.sql` — same-type comparison operators (`<`, `<=`, `=`, `>=`, `>`, `<>`) on `eql_v2.ore_cllw`. Each operator is backed by a single-statement `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` wrapper that reduces to `eql_v2.compare_ore_cllw_term(a, b) <op> 0`. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match. `src/ore_cllw/operator_class.sql` — `eql_v2.ore_cllw_ops` btree opclass registered `DEFAULT FOR TYPE eql_v2.ore_cllw`. FUNCTION 1 is `eql_v2.compare_ore_cllw_term` directly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase build variant via the existing `**/*operator_class.sql` glob in `tasks/build.sh` (operator classes require superuser). `tasks/pin_search_path.sql` — allowlists the six operator backing functions (`ore_cllw_eq` / `_neq` / `_lt` / `_lte` / `_gt` / `_gte`). Pinning would break the inlining chain and prevent the planner from structurally matching predicates to functional indexes. - **No `HASHES` / `MERGES` flags** on the operator declarations. HASHES needs a registered hash function on the type (no, and we don't want one — the CLLW protocol is for ordering, not hashing). MERGES needs an equivalent operator family on both sides, which we'd register separately if/when needed. This is the gap that disabled the pre-2025-06-24 opclass; see issue #220's history. - **Equality via `compare_ore_cllw_term = 0`**, not a `bytea_eq` shortcut. Consistent with the rest of the CLLW path; one source of truth for equality semantics; resilient to any future change in the underlying ciphertext encoding. - **The opclass operators are different from the operators on `eql_v2_encrypted`.** Those (per #211) inline to `ore_block_u64_8_256` and raise on non-Block-ORE columns. The new operators here are on the `eql_v2.ore_cllw` composite type itself — what callers reach through the extractor form `WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1)`. No conflict, different scope. `tests/sqlx/tests/ore_cllw_opclass_tests.rs` covers: - Operator wiring: `=`, `<>`, `<`, `<=`, `>`, `>=` on hand-crafted byte strings under the CLLW per-byte protocol. - Cross-domain ordering via the leading tag byte (`0x00` numeric, `0x01` string) — numeric < string within the same column. - Opclass registration: `pg_opclass.opcdefault = true` for `eql_v2.ore_cllw_ops`. - Functional-index match: build a functional btree on `eql_v2.ore_cllw(value)`, confirm `EXPLAIN` for `ORDER BY eql_v2.ore_cllw(value) LIMIT n` shows `Index Scan` (or `Index Only Scan`) and no `Sort` node. - Inlinability lint: read `pg_proc` directly, assert each backing function is `LANGUAGE sql`, `IMMUTABLE`, `STRICT`, `PARALLEL SAFE`, and not pinned with `SET search_path`. From the bench results in cipherstash/benches#14 (post-#219 baseline): json/field_order/functional @ 1M = 20.0 s (no opclass; seq scan + Sort) With this opclass, `EXPLAIN` flips to `Index Scan + Limit` and the same query should land in single-digit ms on the bench rig. End-to-end bench re-run is the next step on the bench-side branch. Docs: CHANGELOG `Added` entry, U-005 action-required note refreshed, database-indexes.md ORE-CLLW recipe entry refreshed.
coderdan
added a commit
that referenced
this pull request
May 19, 2026
Two fixes for `tests/sqlx/tests/ore_cllw_opclass_tests.rs` after the #219 strict-separation refactor merged into the base branch: - `functional_index_engages_for_order_by`: the index expression and ORDER BY clause referenced `eql_v2.ore_cllw(value)` (where `value` is an `eql_v2_encrypted` column). That overload was removed in #219; the typed replacement is `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` or the (jsonb) form. Switch both to `eql_v2.ore_cllw((value).data)`, which uses the (jsonb) overload and keeps the index/predicate expressions structurally identical so the planner still matches. - Remove the `ANALYZE ore_cllw_test` call. The `value` column is `eql_v2_encrypted` with payloads that carry only `oc` (no root `ob`). ANALYZE samples via the default btree opclass on `eql_v2_encrypted` (FUNCTION 1 = `eql_v2.compare`), and the post-#219 strict-Block-ORE `compare` raises on missing `ob`. The functional-index match still works without stats once `enable_seqscan = off` is set, so dropping ANALYZE is the cleanest path. Also refreshes `tests/sqlx/migrations/001_install_eql.sql` to the current built release SQL (this file is a build artefact that the mise test task copies from `release/`; keeping it in sync with the new opclass + entry types lets `cargo test` run against the up-to-date schema directly).
coderdan
commented
May 19, 2026
`hash_encrypted` no longer falls back to `hashtext((val).data::text)`
when the root payload lacks `hm`. The body reduces to the canonical
inlinable form:
SELECT pg_catalog.hashtext(eql_v2.hmac_256(val)::text)
The fallback existed to keep HashAggregate from collapsing into a
single bucket when a column was misconfigured (no `unique` index, so
no `hm` emitted). But U-002 already documents the contract: equality
on `eql_v2_encrypted` is hm-only at the root, and callers grouping /
distincting an encrypted column must configure `unique` to get `hm`.
Keeping the fallback was defensive code for a scenario we've
explicitly ruled out — and it required the `to_ste_vec_value` unwrap
just to find an `hm` to fall through from.
On a column without `hm` the new body returns NULL; PostgreSQL's hash
opclass support function machinery surfaces this as a clear "function
result is NULL" error at the boundary instead of silently producing
distinct-but-meaningless buckets.
Two tests adapt to the new contract:
- `hash_function_returns_null_when_hmac_absent` (was
`hash_function_falls_back_when_hmac_absent`): asserts NULL on an
ore-only payload.
- `multi_element_ste_vec_returns_null` (was
`multi_element_ste_vec_falls_back`): asserts NULL on a multi-element
ste_vec (no root `hm` — `hm` lives on sv elements there).
Two soft-CT hardening tweaks to `eql_v2.compare_ore_cllw_term_bytes`, per review on #219: 1. The per-byte loop no longer `EXIT`s on the first differing byte. Instead it records `first_diff` (1-based index; 0 = no difference) and walks every byte unconditionally. The previous `EXIT` was an obvious branch whose presence/absence is a stronger timing signal than the rest of the byte comparison itself. 2. The CLLW rotation check switches from `(byte + 1) % 256` to `(byte + 1) & 255`. The two are arithmetically equivalent (the modulus is a power of two), but the bitmask is a single machine instruction with no division-related variance. Documented in the function comment as soft hardening intent, not a CT guarantee — plpgsql, `get_byte`, `SUBSTRING` and the bytea representation all leak timing in ways we can't control from this layer. The change tightens what's controllable here without claiming more than that. `(SUBSTRING ... FROM i FOR 1) != (...)` also drops in favour of direct `get_byte(a, i-1) != get_byte(b, i-1)` byte comparisons, which avoid the per-iteration BYTEA allocation that `SUBSTRING` performs.
… data Reviewer flagged (review on #219) that the existing `tests/ste_vec.sql` fixture carries root-level `c` on its SteVecPayload records, which is invalid per the v2.3 payload schema (root SteVecPayload is `{i, v, sv}` — no `c`). The reviewer suggested generating fixtures dynamically in the test suite instead of shipping a static file. This commit adds the generator — `build_synthetic_ste_vec(id integer)` in `tests/test_helpers.sql`. It returns an `eql_v2_encrypted` carrying a v2.3-compliant SteVecPayload: - **No root `c`** — matches the schema. - Six sv entries per record, all carrying `s` + `c` + `hm` (orderable terms also carry `oc`). The constant `hm` field also prepares the way for a tightened `eql_v2.ste_vec_entry` DOMAIN check (`s` + `c` + `hm` required) in a follow-up. - The `oc` ciphertexts are reused verbatim from the original fixture — they're real cipherstash-suite CLLW output whose byte structure satisfies the `compare_ore_cllw_term` rotation rule. Synthesising new bytes would either need to recreate that protocol or break the ordering tests. - `hm` for `oc`-bearing entries is synthesised deterministically as `md5(s || record_id)` — opaque to callers; what matters is per-record equality semantics for `ste_vec_contains`. - `hm` for hash-only entries (root + nested-object selectors) preserves the original Blake3 hash bytes from the `b3` field that the v2.2 fixture emitted — same bytes, renamed to `hm`. The next commit rewrites `tests/ste_vec.sql` to use this helper.
Inlines 10 fresh INSERTs in `tests/ste_vec.sql`, derived from the
pre-2.3 fixture by applying the v2.3 cleanup mechanically:
- Root-level `c` field removed on every record (the v2.3 schema
forbids root `c` on SteVecPayload; root is `{i, v, sv}`).
- Every sv entry now carries `hm`:
- Entries that previously had `b3` (Blake3) are renamed `b3`→`hm`
with the same bytes (Blake3 was the v2.2 selector-scoped equality
term; `hm` is the v2.3 replacement).
- Entries that previously only had `oc` (ordered terms) get a
synthesised `hm = md5(s || record_id)` — opaque to callers; what
matters is per-record equality semantics for ste_vec_contains.
- The `oc` ciphertexts are kept verbatim — they're real
cipherstash-suite CLLW output whose byte structure satisfies the
`compare_ore_cllw_term` rotation rule. Synthesising new bytes
would either need to recreate that protocol or break ordering.
`build_synthetic_ste_vec(id)` (added in the previous commit) is the
dynamic counterpart for test code that wants to materialise these
rows on the fly. It can't be used as the data source for migration
003 directly — `tests/test_helpers.sql` doesn't run until migration
004 — so the migration uses the inlined form. The two are derived
from the same source data.
Also drops a duplicate `elem ->> 'oc'` line from the hm-synthesis
coalesce in `create_encrypted_json(id)` — likely a leftover from
the `op → oc` rename when two distinct keys collapsed to the same
name. The remaining `coalesce(hm, oc, b3, md5(s||c))` chain is
correct for the legacy `get_numeric_ste_vec_*` fixtures that
`create_encrypted_json` still uses.
Also syncs `tests/sqlx/migrations/003_install_ste_vec_data.sql`
(committed copy of `tests/ste_vec.sql` used by the sqlx test
harness) to the same content.
Reviewer flagged that the original CHECK was too loose: every sv entry that cipherstash-suite emits carries `s` (selector), `c` (ciphertext), and `hm` (HMAC-256). `oc` is optional (only present on orderable terms); the original CHECK only required `s`. The synthesis helper and the rewritten `tests/ste_vec.sql` fixture in the previous two commits now emit `s` + `c` + `hm` on every sv entry, so this tightening doesn't break any tests. Catches misconstructed entries earlier — e.g. accidentally casting a root payload jsonb to `eql_v2.ste_vec_entry` now fails at the cast rather than silently letting the wrong shape flow through to an extractor.
Re-enables two tests that were `#[ignore]`'d because their fixture data lacked `hm` on sv entries: - `selector_less_than_with_ore_cllw` (comparison_tests.rs): rewritten for the post-#219 strict-separation form. The bare-form `<` on `eql_v2_encrypted` reduces to Block-ORE comparison (raises on missing `ob`); the canonical recipe is the entry-typed `<`, with both operands cast via `.data::eql_v2.ste_vec_entry`. With the tighter DOMAIN check from the previous commit, the RHS cast requires `hm` on the extracted sv element — supplied by the legacy fixture update below. - `ste_vec_contains_term` (specialized_tests.rs): just drop the `#[ignore]`. With `hm` now present on every sv entry across both fixture sources, the post-#205 hm-based element comparison in `ste_vec_contains` works as documented. The legacy `get_numeric_ste_vec_{10,20,30,42}()` helpers in `tests/test_helpers.sql` previously emitted sv entries with `b3` on the root entry and bare `oc` on orderable entries (no `hm`). The existing hm-synthesis block inside `create_encrypted_json(id)` papered over this for records inserted through that path, but extractor calls like `get_ste_vec_selector_term()` reached the raw fixture directly and lost the synthesised `hm`. Bake `hm` into the fixture jsonb so every sv entry has it natively: - `b3` is renamed `b3 → hm` (same bytes — Blake3 was the pre-2.3 selector-scoped equality term, `hm` is the v2.3 replacement) - For `oc`-bearing entries, synthesise `hm = md5(s || N)` where N is the fixture's numeric suffix (10/20/30/42). Opaque deterministic; what matters is per-record equality semantics. `tests/sqlx/migrations/004_install_test_helpers.sql` (the committed sqlx-test copy of `tests/test_helpers.sql`) is synced minus the leading `\set ON_ERROR_STOP on` directive which sqlx-migrate can't parse.
coderdan
added a commit
that referenced
this pull request
May 19, 2026
Restores functional-index match for sv-element ordered queries after the consolidation in #219 left the type without an opclass. Closes #220. `src/ore_cllw/operators.sql` — same-type comparison operators (`<`, `<=`, `=`, `>=`, `>`, `<>`) on `eql_v2.ore_cllw`. Each operator is backed by a single-statement `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` wrapper that reduces to `eql_v2.compare_ore_cllw_term(a, b) <op> 0`. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match. `src/ore_cllw/operator_class.sql` — `eql_v2.ore_cllw_ops` btree opclass registered `DEFAULT FOR TYPE eql_v2.ore_cllw`. FUNCTION 1 is `eql_v2.compare_ore_cllw_term` directly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase build variant via the existing `**/*operator_class.sql` glob in `tasks/build.sh` (operator classes require superuser). `tasks/pin_search_path.sql` — allowlists the six operator backing functions (`ore_cllw_eq` / `_neq` / `_lt` / `_lte` / `_gt` / `_gte`). Pinning would break the inlining chain and prevent the planner from structurally matching predicates to functional indexes. - **No `HASHES` / `MERGES` flags** on the operator declarations. HASHES needs a registered hash function on the type (no, and we don't want one — the CLLW protocol is for ordering, not hashing). MERGES needs an equivalent operator family on both sides, which we'd register separately if/when needed. This is the gap that disabled the pre-2025-06-24 opclass; see issue #220's history. - **Equality via `compare_ore_cllw_term = 0`**, not a `bytea_eq` shortcut. Consistent with the rest of the CLLW path; one source of truth for equality semantics; resilient to any future change in the underlying ciphertext encoding. - **The opclass operators are different from the operators on `eql_v2_encrypted`.** Those (per #211) inline to `ore_block_u64_8_256` and raise on non-Block-ORE columns. The new operators here are on the `eql_v2.ore_cllw` composite type itself — what callers reach through the extractor form `WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1)`. No conflict, different scope. `tests/sqlx/tests/ore_cllw_opclass_tests.rs` covers: - Operator wiring: `=`, `<>`, `<`, `<=`, `>`, `>=` on hand-crafted byte strings under the CLLW per-byte protocol. - Cross-domain ordering via the leading tag byte (`0x00` numeric, `0x01` string) — numeric < string within the same column. - Opclass registration: `pg_opclass.opcdefault = true` for `eql_v2.ore_cllw_ops`. - Functional-index match: build a functional btree on `eql_v2.ore_cllw(value)`, confirm `EXPLAIN` for `ORDER BY eql_v2.ore_cllw(value) LIMIT n` shows `Index Scan` (or `Index Only Scan`) and no `Sort` node. - Inlinability lint: read `pg_proc` directly, assert each backing function is `LANGUAGE sql`, `IMMUTABLE`, `STRICT`, `PARALLEL SAFE`, and not pinned with `SET search_path`. From the bench results in cipherstash/benches#14 (post-#219 baseline): json/field_order/functional @ 1M = 20.0 s (no opclass; seq scan + Sort) With this opclass, `EXPLAIN` flips to `Index Scan + Limit` and the same query should land in single-digit ms on the bench rig. End-to-end bench re-run is the next step on the bench-side branch. Docs: CHANGELOG `Added` entry, U-005 action-required note refreshed, database-indexes.md ORE-CLLW recipe entry refreshed.
coderdan
added a commit
that referenced
this pull request
May 19, 2026
Two fixes for `tests/sqlx/tests/ore_cllw_opclass_tests.rs` after the #219 strict-separation refactor merged into the base branch: - `functional_index_engages_for_order_by`: the index expression and ORDER BY clause referenced `eql_v2.ore_cllw(value)` (where `value` is an `eql_v2_encrypted` column). That overload was removed in #219; the typed replacement is `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` or the (jsonb) form. Switch both to `eql_v2.ore_cllw((value).data)`, which uses the (jsonb) overload and keeps the index/predicate expressions structurally identical so the planner still matches. - Remove the `ANALYZE ore_cllw_test` call. The `value` column is `eql_v2_encrypted` with payloads that carry only `oc` (no root `ob`). ANALYZE samples via the default btree opclass on `eql_v2_encrypted` (FUNCTION 1 = `eql_v2.compare`), and the post-#219 strict-Block-ORE `compare` raises on missing `ob`. The functional-index match still works without stats once `enable_seqscan = off` is set, so dropping ANALYZE is the cleanest path. Also refreshes `tests/sqlx/migrations/001_install_eql.sql` to the current built release SQL (this file is a build artefact that the mise test task copies from `release/`; keeping it in sync with the new opclass + entry types lets `cargo test` run against the up-to-date schema directly).
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (2)
tests/sqlx/tests/ore_text_operator_tests.rs (1)
99-100: ⚡ Quick winKeep these tests active by asserting the new hm-only behavior.
Right now the contract change is handled by skipping tests, which drops useful regression coverage. For the ORE-only fixture,
eq/neqshould produce zero matches; assert that explicitly instead of ignoring.Proposed test update
#[sqlx::test] -#[ignore = "Strict equality contract (`#219`): eql_v2.eq is now hm-only (inlines to hmac_256(a) = hmac_256(b)), matching the `=` operator. The ore_text fixture rows carry only `ob` (no `hm`), so eq returns NULL → zero rows match. Equality on ORE-only data has no production analogue — equality is configured via the `unique` index, ordering via `ore`."] async fn ore_text_eq_function(pool: PgPool) -> Result<()> { @@ - QueryAssertion::new(&pool, &sql).count(1).await; + QueryAssertion::new(&pool, &sql).count(0).await; @@ #[sqlx::test] -#[ignore = "Strict equality contract (`#219`): eql_v2.neq is now hm-only. Same rationale as ore_text_eq_function."] async fn ore_text_neq_function(pool: PgPool) -> Result<()> { @@ - QueryAssertion::new(&pool, &sql).count(99).await; + QueryAssertion::new(&pool, &sql).count(0).await;Also applies to: 114-124
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/tests/ore_text_operator_tests.rs` around lines 99 - 100, Remove the #[ignore = ...] attribute from the ore_text_eq_function test and update its assertions to explicitly expect zero matching rows (i.e., assert that queries using eql_v2.eq on the ORE-only fixture return an empty result / count == 0); do the same for the corresponding neq test(s) (e.g., ore_text_neq_function) so they assert zero matches instead of being skipped, and keep references to the eql_v2.eq call and the ORE-only fixture rows (which only contain `ob` not `hm`) when changing the assertions.tests/sqlx/tests/ore_equality_tests.rs (1)
1-13: ⚡ Quick winRename these tests to
ore_block_*(or similar) to match what they execute.Current names (
ore_cllw_*) suggest sv-elementoccoverage, but the SQL compares rootore.epayloads and exercises Block ORE (ob) behavior. Renaming will make failures easier to triage.Also applies to: 27-70
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/tests/ore_equality_tests.rs` around lines 1 - 13, Rename the test functions and any related identifiers that currently use the prefix/suffix "ore_cllw_" to "ore_block_" (or a similar "ore_block_*" naming) so names reflect that these tests exercise Block ORE (compare path eql_v2.compare_ore_block_u64_8_256 against root ore.e payloads) rather than sv-element oc behavior; update the test function names in tests/sqlx/tests/ore_equality_tests.rs, adjust any test descriptions/comments that mention "CLLW" to "Block ORE" and ensure any references (module/test invocation or doc comments) are updated to the new names so failures are clearly attributed to Block ORE coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Line 33: Append the missing PR reference to each user-facing CHANGELOG entry
that currently lacks one—specifically add the PR link in parentheses at the end
of the entry describing the six operators on `eql_v2.ste_vec_entry` (the line
talking about `=`, `<>`, `<`, `<=`, `>`, `>=` that fold to `eql_v2.hmac_256(a) =
eql_v2.hmac_256(b)` and `eql_v2.ore_cllw(a) <op> eql_v2.ore_cllw(b)`), and
likewise add PR links to the other entries called out in the review (the nearby
entries about root-level `=` / `<` inlining and entry-shape behavior); follow
the project guideline format by ending each user-facing sentence with the PR
link in parentheses and keeping the section ordering (Added/Changed/etc.).
In `@docs/reference/database-indexes.md`:
- Line 52: The link fragment "`#u-006-stevec-ore-field-consolidation`" in the
"Range queries" line (the text referencing ste_vec and U-006) likely doesn't
match the actual heading slug for the U-006 section in upgrading/v2.3.md; open
that document, find the U-006 heading slug, and update the anchor in the Range
queries sentence to the exact slug (or replace with a stable full-path link to
the U-006 section) so the reference to U-006 (ste_vec) resolves correctly.
- Line 182: The example range-query uses the untyped extractor call
`eql_v2.ore_cllw(e->'<selector>'::text)` but must use the v2.3 typed ste_vec
entry API: update the example so the selector extraction accesses the JSONB
`.data` field and is cast to `eql_v2.ste_vec_entry` before calling
`eql_v2.ore_cllw`; i.e., replace the bare `eql_v2.ore_cllw(...)` argument that
uses `e->'<selector>'` with an expression that extracts `.data` from the ste_vec
entry and casts it to `eql_v2.ste_vec_entry` so the typed path is used
throughout (adjust both sides of the comparison example where present).
In `@docs/reference/schema/eql-payload-v2.3.schema.json`:
- Around line 99-104: The schema for the SteVecElement currently only requires
["s","c"], which allows elements missing the "hm" property; update the
SteVecElement definition so its "required" array includes "hm" (i.e., require
"s", "c", and "hm") to enforce the v2.3 contract and ensure the element-equality
path validation rejects payloads missing the hm field; modify the schema block
that defines the SteVecElement (the object with properties "c", "hm", "oc")
accordingly.
In `@docs/upgrading/v2.3.md`:
- Line 14: The intra-doc link fragment '`#u-006-stevec-ore-field-consolidation`'
does not match the heading slug for "### U-006: ste_vec ORE field
consolidation", so update the link to the actual heading slug (e.g. change the
reference to '`#u-006-ste_vec-ore-field-consolidation`') or alternatively rename
the heading to match the existing fragment; ensure the fragment and the heading
text for U-006 are identical so the anchor resolves.
- Line 176: The extractor-form example uses the wrong ste_vec entry shape:
update the example so it extracts the .data field and casts to
eql_v2.ste_vec_entry before calling the function; specifically change the
expression that currently shows eql_v2.ore_cllw(e->'<selector>'::text) to
extract e->'<selector>'->'data' (or access .data) and cast that result to
eql_v2.ste_vec_entry, then pass it into eql_v2.ore_cllw so the call becomes
eql_v2.ore_cllw((extracted_value::eql_v2.ste_vec_entry)). Ensure the doc text
mentions the required .data extraction and cast to eql_v2.ste_vec_entry when
demonstrating eql_v2.ore_cllw in the extractor form.
In `@src/encrypted/hash.sql`:
- Around line 30-33: Update the GROUP BY example to cast the JSON selector
result directly to ste_vec_entry instead of accessing .data; specifically
replace `GROUP BY eql_v2.hmac_256((col ->
'<selector>').data::eql_v2.ste_vec_entry)` with `GROUP BY eql_v2.hmac_256((col
-> '<selector>')::eql_v2.ste_vec_entry)` so the example for eql_v2.hmac_256 and
the field-level flow matches the typed ste_vec_entry usage and avoids the
incorrect `.data` access.
In `@src/ore_cllw/functions.sql`:
- Around line 196-220: The function compare_ore_cllw_term must treat a.bytes or
b.bytes as NULL-equivalent inputs; update the initial guard in
compare_ore_cllw_term so it returns NULL if either composite is NULL OR either
composite's bytes field is NULL (i.e. change the check at the start to test "a
IS NULL OR b IS NULL OR a.bytes IS NULL OR b.bytes IS NULL"). This ensures
callers like eql_v2.ore_cllw (which may produce non-NULL composites with bytes =
NULL) cause an immediate NULL return and prevents passing NULL bounds into
compare_ore_cllw_term_bytes.
---
Nitpick comments:
In `@tests/sqlx/tests/ore_equality_tests.rs`:
- Around line 1-13: Rename the test functions and any related identifiers that
currently use the prefix/suffix "ore_cllw_" to "ore_block_" (or a similar
"ore_block_*" naming) so names reflect that these tests exercise Block ORE
(compare path eql_v2.compare_ore_block_u64_8_256 against root ore.e payloads)
rather than sv-element oc behavior; update the test function names in
tests/sqlx/tests/ore_equality_tests.rs, adjust any test descriptions/comments
that mention "CLLW" to "Block ORE" and ensure any references (module/test
invocation or doc comments) are updated to the new names so failures are clearly
attributed to Block ORE coverage.
In `@tests/sqlx/tests/ore_text_operator_tests.rs`:
- Around line 99-100: Remove the #[ignore = ...] attribute from the
ore_text_eq_function test and update its assertions to explicitly expect zero
matching rows (i.e., assert that queries using eql_v2.eq on the ORE-only fixture
return an empty result / count == 0); do the same for the corresponding neq
test(s) (e.g., ore_text_neq_function) so they assert zero matches instead of
being skipped, and keep references to the eql_v2.eq call and the ORE-only
fixture rows (which only contain `ob` not `hm`) when changing the assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 45da87a7-0cbe-4e19-8221-9742cbb58097
📒 Files selected for processing (54)
CHANGELOG.mddocs/reference/database-indexes.mddocs/reference/schema/eql-payload-v2.3.schema.jsondocs/upgrading/v2.3.mdsrc/encrypted/hash.sqlsrc/hmac_256/functions.sqlsrc/ope_cllw_u64_65/compare.sqlsrc/ope_cllw_u64_65/functions.sqlsrc/ope_cllw_u64_65/types.sqlsrc/ope_cllw_var_8/compare.sqlsrc/ope_cllw_var_8/functions.sqlsrc/ope_cllw_var_8/types.sqlsrc/operators/<.sqlsrc/operators/<>.sqlsrc/operators/=.sqlsrc/operators/compare.sqlsrc/operators/order_by.sqlsrc/operators/sort.sqlsrc/operators/ste_vec_entry.sqlsrc/ore_cllw/functions.sqlsrc/ore_cllw/types.sqlsrc/ore_cllw_u64_8/compare.sqlsrc/ore_cllw_u64_8/functions.sqlsrc/ore_cllw_u64_8/operators.sql.skipsrc/ore_cllw_u64_8/types.sqlsrc/ore_cllw_var_8/compare.sqlsrc/ore_cllw_var_8/functions.sqlsrc/ore_cllw_var_8/operator_class.sql.skipsrc/ore_cllw_var_8/operators.sql.skipsrc/ore_cllw_var_8/operators_test_skip.sql.skipsrc/ore_cllw_var_8/types.sqlsrc/ste_vec/functions.sqlsrc/ste_vec/types.sqltasks/pin_search_path.sqltasks/test/splinter.shtests/sqlx/migrations/003_install_ste_vec_data.sqltests/sqlx/migrations/004_install_test_helpers.sqltests/sqlx/migrations/005_install_ste_vec_vast_data.sqltests/sqlx/tests/comparison_tests.rstests/sqlx/tests/hash_operator_tests.rstests/sqlx/tests/hmac_256_selector_tests.rstests/sqlx/tests/hmac_256_terms_tests.rstests/sqlx/tests/index_compare_tests.rstests/sqlx/tests/ope_tests.rstests/sqlx/tests/operator_class_tests.rstests/sqlx/tests/operator_compare_tests.rstests/sqlx/tests/order_by_sort_tests.rstests/sqlx/tests/ore_comparison_tests.rstests/sqlx/tests/ore_equality_tests.rstests/sqlx/tests/ore_text_operator_tests.rstests/sqlx/tests/payload_schema_tests.rstests/sqlx/tests/specialized_tests.rstests/ste_vec.sqltests/test_helpers.sql
💤 Files with no reviewable changes (18)
- src/ore_cllw_var_8/types.sql
- src/ore_cllw_var_8/compare.sql
- src/ope_cllw_u64_65/types.sql
- src/ope_cllw_u64_65/compare.sql
- src/ore_cllw_u64_8/types.sql
- src/ore_cllw_var_8/operators_test_skip.sql.skip
- src/ope_cllw_var_8/compare.sql
- src/ore_cllw_u64_8/functions.sql
- src/ore_cllw_var_8/functions.sql
- src/ope_cllw_var_8/functions.sql
- src/ore_cllw_u64_8/operators.sql.skip
- src/ore_cllw_u64_8/compare.sql
- src/ope_cllw_u64_65/functions.sql
- src/ore_cllw_var_8/operator_class.sql.skip
- src/ore_cllw_var_8/operators.sql.skip
- src/ope_cllw_var_8/types.sql
- tests/sqlx/tests/ope_tests.rs
- src/operators/order_by.sql
Reviewer flagged that the previous CHECK (`s` + `c` + `hm` required) was a bug — it didn't match the cipherstash-suite emission contract. The correct invariant: each sv element carries `s` + `c` (always), plus **exactly one** of `hm` (HMAC-256, for hash-equality-configured selectors) or `oc` (CLLW ORE, for ordering-configured selectors) — mutually exclusive. A given selector / field is configured for one mode or the other; the crypto layer emits the corresponding term and only that term. ```sql -- was (wrong, too strict): AND VALUE ? 's' AND VALUE ? 'c' AND VALUE ? 'hm' -- now: AND VALUE ? 's' AND VALUE ? 'c' AND (VALUE ? 'hm') <> (VALUE ? 'oc') ``` Without this fix, casting `.data::eql_v2.ste_vec_entry` raised against orderable-only sv elements (oc but no hm — the actual shape from cipherstash-suite for selectors configured with `ore`). The benches were silently working around it by using the (jsonb) overload of `ore_cllw`. Downstream changes to restore the XOR invariant in test fixtures and helpers: - `tests/test_helpers.sql:get_numeric_ste_vec_*()`: drop `hm` from `oc`-bearing sv elements that I had added in commit 6bde3e3 to satisfy the old over-strict check. - `tests/ste_vec.sql` (inlined fixture rewritten in a77aeef): same cleanup — every sv element is now XOR-conforming. - `create_encrypted_json(id)`'s hm-backstop block: only synthesise `hm` when neither `hm` nor `oc` is present (was: always, via coalesce). Preserves the v2.2 → v2.3 b3→hm bridge for legacy fixtures while respecting the XOR invariant. - `get_numeric_ste_vec_*` `$.hello` sv element: swap `oc` → `hm` (same bytes, renamed). The fixture's *intent* is "hash equality on a string", which the v2.2 fixture happened to carry as `oc` — swapping aligns the legacy fixture with its semantic role, so field-level `hmac_256(e, '<hello-selector>')` tests work without resurrecting hm-synthesis on oc entries. `src/ste_vec/functions.sql:ste_vec_contains`: element-match logic extended. The function previously matched sv elements by `hm` and fell through to `eql_v2.eq` (root compare, post-#219 strict Block-ORE-only) for non-hash-indexed elements. With the XOR invariant enforced, orderable-only entries never have `hm` — the fallback would always raise. Add a parallel `has_ore_cllw` branch that compares via `compare_ore_cllw_term` on the extracted `oc` bytes, mirroring the `hm` branch. Both `hm` and `oc` are deterministic for same-plaintext at same-selector under the same workspace, so either serves as an equality discriminator between matching sv elements. Tests: 38/38 sqlx test files pass locally after the changes.
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
tests/test_helpers.sql (1)
79-95:⚠️ Potential issue | 🟡 Minor | ⚡ Quick win
get_array_ste_vec()still emits a legacyb3-only sv entry.The first element in this fixture has
b3but neitherhmnoroc, so it no longer matches the v2.3ste_vec_entryinvariant. Any test path that consumesget_array_ste_vec()directly will miss equality matches becauseste_vec_containsonly checkshm/ocnow.Suggested fix
{ - "b3": "7b4ffe5d60e4e4300dc3e28d9c300c87", + "hm": "7b4ffe5d60e4e4300dc3e28d9c300c87", "c": "mBbL9j9(QoRD)R+z?=Fvn#=FRIg79JJM`MCq+nE0*U^ca-cViL884d-TInfY&E9HW@X>!U&lkYne2!EecKG8xwLYb0X#y7|05rrPvwh?Ejvk%78G}b+je+xufQA5mSwHSid)iEOkg@>mpuh", "s": "bca213de9ccce676fa849ff9c4807963" },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_helpers.sql` around lines 79 - 95, The fixture returned by get_array_ste_vec() contains an sv entry with only "b3", which violates the v2.3 ste_vec_entry invariant (ste_vec_contains now checks "hm"/"oc" not "b3"); update the first sv element in get_array_ste_vec() to include a valid "oc" (or "hm") field with an appropriate hex digest (or replace "b3" with "oc") so the fixture matches the ste_vec_entry shape expected by ste_vec_contains; locate get_array_ste_vec() and its sv array and add/rename the field accordingly.tests/sqlx/migrations/004_install_test_helpers.sql (1)
79-95:⚠️ Potential issue | 🟡 Minor | ⚡ Quick win
get_array_ste_vec()still emits a legacyb3-only sv entry.The first element in this fixture has
b3but neitherhmnoroc, so it no longer matches the v2.3ste_vec_entryinvariant. Any test path that consumesget_array_ste_vec()directly will miss equality matches becauseste_vec_containsonly checkshm/ocnow.Suggested fix
{ - "b3": "7b4ffe5d60e4e4300dc3e28d9c300c87", + "hm": "7b4ffe5d60e4e4300dc3e28d9c300c87", "c": "mBbL9j9(QoRD)R+z?=Fvn#=FRIg79JJM`MCq+nE0*U^ca-cViL884d-TInfY&E9HW@X>!U&lkYne2!EecKG8xwLYb0X#y7|05rrPvwh?Ejvk%78G}b+je+xufQA5mSwHSid)iEOkg@>mpuh", "s": "bca213de9ccce676fa849ff9c4807963" },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/migrations/004_install_test_helpers.sql` around lines 79 - 95, The fixture JSON returned by get_array_ste_vec() contains a legacy sv entry with only "b3" which violates the v2.3 ste_vec_entry invariant (tests expect an "hm" or "oc" field that ste_vec_contains checks); update the first sv object inside get_array_ste_vec() to include a valid "oc" (or "hm") value (and remove or replace "b3" if desired) so the entry conforms to the current ste_vec_entry shape and will be matched by ste_vec_contains.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/ste_vec/functions.sql`:
- Line 7: Add the missing REQUIRE entries so this SQL file declares its
dependency on the ORE objects used; specifically, update the REQUIRE block at
the top to include the package/schema that exposes eql_v2.has_ore_cllw,
eql_v2.compare_ore_cllw_term, and eql_v2.ore_cllw (the same src/ore_cllw/*
dependency set used by those symbols) so the install order is enforced and these
functions are available when the statements referencing them execute.
---
Outside diff comments:
In `@tests/sqlx/migrations/004_install_test_helpers.sql`:
- Around line 79-95: The fixture JSON returned by get_array_ste_vec() contains a
legacy sv entry with only "b3" which violates the v2.3 ste_vec_entry invariant
(tests expect an "hm" or "oc" field that ste_vec_contains checks); update the
first sv object inside get_array_ste_vec() to include a valid "oc" (or "hm")
value (and remove or replace "b3" if desired) so the entry conforms to the
current ste_vec_entry shape and will be matched by ste_vec_contains.
In `@tests/test_helpers.sql`:
- Around line 79-95: The fixture returned by get_array_ste_vec() contains an sv
entry with only "b3", which violates the v2.3 ste_vec_entry invariant
(ste_vec_contains now checks "hm"/"oc" not "b3"); update the first sv element in
get_array_ste_vec() to include a valid "oc" (or "hm") field with an appropriate
hex digest (or replace "b3" with "oc") so the fixture matches the ste_vec_entry
shape expected by ste_vec_contains; locate get_array_ste_vec() and its sv array
and add/rename the field accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9f5b3b6d-694e-473d-9a19-660d6b5b9861
📒 Files selected for processing (7)
CHANGELOG.mdsrc/ste_vec/functions.sqlsrc/ste_vec/types.sqltests/sqlx/migrations/003_install_ste_vec_data.sqltests/sqlx/migrations/004_install_test_helpers.sqltests/ste_vec.sqltests/test_helpers.sql
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/ste_vec.sql
- CHANGELOG.md
freshtonic
approved these changes
May 20, 2026
Contributor
freshtonic
left a comment
freshtonic
left a comment
There was a problem hiding this comment.
@coderdan give me a walkthrough of this PR. We uncovered some things that need fixing - which Dan recorded as PR comments.
The changes are generally solid and well-tested.
My approval is conditional on the comments being addressed.
- Reword the domain-tag-byte description: the tag is prepended to the plaintext *prior* to CLLW encryption, not on the ciphertext. CLLW's per-byte order preservation carries the implied ordering through to the encrypted `oc` value. (Dan review) - Tighten the `ob` / `oc` scope statement: they're scope-disjoint (`ob` is root-only, `oc` is sv-only), not "mutually exclusive within a single payload" — the previous wording suggested either term could appear at either scope. (Dan review) - Drop the link to cipherstash-suite#1955 — that repo is private and links won't resolve for external readers. Keep the prose reference. (Dan review) - Simplify the U-006 range-query example to use the typed `<` operator on `ste_vec_entry × ste_vec_entry` (inlines through `ore_cllw(a) < ore_cllw(b)`) instead of calling the extractors explicitly. (Dan review) - Fix broken markdown link fragments — heading slug for "U-006: ste_vec ORE field consolidation" includes the underscore (`u-006-ste_vec-ore-field-consolidation`), not the previously-used `u-006-stevec-...`. Three references updated across v2.3.md and database-indexes.md. (CodeRabbit)
CodeRabbit flagged: several `[Unreleased]` entries didn't end with a PR reference per the CHANGELOG discipline at the top of the file. Add `([#219](…))` to entries that landed in the current PR (ste_vec_entry DOMAIN, typed extractor family, entry-typed operators, ore_cllw consolidation, OPE-CLLW removal) and `([#211](…))` to the range-operator inlining + strict-Block-ORE-compare entries. Also fixes the stale `u-006-stevec-...` anchor slug to `u-006-ste_vec-ore-field-consolidation` in CHANGELOG.md (last straggler after the earlier sweep across v2.3.md / database-indexes.md).
- hash.sql (Dan #16): trim the defensive `@par Contract` block — the contract is "hm is always present when GROUP BY / DISTINCT / hash joins are used; missing hm is a misconfiguration surfaced by U-002 upstream." Remove the verbose hashtext-NULL-fallthrough description that suggested defensive handling here. - ore_cllw/functions.sql (CodeRabbit + Dan #19/#20): widen the null-guard in `compare_ore_cllw_term` to also short-circuit when the composite's `bytes` field is NULL. The (jsonb) and (ste_vec_entry) overloads of `eql_v2.ore_cllw` return `(bytes => NULL)` for payloads lacking `oc`; passing that into `compare_ore_cllw_term_bytes` would RAISE on the NULL bytea length check. Treat as incomparable, return NULL. - ste_vec/functions.sql (CodeRabbit #22): declare the new `src/ore_cllw/*` REQUIRE deps explicitly — `ste_vec_contains`' XOR-aware element-compare branch calls `eql_v2.has_ore_cllw`, `eql_v2.ore_cllw`, and `eql_v2.compare_ore_cllw_term`. Without the REQUIRE the install order is fragile. - ste_vec/functions.sql (Dan #23): rename `eql_v2.selector(eql_v2_encrypted)` to `eql_v2._selector` and mark `@internal`. It's an internal convenience that unwraps the composite and delegates to `eql_v2.selector(jsonb)`. Public callers should use the (jsonb) or (ste_vec_entry) overload. Updates internal call sites in `->`, `->>`, `jsonb_path_*`, and `ste_vec_contains`. One test (`contains_does_not_match_different_hm`) used it for inspection; switched to direct JSONB field access (`((e -> 'sel').data) ->> 's'`).
- ste_vec_vast_data.sql (Dan #25): strip the root-level `"c": "..."` field from all 500 INSERT payloads. `c` is not valid at the root of an SteVec-shape EQL value — per the v2.3 payload schema, sv payloads carry only `i`, `v`, `k`, `sv` at the root. The vast_data fixture was generated from an earlier ste_vec.sql shape that had root c, and it slipped through the v2.3 cleanup. - test_helpers.sql (Dan #34): drop the `coalesce(elem ->> 'b3', ...)` fallback in the hm-backstop CASE. `b3` is removed from EQL 2.3 entirely (the `eql_v2.blake3` family is gone) so the coalesce was dead code. Just synthesise `hm` as `md5(s || c)` for legacy entries lacking both terms.
- specialized_tests::ste_vec_contains_self (Dan #31): drop the `#[ignore]` — the ignore message was stale. The `get_numeric_ste_vec_10` fixture already carries `hm` on its hm-bearing sv elements (root, $.hello) and `oc` on its $.n element; ste_vec_contains_self passes under the strict-equality contract once the EQL post-#219 install is in place. - ore_text_operator_tests::ore_text_eq_function / ore_text_neq_function (Dan #27/#28): add deterministic `hm` to every ore_text fixture row (md5 of plaintext) so the hm-only equality contract can find matches. Drop the `#[ignore]` on both. 100 rows updated. - payload_schema_tests::v2_3_ste_vec_payload_with_hm_in_elements_is_valid (Dan #29): rename to `v2_3_ste_vec_payload_with_mixed_hm_and_oc_elements_is_valid` — the test actually exercises the mixed-element shape (one hm-bearing, one oc-bearing) and the previous name implied hm-only coverage. - payload_schema_tests::v2_3_oc_at_root_is_rejected (Dan #30): new negative test asserting that `oc` at root of an EncryptedPayload is rejected (scope-disjoint per U-006). The schema's `additionalProperties: false` on EncryptedPayload already enforces this; the new test makes the invariant explicit.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/sqlx/migrations/004_install_test_helpers.sql (1)
199-202:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAlign
hmbackfill logic with v2.3 helper to avoid cross-harness drift.Line 199-Line 202 still backfills from legacy
b3, but the sibling helper (tests/test_helpers.sql) now treatsb3as removed and synthesizeshmfrommd5(s || c)only. This makes sqlx-migration fixtures behave differently from non-sqlx fixtures, which can hide or create test-only failures across suites.Proposed fix
- coalesce( - elem ->> 'b3', - md5(coalesce(elem ->> 's', '') || coalesce(elem ->> 'c', '')) - ) + md5(coalesce(elem ->> 's', '') || coalesce(elem ->> 'c', ''))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/migrations/004_install_test_helpers.sql` around lines 199 - 202, The backfill still uses the legacy 'b3' value (coalesce(elem ->> 'b3', md5(...))) which diverges from the v2.3 helper that always synthesizes 'hm' from md5(s || c); change the expression to drop the 'b3' fallback and compute hm directly as md5(coalesce(elem ->> 's', '') || coalesce(elem ->> 'c', '')) so the migration aligns with the sibling helper's behavior and avoids cross-harness drift.
♻️ Duplicate comments (1)
docs/reference/database-indexes.md (1)
182-182:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUse the typed STE-vector extractor form in the range-query example.
At Line 182, the example still shows the untyped call path (
eql_v2.ore_cllw(e->'<selector>'::text)), which is inconsistent with the v2.3 typedste_vec_entryflow used in this PR. Please update the sample to extract/cast via the typed entry path before callingeql_v2.ore_cllw(...).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/reference/database-indexes.md` at line 182, The example uses the untyped extractor e->'<selector>'::text; update it to the typed STE-vector extractor path by calling the ste_vec_entry extractor on the column (with '<selector>') and casting its result to the ste_vec_entry type before passing it into eql_v2.ore_cllw; ensure the same typed-extractor/cast form is used for both operands in the range comparison so the example matches the v2.3 ste_vec_entry flow (symbols: ste_vec_entry, eql_v2.ore_cllw).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@tests/sqlx/migrations/004_install_test_helpers.sql`:
- Around line 199-202: The backfill still uses the legacy 'b3' value
(coalesce(elem ->> 'b3', md5(...))) which diverges from the v2.3 helper that
always synthesizes 'hm' from md5(s || c); change the expression to drop the 'b3'
fallback and compute hm directly as md5(coalesce(elem ->> 's', '') ||
coalesce(elem ->> 'c', '')) so the migration aligns with the sibling helper's
behavior and avoids cross-harness drift.
---
Duplicate comments:
In `@docs/reference/database-indexes.md`:
- Line 182: The example uses the untyped extractor e->'<selector>'::text; update
it to the typed STE-vector extractor path by calling the ste_vec_entry extractor
on the column (with '<selector>') and casting its result to the ste_vec_entry
type before passing it into eql_v2.ore_cllw; ensure the same
typed-extractor/cast form is used for both operands in the range comparison so
the example matches the v2.3 ste_vec_entry flow (symbols: ste_vec_entry,
eql_v2.ore_cllw).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f9efba61-62a1-40eb-ac0a-bb3ccc7a9e86
📒 Files selected for processing (17)
CHANGELOG.mddocs/reference/database-indexes.mddocs/reference/schema/eql-payload-v2.3.schema.jsondocs/upgrading/v2.3.mdsrc/encrypted/hash.sqlsrc/jsonb/functions.sqlsrc/operators/->.sqlsrc/operators/->>.sqlsrc/ore_cllw/functions.sqlsrc/ste_vec/functions.sqltests/sqlx/migrations/004_install_test_helpers.sqltests/sqlx/migrations/005_install_ste_vec_vast_data.sqltests/sqlx/migrations/006_install_ore_text_data.sqltests/sqlx/tests/containment_tests.rstests/sqlx/tests/payload_schema_tests.rstests/sqlx/tests/specialized_tests.rstests/test_helpers.sql
💤 Files with no reviewable changes (1)
- tests/sqlx/tests/specialized_tests.rs
✅ Files skipped from review due to trivial changes (2)
- CHANGELOG.md
- docs/upgrading/v2.3.md
coderdan
added a commit
that referenced
this pull request
May 20, 2026
Restores functional-index match for sv-element ordered queries after the consolidation in #219 left the type without an opclass. Closes #220. `src/ore_cllw/operators.sql` — same-type comparison operators (`<`, `<=`, `=`, `>=`, `>`, `<>`) on `eql_v2.ore_cllw`. Each operator is backed by a single-statement `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` wrapper that reduces to `eql_v2.compare_ore_cllw_term(a, b) <op> 0`. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match. `src/ore_cllw/operator_class.sql` — `eql_v2.ore_cllw_ops` btree opclass registered `DEFAULT FOR TYPE eql_v2.ore_cllw`. FUNCTION 1 is `eql_v2.compare_ore_cllw_term` directly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase build variant via the existing `**/*operator_class.sql` glob in `tasks/build.sh` (operator classes require superuser). `tasks/pin_search_path.sql` — allowlists the six operator backing functions (`ore_cllw_eq` / `_neq` / `_lt` / `_lte` / `_gt` / `_gte`). Pinning would break the inlining chain and prevent the planner from structurally matching predicates to functional indexes. - **No `HASHES` / `MERGES` flags** on the operator declarations. HASHES needs a registered hash function on the type (no, and we don't want one — the CLLW protocol is for ordering, not hashing). MERGES needs an equivalent operator family on both sides, which we'd register separately if/when needed. This is the gap that disabled the pre-2025-06-24 opclass; see issue #220's history. - **Equality via `compare_ore_cllw_term = 0`**, not a `bytea_eq` shortcut. Consistent with the rest of the CLLW path; one source of truth for equality semantics; resilient to any future change in the underlying ciphertext encoding. - **The opclass operators are different from the operators on `eql_v2_encrypted`.** Those (per #211) inline to `ore_block_u64_8_256` and raise on non-Block-ORE columns. The new operators here are on the `eql_v2.ore_cllw` composite type itself — what callers reach through the extractor form `WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1)`. No conflict, different scope. `tests/sqlx/tests/ore_cllw_opclass_tests.rs` covers: - Operator wiring: `=`, `<>`, `<`, `<=`, `>`, `>=` on hand-crafted byte strings under the CLLW per-byte protocol. - Cross-domain ordering via the leading tag byte (`0x00` numeric, `0x01` string) — numeric < string within the same column. - Opclass registration: `pg_opclass.opcdefault = true` for `eql_v2.ore_cllw_ops`. - Functional-index match: build a functional btree on `eql_v2.ore_cllw(value)`, confirm `EXPLAIN` for `ORDER BY eql_v2.ore_cllw(value) LIMIT n` shows `Index Scan` (or `Index Only Scan`) and no `Sort` node. - Inlinability lint: read `pg_proc` directly, assert each backing function is `LANGUAGE sql`, `IMMUTABLE`, `STRICT`, `PARALLEL SAFE`, and not pinned with `SET search_path`. From the bench results in cipherstash/benches#14 (post-#219 baseline): json/field_order/functional @ 1M = 20.0 s (no opclass; seq scan + Sort) With this opclass, `EXPLAIN` flips to `Index Scan + Limit` and the same query should land in single-digit ms on the bench rig. End-to-end bench re-run is the next step on the bench-side branch. Docs: CHANGELOG `Added` entry, U-005 action-required note refreshed, database-indexes.md ORE-CLLW recipe entry refreshed.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
Two fixes for `tests/sqlx/tests/ore_cllw_opclass_tests.rs` after the #219 strict-separation refactor merged into the base branch: - `functional_index_engages_for_order_by`: the index expression and ORDER BY clause referenced `eql_v2.ore_cllw(value)` (where `value` is an `eql_v2_encrypted` column). That overload was removed in #219; the typed replacement is `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` or the (jsonb) form. Switch both to `eql_v2.ore_cllw((value).data)`, which uses the (jsonb) overload and keeps the index/predicate expressions structurally identical so the planner still matches. - Remove the `ANALYZE ore_cllw_test` call. The `value` column is `eql_v2_encrypted` with payloads that carry only `oc` (no root `ob`). ANALYZE samples via the default btree opclass on `eql_v2_encrypted` (FUNCTION 1 = `eql_v2.compare`), and the post-#219 strict-Block-ORE `compare` raises on missing `ob`. The functional-index match still works without stats once `enable_seqscan = off` is set, so dropping ANALYZE is the cleanest path. Also refreshes `tests/sqlx/migrations/001_install_eql.sql` to the current built release SQL (this file is a build artefact that the mise test task copies from `release/`; keeping it in sync with the new opclass + entry types lets `cargo test` run against the up-to-date schema directly).
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
Restores functional-index match for sv-element ordered queries after the consolidation in #219 left the type without an opclass. Closes #220. `src/ore_cllw/operators.sql` — same-type comparison operators (`<`, `<=`, `=`, `>=`, `>`, `<>`) on `eql_v2.ore_cllw`. Each operator is backed by a single-statement `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` wrapper that reduces to `eql_v2.compare_ore_cllw_term(a, b) <op> 0`. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match. `src/ore_cllw/operator_class.sql` — `eql_v2.ore_cllw_ops` btree opclass registered `DEFAULT FOR TYPE eql_v2.ore_cllw`. FUNCTION 1 is `eql_v2.compare_ore_cllw_term` directly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase build variant via the existing `**/*operator_class.sql` glob in `tasks/build.sh` (operator classes require superuser). `tasks/pin_search_path.sql` — allowlists the six operator backing functions (`ore_cllw_eq` / `_neq` / `_lt` / `_lte` / `_gt` / `_gte`). Pinning would break the inlining chain and prevent the planner from structurally matching predicates to functional indexes. - **No `HASHES` / `MERGES` flags** on the operator declarations. HASHES needs a registered hash function on the type (no, and we don't want one — the CLLW protocol is for ordering, not hashing). MERGES needs an equivalent operator family on both sides, which we'd register separately if/when needed. This is the gap that disabled the pre-2025-06-24 opclass; see issue #220's history. - **Equality via `compare_ore_cllw_term = 0`**, not a `bytea_eq` shortcut. Consistent with the rest of the CLLW path; one source of truth for equality semantics; resilient to any future change in the underlying ciphertext encoding. - **The opclass operators are different from the operators on `eql_v2_encrypted`.** Those (per #211) inline to `ore_block_u64_8_256` and raise on non-Block-ORE columns. The new operators here are on the `eql_v2.ore_cllw` composite type itself — what callers reach through the extractor form `WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1)`. No conflict, different scope. `tests/sqlx/tests/ore_cllw_opclass_tests.rs` covers: - Operator wiring: `=`, `<>`, `<`, `<=`, `>`, `>=` on hand-crafted byte strings under the CLLW per-byte protocol. - Cross-domain ordering via the leading tag byte (`0x00` numeric, `0x01` string) — numeric < string within the same column. - Opclass registration: `pg_opclass.opcdefault = true` for `eql_v2.ore_cllw_ops`. - Functional-index match: build a functional btree on `eql_v2.ore_cllw(value)`, confirm `EXPLAIN` for `ORDER BY eql_v2.ore_cllw(value) LIMIT n` shows `Index Scan` (or `Index Only Scan`) and no `Sort` node. - Inlinability lint: read `pg_proc` directly, assert each backing function is `LANGUAGE sql`, `IMMUTABLE`, `STRICT`, `PARALLEL SAFE`, and not pinned with `SET search_path`. From the bench results in cipherstash/benches#14 (post-#219 baseline): json/field_order/functional @ 1M = 20.0 s (no opclass; seq scan + Sort) With this opclass, `EXPLAIN` flips to `Index Scan + Limit` and the same query should land in single-digit ms on the bench rig. End-to-end bench re-run is the next step on the bench-side branch. Docs: CHANGELOG `Added` entry, U-005 action-required note refreshed, database-indexes.md ORE-CLLW recipe entry refreshed.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
Two fixes for `tests/sqlx/tests/ore_cllw_opclass_tests.rs` after the #219 strict-separation refactor merged into the base branch: - `functional_index_engages_for_order_by`: the index expression and ORDER BY clause referenced `eql_v2.ore_cllw(value)` (where `value` is an `eql_v2_encrypted` column). That overload was removed in #219; the typed replacement is `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` or the (jsonb) form. Switch both to `eql_v2.ore_cllw((value).data)`, which uses the (jsonb) overload and keeps the index/predicate expressions structurally identical so the planner still matches. - Remove the `ANALYZE ore_cllw_test` call. The `value` column is `eql_v2_encrypted` with payloads that carry only `oc` (no root `ob`). ANALYZE samples via the default btree opclass on `eql_v2_encrypted` (FUNCTION 1 = `eql_v2.compare`), and the post-#219 strict-Block-ORE `compare` raises on missing `ob`. The functional-index match still works without stats once `enable_seqscan = off` is set, so dropping ANALYZE is the cleanest path. Also refreshes `tests/sqlx/migrations/001_install_eql.sql` to the current built release SQL (this file is a build artefact that the mise test task copies from `release/`; keeping it in sync with the new opclass + entry types lets `cargo test` run against the up-to-date schema directly).
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…edles This is the centerpiece of the StEVec query-path correctness fix. The single root cause behind several silent-wrong-result bugs was that `->` returned the synthetic-root `eql_v2_encrypted` rather than a typed `eql_v2.ste_vec_entry`. Cascading consequences: - `WHERE col -> 'sel' = $1` resolved to root-level `=` on `eql_v2_encrypted`, which extracts `hm` from the synthetic root — worked incidentally on hm-bearing selectors and broke silently on oc-bearing ones (NULL = NULL → false). - `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts `ob` (Block ORE). sv entries have no `ob`, so the strict extractor raised. - `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219. Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround. This commit flips `->` (all three overloads — text selector, encrypted selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text overload now uses `jsonb_path_query_first` over the sv array; the integer overload uses direct jsonb indexing; both merge the root's `{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK on `ste_vec_entry` tolerates extra fields, so the merged shape `{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors (`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All three overloads are inlinable single-statement SQL. With the typed selection in place, this commit also lands the typed containment needles: - `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe. `stevec_query` is the right-hand sv-shaped payload type (no `c` fields) defined in the previous commit. The operator delegates to `ste_vec_contains` after wrapping via `to_encrypted`. - `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for "does this payload contain this specific entry?", e.g. `e @> (e -> 'sel')`. Wraps the entry into a single-element sv array (stripping `c`, which the containment logic ignores anyway). - `<@` mirror overloads for both forms. All new operators are allowlisted in `tasks/pin_search_path.sql` so the post-install `ALTER FUNCTION ... SET search_path` pass doesn't defeat inlinability. Tests that depended on the prior synthetic-root return are updated: the `term` text returned by helpers like `get_encrypted_term` is now a JSON object literal (since `->` returns jsonb-domain), so it casts naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment test in `containment_tests.rs` is re-expressed via stevec_query (the original `(entry) @> encrypted` shape is now type-prevented); `specialized_tests.rs` similarly re-expresses through the typed operators rather than direct `ste_vec_contains` calls. `assertions.rs::QueryAssertion::count`/`returns_rows` now surface the PG error string in the panic message — saves a round of bisection when the SQL is shape-wrong rather than semantically wrong. The synthetic-root caveat in the prior `->` doc comment is gone: this commit *is* the refactor that comment foresaw.
coderdan
added a commit
that referenced
this pull request
May 20, 2026
…hain The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was a fused selector-match + hm-extract introduced as the migration path off Blake3 for U-004. It bypasses the typed API and silently NULLs in two distinct cases (selector miss, hm absent on the matched oc-bearing element) without distinguishing them — which became actively misleading once selectors can be either hm-bearing or oc-bearing under the XOR contract. Post the `->` flip and the new `eq_term` extractor, the canonical recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and oc selectors with one expression, and the chained form composes with the rest of the typed model (operators, casts, containment). Drops: - `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)` function definition. - `tasks/pin_search_path.sql` — allowlist entry for the dropped function. - `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests for the dropped function. Adds equivalent coverage: - `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and oc-bearing entries, STRICT NULL propagation, target-element pick in multi-entry sv arrays, functional hash index engagement for bare `WHERE` and `GROUP BY`. Recipe migration in docs: - `docs/reference/database-indexes.md` — per-selector hash index recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`. - `src/encrypted/hash.sql` — `@note` rewritten to point at the new recipe (the post-#219 cast workaround is no longer needed). Companion updates: - `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index` now reads the entry's `hm` hex via JSONB field access on the new `ste_vec_entry`-typed `->` result, instead of the dropped fused form. - `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)` on `hmac_256_terms` swapped for the active `eq_term` reference. Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow in a later commit alongside the broader changelog/upgrade work.
tobyhede
added a commit
that referenced
this pull request
May 20, 2026
PR #219 removed OPE-CLLW from EQL; eql_v2_int4_ord_ope was built entirely on the removed ope_cllw_u64_65 module. Removing it restores a green build before the int4-family rework.
4 tasks
tobyhede
added a commit
that referenced
this pull request
May 21, 2026
PR #219 removed OPE-CLLW from EQL; eql_v2_int4_ord_ope was built entirely on the removed ope_cllw_u64_65 module. Removing it restores a green build before the int4-family rework.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #207.
Summary
The pre-2.3
ocf(fixed-width, numeric) andocv(variable-width, text) ORE-CLLW split on sv elements collapses to a singleocfield. Width information that used to be carried by the field name is now part of the plaintext via a leading domain-tag byte (0x00numeric,0x01string) and encrypted into the CLLW ciphertext. Walking the CLLW per-byte protocol over[tag][ciphertext]preserves cross-domain separation (numeric < string) and within-domain ordering — so a single column can hold mixed numeric and string sv elements with a consistent total order.OPE removed from
eql_v2_encrypted. Per recent product decision, OPE-based ordered comparison moves to a separate encrypted column type in a future release;eql_v2_encryptednow exclusively handles ORE for ordered comparisons. v2.2 didn't emitopf/opvfrom production@cipherstash/protect, so this isn't a customer-facing data migration — it's a tightening of the documented payload surface.What changes
SQL source
src/ore_cllw_u64_8/+src/ore_cllw_var_8/→src/ore_cllw/(single composite type, single extractor, single compare)eql_v2.ore_cllw(jsonb)extractor: inlinable single-statement SQL. Returns composite withbytes IS NULLrather than raising on missingoc— required for inlinability. Allowlisted intasks/pin_search_path.sql.eql_v2.compare_ore_cllwstays plpgsql: the CLLW per-byte protocol (y + 1 == xmod 256) can't be expressed as a single SELECT.ore_cllw_u64_8/ore_cllw_var_8→ore_cllw; same forhas_*andcompare_*.eql_v2.compare()priority list shortens:ob → oc → hm → literal.src/ope_cllw_*/directories deleted;eql_v2.order_by_operemoved;src/operators/sort.sqldrops the entire'ope'strategy +_compare_ope_keyhelper +ope_keysplumbing.eql_v2.hash_encryptedflipped to inlinable single-statement SQL with data-hash fallback (required for ANALYZE to succeed on sv-shaped payloads).Schema (
docs/reference/schema/eql-payload-v2.3.schema.json)EncryptedPayload+SteVecElement:ocf/ocvremoved, singleocadded.opremoved entirely.OrderedExclusioncollapses to a singlenot (ob AND oc).IndexTerms:ocdescription documents the domain-tag byte.Tests
tests/test_helpers.sql+tests/ste_vec.sql+tests/sqlx/migrations/{003,004,005}_*.sqlfixtures: bulk renameocf/ocv→ocin JSON literals.ope_tests.rsdeleted.payload_schema_tests.rs: legacyocf/ocv/opf/opvrejected by v2.3 schema;opalso rejected (no OPE path);ob×ocmutual exclusion asserted at both root and sv-element levels._strsuffix on the second occurrence.Docs
CHANGELOG.md: entries underAdded/Changed/Removedrewritten for ORE-only consolidation.Removeddocuments OPE-CLLW removal fromeql_v2_encrypted.U-006: ste_vec ORE field consolidationwith theocf/ocv→octable, re-encryption action, and forward-pointer about OPE's future home.docs/reference/database-indexes.md: ORE-CLLW recipe refreshed; OPE references removed.Behaviour callers should know about
ocf/ocvno longer satisfyeql_v2.has_ore_cllwand fall through to the literal-jsonb tiebreaker ineql_v2.compare.ore_cllwfamily. The composite typeseql_v2.ore_cllw_u64_8/eql_v2.ore_cllw_var_8are removed.Full migration notes: U-006.
Test plan
mise run buildsucceeds; release SQL contains noope_cllw/ocf/ocv/opreferences.mise run testgreen — SQLx Rust tests pass, lint clean.ocf/ocv/opf/opv/opall rejected by v2.3.eql_v2.ore_cllw— see follow-up Restore operator class for CLLW ORE (eql_v2.ore_cllw) #220).Follow-ups
field_orderqueries.Summary by CodeRabbit
Release Notes
New Features
Changed
hmonly; range queries useobfor root scalars andocfor vector elements.Removed