Repository navigation
Go v0.1.0-rc.3
Pre-releaseGo v0.1.0-rc.3 adds JSONPath filters and queries that match several members of the same JSON array element. It also fixes validation, pagination and token cleanup, and changes how EQL values print and log.
Upgrading from rc.2
Install this release's EQL v4 SQL bundle before using the new JSON queries. The rc.2 bundle does not contain the required types and functions, so PostgreSQL will reject those queries with a missing-type error.
Download the attached eql-v4-install.sql, then run:
psql -f eql-v4-install.sqlThe new queries require:
| Go API | SQL parameter type | SQL function |
|---|---|---|
Fields.<Field>.Match |
eql_v4.query_json_predicate |
eql_v4.json_tree_match |
Fields.<Field>.Elements.<Array>.Match |
eql_v4.query_json_elements |
eql_v4.json_tree_element_match |
Restore permissions and indexes after installation
Re-running the bundle preserves your columns and data because the column types live in public.
It drops and re-creates the eql_v4 and eql_v4_internal schemas with CASCADE. This removes indexes built on EQL functions, such as eql_v4.ord_term(col), along with schema and function grants and any ALTER DEFAULT PRIVILEGES in those schemas. The installer issues no GRANT statements.
After installation, before the application sends queries:
-
Restore permissions if the application connects as a different role from the installer. Grant
USAGEon botheql_v4andeql_v4_internal, andEXECUTEon all functions in both schemas. Equality, ordering and JSON queries reach the internal schema too. Restore any default privileges you previously configured.Without these grants, application queries fail with errors such as
permission denied for schema eql_v4. The permissions reference lists the statements; replaceeql_v3in its examples witheql_v4. -
Re-create the indexes, then run
ANALYZE.
Added
Bind a whole JSONPath filter as one parameter (#1443)
Every JsonSearch field now has:
Fields.<Field>.Match(ctx, cipher, predicate, vars, opts)It accepts the same inputs as plaintext jsonb_path_match(col, $1, $2): a JSONPath predicate and its variables. It returns an eql.JSONSearchPredicateQuery for either SQL form:
col @@ $1::eql_v4.query_json_predicate
eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)Variables
- Pass any value that
encoding/jsonwrites as an object, or ajson.RawMessagecontaining an object. - Use an empty map when there are no variables.
- A nil
varsvalue—including a nil map, slice or pointer—means SQLNULL, as it does when pgx ordatabase/sqlbinds it for plaintext queries. The operand is empty, binds asNULLand matches no rows. - A value that encodes as JSON
nullis rejected because it is not an object, matching plaintext behavior.
Supported grammar
| Construct | Supported forms |
|---|---|
| Paths | .key, ."quoted key", [*] |
| Comparisons against a variable or JSON literal | ==, !=, <, <=, >, >= |
| Existence | exists(path) |
| Filtered existence | exists(path ? (@.a == $x && @.b > $y)) |
| Boolean expressions | &&, ||, !(…) |
The engine parses the predicate and compiles it into a tree of the operands derived by EqAt, Selector, At.<Path> and Elements.<Array>.Match. The Go side does not parse the predicate.
Errors and query behavior
- Unsupported constructs return an
*encrypt.Diagnosticwrappingencrypt.ErrEncoding. The compiler never silently drops a leaf from the tree. - Ordering at a path that the field's tag does not order returns
encrypt.ErrTerm. - Results follow plaintext
jsonb_path_match, with five documented exceptions: lax array unwrapping, lax array wrapping, comparisons between strings and numbers, folded ordering terms, and ordering comparisons involvingnull. In the last case, the result is SQLNULLrather than false outside an odd number of!operators, which returns the same rows.
Index use: the tree is evaluated inside a function, so these queries do not use an index. Use EqAt, Contains or At.<Path> for indexed queries.
The embedded EQL guest now names the eql_v4.query_json_predicate domain. Checker.Operators lists match with input predicate on every JsonSearch field, and rejected predicates are reported as malformed input rather than internal failures.
More precise predicate diagnostics (#1443)
The new Diagnostic.Construct and Diagnostic.At fields identify:
- The rejected construct, using the grammar's terms—for example,
like_regexor a position such as[2]. - The byte offset where the construct starts.
Neither field includes text from the predicate.
Match several members of the same array element (#1442)
A JsonSearch field with ordered paths beneath an array wildcard now gets an Elements.<Array>.Match accessor. For example, these paths produce Elements.Contacts:
$.contacts[*].relationship;$.contacts[*].calls
The accessor is named using the same path naming rules as At:
Fields.<Field>.Elements.<Array>.Match(ctx, cipher, opts, predicates...)It returns an eql.JSONSearchElementsQuery for:
eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)This finds rows where one array element satisfies every predicate. Separate EqAt and ordering queries at the array's paths can each be satisfied by different elements.
Building predicates
- Generated
Pathsmirrors the JSON structure and names each array's ordered members, such asPaths.Profile.Contacts.Relationship. encrypt.AtPath(path)providesEq,Lt,Lte,Gt,GteandBetweenpredicates.encrypt.Path[E, T]carries both the array and the member's Go type from the document struct. Passing the wrong value type or a predicate for another array does not compile. Members of ajson.RawMessageuseany.Pathscontains only fields, so no JSON key is reserved.- For a member named at runtime, use the array accessor's
Member("relationship"); it provides the same predicates withanyvalues.
Comparison semantics and limits
- Comparisons use the member's ordering term.
Eqtherefore compares text with case and accent folding, and integers above2^53can tie.EqAtremains exact. - A runtime member that the field does not order returns
encrypt.ErrTermbefore calling the engine. The error names the member and array. - No predicates, a zero
ElementPredicate, or a member containing a wildcard returnsencrypt.ErrEncoding. - An array whose members cannot be named in
Paths—because a key has no Go name or two paths share a name—gets no accessor and produces a notice. The same applies when two arrays would share an accessor name. - If a row's array exceeds
JSONReadOptions.WildcardBound(default:64), the query fails with SQLSTATE22023. - Printing or logging an
ElementPredicateshows its member and comparison, never its values.
The embedded EQL guest now names the eql_v4.query_json_elements domain. Checker.Operators lists element_match with input path_predicates on fields that order a path.
Fixed
Cipher.Seal rejects unknown keys in every row
A record.Source row that omitted a null=sql field but included an unknown key could have the expected key count. Previously, that unknown key was silently dropped.
It now returns an ErrEncoding diagnostic with unknown_field, naming the key, just as it already did when the key counts differed. Code that passed such a row now receives an error.
Unbounded Limit respects the largest SQL LIMIT
Limit.FetchOr previously returned any page size it received. A size of 2^63 or more reached database/sql as a uint64 that it could not accept. The returned size is now capped at math.MaxInt64.
The Stats.Rows documentation also no longer recommends greatest(reltuples, 0). A reltuples value of -1 means the table has never been analyzed and its row count is unknown—not zero. Use a reltuples >= 0 filter and continue paging while the count is unknown.
Rejected guest requests wipe bearer tokens
A request with no method or URL was previously rejected without wiping its headers, which carry the bearer token, or its body. Both are now wiped, as they are for other rejection paths.
Breaking change: EQL values print and log as redacted (#1443)
Every encrypt/eql type now implements String, Format and LogValue to show the type name and [redacted], never its bytes. This applies to every formatting verb, including %#v.
eql.TextEq{[redacted]}
eql.TextEq{} // Empty value
Stored values contain ciphertext and terms; query values contain terms. Previously, these types appeared to fmt and slog as bare []byte values:
%sand%qexposed the JSON.fmt.Sprint(v),%vand%xexposed the bytes as numbers or hex.
These calls now return redacted text without an error.
Updating callers
If you intentionally need to print or log the JSON, call the new Reveal() method at that call site. It returns the JSON as a string:
slog.Debug("stored", "email", v.Reveal())Code that reads the JSON can use any of these existing forms:
string(v)
json.RawMessage(v)
v.MarshalJSON()Value is unchanged. These methods are generated by eql-codegen for every type.
Install
Module: github.com/cipherstash/stack/languages/golang
Version: v0.1.0-rc.3
Built from: 929451b4988bae40760673fe333dda809d468028
go get github.com/cipherstash/stack/languages/golang@v0.1.0-rc.3
go install github.com/cipherstash/stack/languages/golang/cmd/stashgen@v0.1.0-rc.3Values written by this version are stored in EQL v4 columns. Install the attached eql-v4-install.sql into PostgreSQL:
psql -f eql-v4-install.sqlIf upgrading from rc.2, follow the permissions and index restoration steps above.
SHA-256 checksums of the embedded WASI guests
1bdbac977548e9df38040651c0abdfb00e23d1ca546a892c890170cc9e64ad30 languages/golang/encrypt/wasm/stack_encrypt_guest.wasm
328cc48392a03d037a33ee560fc6a6a25ff6eb45132f5773ba7a06518a2d90a2 languages/golang/encrypt/eql/wasm/stack_encrypt_guest_eql.wasm
f378be4d1a57fb829052c0eb3bd22cde0f596a1574c2e5e13c7f9801c331b514 languages/golang/auth/wasm/stack_auth_guest.wasm