Skip to content

Go v0.1.0-rc.3

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 02 Nov 12:03
· 181 commits to main since this release

Go v0.1.0-rc.3 adds JSONPath filters and queries that match several members of the same JSON array element. It also fixes validation, pagination and token cleanup, and changes how EQL values print and log.

Upgrading from rc.2

Install this release's EQL v4 SQL bundle before using the new JSON queries. The rc.2 bundle does not contain the required types and functions, so PostgreSQL will reject those queries with a missing-type error.

Download the attached eql-v4-install.sql, then run:

psql -f eql-v4-install.sql

The new queries require:

Go API SQL parameter type SQL function
Fields.<Field>.Match eql_v4.query_json_predicate eql_v4.json_tree_match
Fields.<Field>.Elements.<Array>.Match eql_v4.query_json_elements eql_v4.json_tree_element_match

Restore permissions and indexes after installation

Re-running the bundle preserves your columns and data because the column types live in public.

It drops and re-creates the eql_v4 and eql_v4_internal schemas with CASCADE. This removes indexes built on EQL functions, such as eql_v4.ord_term(col), along with schema and function grants and any ALTER DEFAULT PRIVILEGES in those schemas. The installer issues no GRANT statements.

After installation, before the application sends queries:

  1. Restore permissions if the application connects as a different role from the installer. Grant USAGE on both eql_v4 and eql_v4_internal, and EXECUTE on all functions in both schemas. Equality, ordering and JSON queries reach the internal schema too. Restore any default privileges you previously configured.

    Without these grants, application queries fail with errors such as permission denied for schema eql_v4. The permissions reference lists the statements; replace eql_v3 in its examples with eql_v4.

  2. Re-create the indexes, then run ANALYZE.

Added

Bind a whole JSONPath filter as one parameter (#1443)

Every JsonSearch field now has:

Fields.<Field>.Match(ctx, cipher, predicate, vars, opts)

It accepts the same inputs as plaintext jsonb_path_match(col, $1, $2): a JSONPath predicate and its variables. It returns an eql.JSONSearchPredicateQuery for either SQL form:

col @@ $1::eql_v4.query_json_predicate

eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)

Variables

  • Pass any value that encoding/json writes as an object, or a json.RawMessage containing an object.
  • Use an empty map when there are no variables.
  • A nil vars value—including a nil map, slice or pointer—means SQL NULL, as it does when pgx or database/sql binds it for plaintext queries. The operand is empty, binds as NULL and matches no rows.
  • A value that encodes as JSON null is rejected because it is not an object, matching plaintext behavior.

Supported grammar

Construct Supported forms
Paths .key, ."quoted key", [*]
Comparisons against a variable or JSON literal ==, !=, <, <=, >, >=
Existence exists(path)
Filtered existence exists(path ? (@.a == $x && @.b > $y))
Boolean expressions &&, ||, !(…)

The engine parses the predicate and compiles it into a tree of the operands derived by EqAt, Selector, At.<Path> and Elements.<Array>.Match. The Go side does not parse the predicate.

Errors and query behavior

  • Unsupported constructs return an *encrypt.Diagnostic wrapping encrypt.ErrEncoding. The compiler never silently drops a leaf from the tree.
  • Ordering at a path that the field's tag does not order returns encrypt.ErrTerm.
  • Results follow plaintext jsonb_path_match, with five documented exceptions: lax array unwrapping, lax array wrapping, comparisons between strings and numbers, folded ordering terms, and ordering comparisons involving null. In the last case, the result is SQL NULL rather than false outside an odd number of ! operators, which returns the same rows.

Index use: the tree is evaluated inside a function, so these queries do not use an index. Use EqAt, Contains or At.<Path> for indexed queries.

The embedded EQL guest now names the eql_v4.query_json_predicate domain. Checker.Operators lists match with input predicate on every JsonSearch field, and rejected predicates are reported as malformed input rather than internal failures.

More precise predicate diagnostics (#1443)

The new Diagnostic.Construct and Diagnostic.At fields identify:

  • The rejected construct, using the grammar's terms—for example, like_regex or a position such as [2].
  • The byte offset where the construct starts.

Neither field includes text from the predicate.

Match several members of the same array element (#1442)

A JsonSearch field with ordered paths beneath an array wildcard now gets an Elements.<Array>.Match accessor. For example, these paths produce Elements.Contacts:

$.contacts[*].relationship;$.contacts[*].calls

The accessor is named using the same path naming rules as At:

Fields.<Field>.Elements.<Array>.Match(ctx, cipher, opts, predicates...)

It returns an eql.JSONSearchElementsQuery for:

eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)

This finds rows where one array element satisfies every predicate. Separate EqAt and ordering queries at the array's paths can each be satisfied by different elements.

Building predicates

  • Generated Paths mirrors the JSON structure and names each array's ordered members, such as Paths.Profile.Contacts.Relationship.
  • encrypt.AtPath(path) provides Eq, Lt, Lte, Gt, Gte and Between predicates.
  • encrypt.Path[E, T] carries both the array and the member's Go type from the document struct. Passing the wrong value type or a predicate for another array does not compile. Members of a json.RawMessage use any.
  • Paths contains only fields, so no JSON key is reserved.
  • For a member named at runtime, use the array accessor's Member("relationship"); it provides the same predicates with any values.

Comparison semantics and limits

  • Comparisons use the member's ordering term. Eq therefore compares text with case and accent folding, and integers above 2^53 can tie. EqAt remains exact.
  • A runtime member that the field does not order returns encrypt.ErrTerm before calling the engine. The error names the member and array.
  • No predicates, a zero ElementPredicate, or a member containing a wildcard returns encrypt.ErrEncoding.
  • An array whose members cannot be named in Paths—because a key has no Go name or two paths share a name—gets no accessor and produces a notice. The same applies when two arrays would share an accessor name.
  • If a row's array exceeds JSONReadOptions.WildcardBound (default: 64), the query fails with SQLSTATE 22023.
  • Printing or logging an ElementPredicate shows its member and comparison, never its values.

The embedded EQL guest now names the eql_v4.query_json_elements domain. Checker.Operators lists element_match with input path_predicates on fields that order a path.

Fixed

Cipher.Seal rejects unknown keys in every row

A record.Source row that omitted a null=sql field but included an unknown key could have the expected key count. Previously, that unknown key was silently dropped.

It now returns an ErrEncoding diagnostic with unknown_field, naming the key, just as it already did when the key counts differed. Code that passed such a row now receives an error.

Unbounded Limit respects the largest SQL LIMIT

Limit.FetchOr previously returned any page size it received. A size of 2^63 or more reached database/sql as a uint64 that it could not accept. The returned size is now capped at math.MaxInt64.

The Stats.Rows documentation also no longer recommends greatest(reltuples, 0). A reltuples value of -1 means the table has never been analyzed and its row count is unknown—not zero. Use a reltuples >= 0 filter and continue paging while the count is unknown.

Rejected guest requests wipe bearer tokens

A request with no method or URL was previously rejected without wiping its headers, which carry the bearer token, or its body. Both are now wiped, as they are for other rejection paths.

Breaking change: EQL values print and log as redacted (#1443)

Every encrypt/eql type now implements String, Format and LogValue to show the type name and [redacted], never its bytes. This applies to every formatting verb, including %#v.

eql.TextEq{[redacted]}
eql.TextEq{}             // Empty value

Stored values contain ciphertext and terms; query values contain terms. Previously, these types appeared to fmt and slog as bare []byte values:

  • %s and %q exposed the JSON.
  • fmt.Sprint(v), %v and %x exposed the bytes as numbers or hex.

These calls now return redacted text without an error.

Updating callers

If you intentionally need to print or log the JSON, call the new Reveal() method at that call site. It returns the JSON as a string:

slog.Debug("stored", "email", v.Reveal())

Code that reads the JSON can use any of these existing forms:

string(v)
json.RawMessage(v)
v.MarshalJSON()

Value is unchanged. These methods are generated by eql-codegen for every type.

Install

Module: github.com/cipherstash/stack/languages/golang
Version: v0.1.0-rc.3
Built from: 929451b4988bae40760673fe333dda809d468028

go get github.com/cipherstash/stack/languages/golang@v0.1.0-rc.3
go install github.com/cipherstash/stack/languages/golang/cmd/stashgen@v0.1.0-rc.3

Values written by this version are stored in EQL v4 columns. Install the attached eql-v4-install.sql into PostgreSQL:

psql -f eql-v4-install.sql

If upgrading from rc.2, follow the permissions and index restoration steps above.

SHA-256 checksums of the embedded WASI guests
1bdbac977548e9df38040651c0abdfb00e23d1ca546a892c890170cc9e64ad30  languages/golang/encrypt/wasm/stack_encrypt_guest.wasm
328cc48392a03d037a33ee560fc6a6a25ff6eb45132f5773ba7a06518a2d90a2  languages/golang/encrypt/eql/wasm/stack_encrypt_guest_eql.wasm
f378be4d1a57fb829052c0eb3bd22cde0f596a1574c2e5e13c7f9801c331b514  languages/golang/auth/wasm/stack_auth_guest.wasm