Repository navigation
v0.1.7
Added — opt-in lossy decoding for q text that is not valid UTF-8
q stores symbols and strings as raw bytes and never validates them, while Arrow
string columns and Rust strings must be valid UTF-8. Until now a single stray
Latin-1 or binary byte anywhere in a response failed the whole result:
"q symbol-list value is not valid UTF-8: invalid utf-8 sequence of 1 bytes from index 3"
A connection can now opt in to transcoding instead. The policy is
symbolEncoding on Node (QOptions and readBinary6(path, { symbolEncoding }))
and symbol_encoding on Python (Q(...), the Q.symbol_encoding property, and
read_binary6(path, symbol_encoding=...)), accepting "strict" or "lossy":
"strict"(default) keeps the existing behaviour: the first invalid byte
sequence fails the response with an error naming the offending value."lossy"replaces each maximal invalid subsequence with U+FFFD REPLACEMENT
CHARACTER — the same substitutionString::from_utf8_lossyperforms — so
every other value in the response survives intact. A Latin-1caf\xe9decodes
ascaf\ufffd.
The policy applies to every place wire text becomes a string: symbol atoms,
symbol lists and columns, nested symbol lists, char columns, string columns,
lambda source and context, dictionary keys, and table column names. String
column rows are transcoded individually, so a multi-byte code point split across
two rows is replaced in both rather than silently validated as a whole. Valid
text decodes identically under both policies.
Any other value ("latin1", "Lossy", …) is rejected at construction: a
RangeError on Node, a ValueError on Python, and XQDB_CONVERSION from the
native readBinary6.
Changed
- q error messages are now always decoded lossily. A server error whose text
contained a non-UTF-8 byte used to surface as a deserialization failure that
hid the actual error; it now surfaces as the server error with replacement
characters, under either policy. - Symbol lists are validated once instead of twice: the pass that measures a
list's wire length no longer decodes each symbol, leaving that to the pass that
materializes it. Strict-mode errors for invalid symbols are unchanged in type
and still name the offending value.
Every buffer the lossy path allocates is sized from wire data and reserved
fallibly, in line with 0.1.5 and 0.1.6.
Note on 0.1.6
The 0.1.6 npm release built and validated successfully but was never approved
for publication, so @xbbg/xqdb@0.1.6 does not exist on npm; that run has been
cancelled in favour of this release. 0.1.7 contains everything 0.1.6 did.
Python users upgrading from 0.1.3
This is the first PyPI release since 0.1.3, so it also delivers the 0.1.4
through 0.1.6 changes. One of them is breaking:
q timestamp atoms are naive. 0.1.3 returned UTC-aware datetime values;
0.1.4 and later return naive values with tzinfo is None, matching Arrow
timestamp[ns] columns, so the same q value compares equal whether read as an
atom or out of a column and can be passed back as a query argument. If you need
an aware value, attach the zone yourself — and only when the data really is UTC,
because a process that stores .z.P holds local wall-clock times:
ts = conn.sync("first exec time from trade") # naive datetime
ts = ts.replace(tzinfo=timezone.utc)Query arguments accept both shapes: a naive datetime keeps its wall clock and
an aware one is normalized through its UTC offset.
The rest is additive: no fixed size ceiling on results, every wire-sized
allocation reports an error instead of aborting the interpreter, 40-bit framing
for messages above 4 GiB, and the dependency advisories patched in 0.1.4. See
docs/release-notes/v0.1.4.md through v0.1.6.md for the details.