v2.4.6

HexDB 2.4.6 is a security and reliability release. It fixes the 14 findings of an independent code review of 2.4.5, plus two problems found while checking them. Upgrade soon if you use row filters, hidden or masked fields, tokenization, min_acks or file quotas.

Security

• Detokenizing follows the caller's access. POST /tessellations/{name}/detokenize checked the permission and the field, but not the caller's row filter or a hidden field around the vault field. A caller with detokenize could open a token copied from a row they can't read. Now a token opens only if the caller may read its field (that field and every field around it, unmasked) and, with a row filter, only if a document they can read holds it at that field. Any other token gives null.
• Protections reach inside arrays. hide, mask, sealed fields and tokenize on a path such as items.cost didn't apply to the elements of an items array. They now apply to every element.
• Combined roles never show what both hide. With two restricted roles, a field one role hid as part of a parent (card) and the other hid by itself (card.number) came through in the clear. A field is now visible only if some role shows it and every field around it.
• Large integers compare exactly. Numbers were compared as doubles, so integers beyond 2^53 that differ (9007199254740993 and 9007199254740992) compared equal. That affected filters, sorting, indexes and row filters, which could let a row filter match another tenant's ID. Numbers now compare exactly everywhere. Field indexes are rebuilt once at startup (new snapshot format).
• Idempotent replays use today's access. A retry with the same Idempotency-Key returned the stored answer even if the caller had since lost access to the document, or fields had become hidden or masked. A replay is now refused with 403 when the caller can't read the document any more, and hidden and masked fields follow their current roles.

Fixed

• Writes to vault-tokenized fields work under row filters. The vault entry a write adds was counted as one of the caller's documents: a one-document bulk insert returned two IDs, and an insert by a role with a row filter (a tenant filter, say) was refused because the vault entry has no tenant. The caller's documents no longer include vault entries, which still commit atomically with them.
• Retries wait for replicas. With replication.min_acks, a retry of a write that had timed out waiting for replicas answered success at once. It now waits for the replicas like the original write.
• The change feed can't get stuck. A write whose request was cancelled while its commit was queued could leave the change feed waiting forever, which stopped replication. A commit that's accepted now reaches the feed whether or not its request is still there.
• Failover picks the replica with the newest writes. In a lattice without slicing, an election ranked replicas by preference and hardware only, so a replica missing recent writes could take over and lose them. The data now decides first: only the hexes furthest along the history most of them follow can take over, then preference, slices and hardware choose among them.
• Replication never splits a transaction. A page of changes could end in the middle of a transaction, so a replica (or a hex taking over at that moment) could expose half of it. Pages now end only between commits.
• A busy peer list doesn't count as a quorum. With replication.quorum, a check that found the peer list locked let the write through. It now uses the last count instead.
• Documents with very many index keys stay findable. A document with more than 1,000 keys in one index (a long array, or arrays multiplied in a composite index) was indexed by its first 1,000 only, so lookups could miss it. Such a document is now checked on every lookup instead, and a unique index refuses it.
• Restored documents keep their vault values. Vault cleanup deleted the values of tokens held only by trashed documents, so a restored document came back with tokens that open nothing. Vault entries now stay until the last document that could come back with them is purged or expires from the trash. A restore whose vault values are gone anyway is refused with 409.
• Key rotation keeps tokens. The tokenization key was derived from the current lattice key, so rotating storage.encryption_key or network.lattice_secret changed every new token and broke lookups by value on existing ones. The key is now fixed when the data is created, kept in the catalog and in backups, and shared with every hex that joins.
• Concurrent uploads respect quotas. Uploads committing at the same time each saw the same room left, so together they could go far past a user's or tessellation's quota. Each upload now reserves its size from the quota check until its commit.
• Hexes started together settle on one Overseer: a hex that started later could keep its own claim while an earlier one never learned of it. And two first deletes at the same moment no longer fail by creating the trash's internal indexes twice.

Upgrading

Restart the hexes one at a time, as usual: replicas first, the Overseer last. Each hex rebuilds its field indexes once on its first start (the index snapshot format changed). An existing lattice keeps the tokenization key it uses today, so tokens written since the last key rotation keep matching. Coming from 2.4.3 or earlier, read the 2.4.4 notes first: its storage format can't be read by earlier versions.

Full Changelog: v2.4.5...v2.4.6