diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index 4b10d29..0a83305 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -6,7 +6,7 @@ **Feature parity and compliance status across all CacheKit SDK implementations.** -*Last updated: 2026-07-28 — LAB-998: interop/v1 ship-status corrected — the row no longer reads `unreleased`; all three SDKs have published it (PyPI 0.14.0+, crates.io 0.4.0+, npm 0.1.3+), stated as floors per footnote ⁴, aligned with docs.cachekit.io (LAB-996). LAB-729: rs backpressure flipped ❌ → ✅ (semaphore + bounded queue in the rs reliability stack; decision footnote records why the LAB-519 ts rationale doesn't transfer to tokio). LAB-430 shipped TypeScript Node-only Memcached and File backends; the protocol-owned File format and vectors now define fail-closed flag negotiation. LAB-446: Python File backend gains full TTL inspection/refresh; Memcached gains `refresh_ttl` (touch) only (see [TTL management note](#reliability-features)). LAB-595 shipped: ts Cloudflare Workers flipped ❌ → ✅ via the `@cachekit-io/cachekit/workers` entrypoint on a wasm32 cachekit-core build (~55 KB gz measured); footnote ¹ records the phase-1 surface and semantics deltas. LAB-519: ts cold-miss single-flight (in-process, always on) + LockableBackend wired into `wrap()`'s miss path (opt-in); ts backpressure decision recorded; ts Redis lock/TTL capability cells refreshed for LAB-427. LAB-272 code-verified protocol-adherence audit (2026-07-22): interop/v1 merged in Python ([cachekit-py#220](https://github.com/cachekit-io/cachekit-py/pull/220)), TypeScript ([cachekit-ts#71](https://github.com/cachekit-io/cachekit-ts/pull/71)), and Rust ([cachekit-rs#33](https://github.com/cachekit-io/cachekit-rs/pull/33)); test-vector CI coverage corrected* +*Last updated: 2026-07-29 — LAB-903: SDK Overview versions refreshed against the registries after the protocol 1.1 (envelope `bin` encoding, cachekit-core 0.4.0) rollout — py 0.17.0, core 0.4.0, rs 0.5.0 + macros 0.5.0, ts 0.1.4; per-SDK rollout state (which releases carry the flip, which bumps sit merged-unreleased on main) is recorded once, in the [cachekit-core architecture note](#architecture-notes). LAB-998: interop/v1 ship-status corrected — the row no longer reads `unreleased`; all three SDKs have published it (PyPI 0.14.0+, crates.io 0.4.0+, npm 0.1.3+), stated as floors per footnote ⁴, aligned with docs.cachekit.io (LAB-996). LAB-729: rs backpressure flipped ❌ → ✅ (semaphore + bounded queue in the rs reliability stack; decision footnote records why the LAB-519 ts rationale doesn't transfer to tokio). LAB-430 shipped TypeScript Node-only Memcached and File backends; the protocol-owned File format and vectors now define fail-closed flag negotiation. LAB-446: Python File backend gains full TTL inspection/refresh; Memcached gains `refresh_ttl` (touch) only (see [TTL management note](#reliability-features)). LAB-595 shipped: ts Cloudflare Workers flipped ❌ → ✅ via the `@cachekit-io/cachekit/workers` entrypoint on a wasm32 cachekit-core build (~55 KB gz measured); footnote ¹ records the phase-1 surface and semantics deltas. LAB-519: ts cold-miss single-flight (in-process, always on) + LockableBackend wired into `wrap()`'s miss path (opt-in); ts backpressure decision recorded; ts Redis lock/TTL capability cells refreshed for LAB-427. LAB-272 code-verified protocol-adherence audit (2026-07-22): interop/v1 merged in Python ([cachekit-py#220](https://github.com/cachekit-io/cachekit-py/pull/220)), TypeScript ([cachekit-ts#71](https://github.com/cachekit-io/cachekit-ts/pull/71)), and Rust ([cachekit-rs#33](https://github.com/cachekit-io/cachekit-rs/pull/33)); test-vector CI coverage corrected* @@ -30,10 +30,10 @@ | SDK | Package | Version | Language | Status | | :--- | :--- | :---: | :--- | :---: | -| cachekit-py | `cachekit` (PyPI) | 0.12.0 | Python 3.10+ | ✅ Production | -| cachekit-rs | `cachekit-rs` (crates.io) | 0.3.0 | Rust 1.82+ | ✅ Production | -| cachekit-core | `cachekit-core` (crates.io) | 0.3.0 | Rust (shared core) | ✅ Production | -| cachekit-ts | `@cachekit-io/cachekit` (npm) | 0.1.2 | TypeScript | ✅ Production | +| cachekit-py | `cachekit` (PyPI) | 0.17.0 | Python 3.10+ | ✅ Production | +| cachekit-rs | `cachekit-rs` (crates.io) | 0.5.0 | Rust 1.82+ | ✅ Production | +| cachekit-core | `cachekit-core` (crates.io) | 0.4.0 | Rust (shared core) | ✅ Production | +| cachekit-ts | `@cachekit-io/cachekit` (npm) | 0.1.4 | TypeScript | ✅ Production | | cachekit-php | — | — | PHP 8.1+ | 🔜 Development | --- @@ -224,7 +224,7 @@ its spec:
Rust SDK (cachekit-rs) -- Published on crates.io as `cachekit-rs` v0.3.0 + `cachekit-macros` v0.3.0 +- Published on crates.io as `cachekit-rs` v0.5.0 + `cachekit-macros` v0.5.0 - Feature flags: `redis`, `cachekitio`, `encryption`, `l1`, `macros`, `workers` - Backends: `RedisBackend` (fred), `CachekitIO` (reqwest), `WorkersCachekitIO` (CF Workers fetch) - L1 cache via moka (native only, `l1` feature) @@ -232,14 +232,14 @@ its spec: - `SecureCache` for zero-knowledge encrypted caching - SSRF protection, credential redaction, `Zeroizing` key material - WASM/Workers support: `?Send` + `Rc` paths via `cfg(target_arch = "wasm32")` -- Depends on `cachekit-core` v0.2.0 for ByteStorage and encryption primitives +- Depends on `cachekit-core` for encryption primitives only (the envelope is unused for stored values — see [Compliance Status](#compliance-status) note ²); core-version rollout state is recorded in the cachekit-core note below
Rust Core (cachekit-core) -- Published on crates.io as `cachekit-core` v0.3.0 (`cachekit-rs` still depends on the 0.2 line — Renovate bump tracked separately) +- Published on crates.io as `cachekit-core` v0.4.0 — the protocol 1.1 writer flip: `StorageEnvelope.compressed_data` now *emits* msgpack `bin` (`serde_bytes`); readers dual-decode both `bin` and the legacy array-of-ints ([spec/wire-format.md](spec/wire-format.md), [decisions/envelope-bin-encoding.md](decisions/envelope-bin-encoding.md)). Consumers: cachekit-py ≥ 0.17.0 ships it ([cachekit-py#249](https://github.com/cachekit-io/cachekit-py/pull/249)); released cachekit-rs 0.5.0 and cachekit-ts 0.1.4 still pin the 0.3 line — their 0.4.0 bumps ([cachekit-rs#53](https://github.com/cachekit-io/cachekit-rs/pull/53), [cachekit-ts#91](https://github.com/cachekit-io/cachekit-ts/pull/91)) are merged on main, unreleased - Provides: `ByteStorage`, `ZeroKnowledgeEncryptor`, `derive_domain_key`, `derive_tenant_keys` - Dependencies: `lz4_flex`, `xxhash-rust`, `ring` (native) / `aes-gcm` (wasm32), `hkdf`, `sha2`, `rmp-serde` - Formally verified security properties via Kani diff --git a/spec/wire-format.md b/spec/wire-format.md index 11aa127..8dfdbc2 100644 --- a/spec/wire-format.md +++ b/spec/wire-format.md @@ -6,7 +6,7 @@ **LZ4 compression + xxHash3-64 integrity wrapping for cached payloads that use the envelope.** -*Protocol Version 1.1 · Verified against `cachekit-core` v0.3.0 (`src/byte_storage.rs`); legacy envelope test vectors generated at v0.2.0 and unchanged since — `bin`-encoded twins added in protocol 1.1 ([decisions/envelope-bin-encoding.md](../decisions/envelope-bin-encoding.md))* +*Protocol Version 1.1 · Verified against `cachekit-core` v0.4.0 (`src/byte_storage.rs`); legacy envelope test vectors generated at v0.2.0 and unchanged since — `bin`-encoded twins added in protocol 1.1 ([decisions/envelope-bin-encoding.md](../decisions/envelope-bin-encoding.md))* @@ -85,10 +85,12 @@ accept the legacy encoding below — a stored envelope never expires on a schedu so legacy-read support is permanent. > [!NOTE] -> **Implementation status:** no shipped release emits the canonical `bin` -> encoding yet — `cachekit-core` v0.3.0 (current) still writes the legacy -> encoding. The writer flip is tracked in LAB-764 / -> [cachekit-core#54](https://github.com/cachekit-io/cachekit-core/issues/54). +> **Implementation status:** the canonical `bin` encoding **is** shipped — +> `cachekit-core` v0.4.0 carries the writer flip, and `cachekit` (Python) +> ≥ 0.17.0 emits it. The released `cachekit-rs` 0.5.0 and `cachekit-ts` 0.1.4 +> lines still pin core 0.3 and therefore still write the legacy encoding, so +> readers encounter both on the wire today. Per-SDK rollout state is tracked in +> [sdk-feature-matrix.md](../sdk-feature-matrix.md#architecture-notes). `checksum` (element `[1]`) is **deliberately excluded** from the `bin` encoding: it stays an array of 8 integers. The saving would be 1–7 bytes per envelope, and the @@ -481,8 +483,10 @@ interop value (consume exactly one document, reject trailing bytes, the [`test-vectors/python-frame.json`](../test-vectors/python-frame.json) pins the CK v3 frame against the real `cachekit-py` implementation: a minimal frame, a complete default-path write (frame → ByteStorage envelope → inner MessagePack → value, full -round-trip), an Arrow-envelope frame (structural checks), and must-reject error -vectors — including a CK frame fed to a strict interop reader. +round-trip) in both envelope encodings — the legacy array-of-ints original +(cachekit 0.11.1) and its protocol 1.1 `bin` twin (cachekit 0.17.0, the first +release emitting `bin`) — an Arrow-envelope frame (structural checks), and +must-reject error vectors — including a CK frame fed to a strict interop reader. Verify: diff --git a/test-vectors/python-frame.json b/test-vectors/python-frame.json index e3bed75..efbd433 100644 --- a/test-vectors/python-frame.json +++ b/test-vectors/python-frame.json @@ -1,7 +1,7 @@ { "description": "Python SDK (cachekit-py) auto-mode storage container: CK v3 frame. Python-SDK-internal \u2014 other SDKs identify and reject, never decode. See spec/wire-format.md 'SDK Storage Containers (auto mode)'.", "frame_layout": "MAGIC 'CK' (0x43 0x4B) | VERSION u8 (0x03) | HDR_LEN u32 big-endian | HEADER (UTF-8 JSON: {s, m, v}) | PAYLOAD (raw bytes)", - "generator": "cachekit 0.11.1 (PyPI wheel; Rust core via PyO3), generated by tools/python-frame-reference.py generate", + "generator": "legacy vectors: cachekit 0.11.1 (PyPI wheel; Rust core via PyO3), generated by tools/python-frame-reference.py generate (unchanged since); *_bin twins carry their own per-vector 'generator' field", "frame_vectors": [ { "name": "raw_payload_frame", @@ -39,6 +39,7 @@ "expected_payload_hex": "94dc0022ccf011cc83cca7757365725f69642acca46e616d65cca863616368656b6974cca6616374697665ccc398ccdaccf46861cc8e624bccfe20a76d73677061636b", "payload_envelope": { "encoding": "rmp_serde::to_vec positional fixarray(4); Vec/[u8;8] fields encode as msgpack arrays of integers", + "envelope_encoding": "int-array", "compressed_data_hex": "f01183a7757365725f69642aa46e616d65a863616368656b6974a6616374697665c3", "checksum_hex": "daf468618e624bfe", "original_size": 32, @@ -66,6 +67,37 @@ "ipc_magic_offset": 8, "ipc_magic": "ARROW1" } + }, + { + "name": "default_saas_write_msgpack_bytestorage_bin", + "description": "Protocol 1.1 twin of default_saas_write_msgpack_bytestorage: same value, same default @cache write path, but the ByteStorage envelope's compressed_data is msgpack bin (serde_bytes) instead of an array of integers. Readers MUST accept both encodings; the legacy encoding stays pinned by the legacy vector's bytes.", + "value_json": { + "user_id": 42, + "name": "cachekit", + "active": true + }, + "frame_hex": "434b030000007d7b2273223a202264656661756c74222c20226d223a207b22666f726d6174223a20226d73677061636b222c2022656e636f64696e67223a20227574662d38222c2022636f6d70726573736564223a20747275652c20226f726967696e616c5f74797065223a20226d73677061636b227d2c202276223a2022322e30227d94c422f01183a7757365725f69642aa46e616d65a863616368656b6974a6616374697665c398ccdaccf46861cc8e624bccfe20a76d73677061636b", + "expected_header": { + "s": "default", + "m": { + "format": "msgpack", + "encoding": "utf-8", + "compressed": true, + "original_type": "msgpack" + }, + "v": "2.0" + }, + "expected_payload_hex": "94c422f01183a7757365725f69642aa46e616d65a863616368656b6974a6616374697665c398ccdaccf46861cc8e624bccfe20a76d73677061636b", + "payload_envelope": { + "encoding": "rmp_serde positional fixarray(4); compressed_data encodes as msgpack bin (serde_bytes, protocol 1.1); checksum [u8;8] stays an array of integers", + "envelope_encoding": "bin", + "compressed_data_hex": "f01183a7757365725f69642aa46e616d65a863616368656b6974a6616374697665c3", + "checksum_hex": "daf468618e624bfe", + "original_size": 32, + "format": "msgpack", + "inner_msgpack_hex": "83a7757365725f69642aa46e616d65a863616368656b6974a6616374697665c3" + }, + "generator": "cachekit 0.17.0 (PyPI wheel; Rust core via PyO3), generated by tools/python-frame-reference.py generate-bin-twin" } ], "error_vectors": [ diff --git a/tools/frame-crosscheck.mjs b/tools/frame-crosscheck.mjs index 5944489..b080894 100644 --- a/tools/frame-crosscheck.mjs +++ b/tools/frame-crosscheck.mjs @@ -28,13 +28,17 @@ const hexToBytes = (hex) => { for (let i = 0; i < out.length; i++) out[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16); return out; }; -// Envelope byte fields are msgpack arrays of integers — reject anything a -// Uint8Array would silently coerce (fractional, negative, >255, non-numeric). -const intArrayToBytes = (arr, what) => { - if (!Array.isArray(arr) || arr.some((x) => !Number.isInteger(x) || x < 0 || x > 255)) { - throw new Error(`${what} is not an array of integers in 0..255`); +// Envelope byte fields arrive as msgpack arrays of integers (legacy) or as +// msgpack bin (protocol 1.1) — accept both, like every real reader. For the +// array form, reject anything a Uint8Array would silently coerce (fractional, +// negative, >255, non-numeric). Returns the bytes plus the observed encoding +// so vectors can pin which one they carry. +const fieldToBytes = (field, what) => { + if (field instanceof Uint8Array) return { bytes: field, encoding: "bin" }; + if (!Array.isArray(field) || field.some((x) => !Number.isInteger(x) || x < 0 || x > 255)) { + throw new Error(`${what} is not msgpack bin or an array of integers in 0..255`); } - return Uint8Array.from(arr); + return { bytes: Uint8Array.from(field), encoding: "int-array" }; }; const bytesToHex = (b) => Array.from(b, (x) => x.toString(16).padStart(2, "0")).join(""); @@ -54,10 +58,11 @@ function parseFrame(frame) { } // ------------------------------------------------- minimal MessagePack decode -// Covers common MessagePack types. `bin` (0xc4/0xc5/0xc6) is DELIBERATELY -// unsupported: the corrected spec pins ByteStorage byte fields as arrays of -// integers, and accepting bin here would silently mask a drift back to -// bin-encoded fields. decodeOne returns [value, nextOffset]; decodeDocument +// Covers common MessagePack types, including `bin` (0xc4/0xc5/0xc6): under +// protocol 1.1 ByteStorage byte fields are dual-encoded (legacy arrays of +// integers AND msgpack bin), so a reader must accept both. The legacy +// encoding stays pinned by the legacy vectors' byte-identity, not by this +// reader rejecting bin. decodeOne returns [value, nextOffset]; decodeDocument // additionally enforces single-document strictness (no trailing bytes) — the // property that makes CK frames fail loudly in interop readers. @@ -73,6 +78,12 @@ function decodeOne(b, at) { case 0xc0: return [null, at + 1]; case 0xc2: return [false, at + 1]; case 0xc3: return [true, at + 1]; + case 0xc4: return decodeBin(b, at + 2, u8len(b, at + 1, "bin8")); + // LAB-868: no committed vector exceeds bin8, so these two widths are + // implemented but unexercised — the coverage floor below proves `bin`, not + // every width tier. The width-boundary vector is tracked separately. + case 0xc5: return decodeBin(b, at + 3, dv.getUint16(at + 1)); + case 0xc6: return decodeBin(b, at + 5, dv.getUint32(at + 1)); case 0xcb: return [dv.getFloat64(at + 1), at + 9]; case 0xcc: return [b[at + 1], at + 2]; case 0xcd: return [dv.getUint16(at + 1), at + 3]; @@ -80,7 +91,7 @@ function decodeOne(b, at) { case 0xd0: return [dv.getInt8(at + 1), at + 2]; case 0xd1: return [dv.getInt16(at + 1), at + 3]; case 0xd2: return [dv.getInt32(at + 1), at + 5]; - case 0xd9: return decodeStr(b, at + 2, b[at + 1]); + case 0xd9: return decodeStr(b, at + 2, u8len(b, at + 1, "str8")); case 0xda: return decodeStr(b, at + 3, dv.getUint16(at + 1)); case 0xdc: return decodeArray(b, at + 3, dv.getUint16(at + 1)); case 0xdd: return decodeArray(b, at + 5, dv.getUint32(at + 1)); @@ -88,10 +99,25 @@ function decodeOne(b, at) { default: throw new Error(`msgpack type 0x${t.toString(16)} not supported by this cross-check`); } } +// Read a 1-byte declared length, validating the prefix byte EXISTS first. +// Without this, `b[at]` on a truncated buffer is `undefined`, every downstream +// `at + len > b.length` bound check evaluates `NaN > n` === false, and the +// overrun guard is silently bypassed — the reader returns an empty field and a +// NaN offset instead of failing. The 0xc5/0xc6 widths get this for free from +// DataView (RangeError). Protocol 1.1 security rule: validate declared length +// headers before using them (decisions/envelope-bin-encoding.md). +function u8len(b, at, what) { + if (at >= b.length) throw new Error(`msgpack ${what} length prefix overruns buffer`); + return b[at]; +} function decodeStr(b, at, len) { if (at + len > b.length) throw new Error("msgpack str overruns buffer"); return [new TextDecoder("utf-8", { fatal: true }).decode(b.subarray(at, at + len)), at + len]; } +function decodeBin(b, at, len) { + if (at + len > b.length) throw new Error("msgpack bin overruns buffer"); + return [b.subarray(at, at + len), at + len]; +} function decodeArray(b, at, count) { const out = []; for (let i = 0; i < count; i++) { @@ -169,6 +195,8 @@ const deepEqual = (a, b) => JSON.stringify(a) === JSON.stringify(b); const doc = JSON.parse(readFileSync(vectorPath, "utf-8")); +const observedEncodings = new Set(); + for (const vec of doc.frame_vectors) { const frame = hexToBytes(vec.frame_hex); let parsed; @@ -194,18 +222,34 @@ for (const vec of doc.frame_vectors) { const envelope = decodeDocument(parsed.payload); if (!Array.isArray(envelope) || envelope.length !== 4) throw new Error("envelope is not a 4-element msgpack array"); const [compressedData, checksum, originalSize, format] = envelope; - const compressedBytes = intArrayToBytes(compressedData, "compressed_data"); - if (bytesToHex(compressedBytes) !== env.compressed_data_hex) throw new Error("compressed_data mismatch"); - const checksumBytes = intArrayToBytes(checksum, "checksum"); + const compressed = fieldToBytes(compressedData, "compressed_data"); + // Mandatory, not opt-in: compressed_data_hex/checksum_hex/inner_msgpack_hex/ + // original_size/format are byte-identical between a legacy vector and its bin + // twin, so envelope_encoding is the ONLY assertion that catches an encoding + // drift. A vector allowed to omit it pins nothing about the encoding. + if (!env.envelope_encoding) throw new Error("payload_envelope must declare envelope_encoding ('bin' or 'int-array')"); + if (compressed.encoding !== env.envelope_encoding) { + throw new Error(`compressed_data is ${compressed.encoding}, vector declares ${env.envelope_encoding}`); + } + observedEncodings.add(compressed.encoding); + if (bytesToHex(compressed.bytes) !== env.compressed_data_hex) throw new Error("compressed_data mismatch"); + const checksumField = fieldToBytes(checksum, "checksum"); + // Protocol 1.1 scopes the bin flip to compressed_data ONLY — checksum + // [u8;8] stays an array of integers in both encodings. + if (checksumField.encoding !== "int-array") throw new Error("checksum must be an array of integers (excluded from the protocol 1.1 bin flip)"); + const checksumBytes = checksumField.bytes; if (checksumBytes.length !== 8) throw new Error(`checksum is ${checksumBytes.length} bytes (envelope requires exactly 8)`); if (bytesToHex(checksumBytes) !== env.checksum_hex) throw new Error("checksum field mismatch"); if (originalSize !== env.original_size) throw new Error("original_size mismatch"); + // `format` is excluded from the bin flip alongside `checksum`, and it feeds + // AAD construction — a drift to bin must be named, not surface as a type pun. + if (typeof format !== "string") throw new Error("format must be a msgpack str (excluded from the protocol 1.1 bin flip)"); if (format !== env.format) throw new Error("format mismatch"); - const inner = lz4BlockDecompress(compressedBytes, originalSize); + const inner = lz4BlockDecompress(compressed.bytes, originalSize); if (bytesToHex(inner) !== env.inner_msgpack_hex) throw new Error("decompressed payload mismatch"); const value = decodeDocument(inner); if (!deepEqual(value, vec.value_json)) throw new Error("decoded value != value_json"); - ok(vec.name, "full round-trip: frame -> envelope -> LZ4 -> msgpack -> value"); + ok(vec.name, `full round-trip: frame -> envelope (${compressed.encoding}) -> LZ4 -> msgpack -> value`); } catch (e) { fail(vec.name, e.message); } @@ -251,6 +295,17 @@ for (const vec of doc.error_vectors) { } } +// Coverage floor. Protocol 1.1 is "writers emit bin, readers accept legacy +// FOREVER" — a dual-read guarantee is only proven while the fixture carries a +// vector in each encoding. Without this, deleting the bin twin (or flipping it +// to int-array) exits 0 and CI reports the dual-encoding gate as green while +// proving one encoding, or neither. +for (const want of ["int-array", "bin"]) { + if (!observedEncodings.has(want)) { + fail("envelope-encoding coverage", `no frame vector exercises the ${want} envelope encoding`); + } +} + if (failures) { console.log(`\n${failures} failure(s)`); process.exit(1); diff --git a/tools/python-frame-reference.py b/tools/python-frame-reference.py index daf2810..a9ec739 100644 --- a/tools/python-frame-reference.py +++ b/tools/python-frame-reference.py @@ -18,6 +18,19 @@ the Arrow vector additionally needs `pyarrow` + `pandas`. Every generated frame is round-tripped through the real cachekit-py deserialization path before being written. + Rewrites the whole fixture, so it refuses to write if that + would drop any already-committed vector — the legacy vectors + were generated by cachekit 0.11.1 (array-of-ints envelopes) + and no post-0.4.0-core wheel (bin envelopes) can reproduce + them. Use generate-bin-twin for post-protocol-1.1 updates. + + generate-bin-twin + Append-only: generates ONLY the protocol 1.1 `bin`-envelope + twin of the default-path vector from the installed `cachekit` + wheel and appends it to the existing fixture, leaving every + other vector byte-untouched (LAB-903). Requires `cachekit` + + `msgpack`; refuses to write if the wheel still emits + array-of-ints envelopes. The independent parser below implements exactly the layout documented in spec/wire-format.md: @@ -42,6 +55,28 @@ class FrameError(ValueError): pass +def _require(condition: bool, what: str) -> None: + """Generation-time invariant that survives ``python -O``. + + ``assert`` is stripped under ``-O``; a silently skipped check here could + write a corrupt vector into the cross-SDK source of truth, so these + invariants must always execute. + """ + if not condition: + raise ValueError(f"generation invariant violated: {what}") + + +def _load_fixture() -> dict: + try: + return json.loads(VECTOR_PATH.read_text()) + except OSError as exc: + print(f"cannot read fixture {VECTOR_PATH}: {exc}", file=sys.stderr) + raise SystemExit(1) from exc + except json.JSONDecodeError as exc: + print(f"fixture {VECTOR_PATH} is not valid JSON: {exc}", file=sys.stderr) + raise SystemExit(1) from exc + + def parse_frame(frame: bytes) -> tuple[dict, bytes]: """Independent CK v3 frame parser (deliberately not importing cachekit).""" if frame[:2] != MAGIC: @@ -59,9 +94,33 @@ def parse_frame(frame: bytes) -> tuple[dict, bytes]: return header, frame[header_end:] +#: msgpack tags a ByteStorage envelope's compressed_data field can legally carry. +#: bin8/bin16/bin32 are the protocol 1.1 canonical encoding; fixarray/array16/ +#: array32 are the legacy array-of-ints encoding readers must accept forever. +_BIN_TAGS = (0xC4, 0xC5, 0xC6) +_INT_ARRAY_TAGS = (0xDC, 0xDD, *range(0x90, 0xA0)) + + +def _envelope_encoding(payload: bytes) -> str: + """Classify an envelope's compressed_data encoding from its msgpack tag. + + Stdlib-only, so the no-dependency CI leg proves the protocol 1.1 dual-read + property on its own instead of delegating it to the Node cross-check. + """ + if len(payload) < 2 or payload[0] != 0x94: + raise FrameError(f"envelope is not a msgpack fixarray(4) (leading byte 0x{payload[:1].hex()})") + tag = payload[1] + if tag in _BIN_TAGS: + return "bin" + if tag in _INT_ARRAY_TAGS: + return "int-array" + raise FrameError(f"compressed_data tag 0x{tag:02x} is neither msgpack bin nor an array") + + def verify() -> int: - doc = json.loads(VECTOR_PATH.read_text()) + doc = _load_fixture() failures = 0 + observed_encodings: set[str] = set() for vec in doc["frame_vectors"]: name = vec["name"] @@ -79,6 +138,24 @@ def verify() -> int: if "expected_payload_hex" in vec and payload.hex() != vec["expected_payload_hex"]: print(f"FAIL {name}: payload mismatch") vec_failed += 1 + env = vec.get("payload_envelope") + if env: + declared = env.get("envelope_encoding") + if declared is None: + print(f"FAIL {name}: payload_envelope must declare envelope_encoding ('bin' or 'int-array')") + vec_failed += 1 + else: + try: + actual = _envelope_encoding(payload) + except FrameError as e: + print(f"FAIL {name}: {e}") + vec_failed += 1 + else: + if actual != declared: + print(f"FAIL {name}: compressed_data is {actual}, vector declares {declared}") + vec_failed += 1 + else: + observed_encodings.add(actual) det = vec.get("arrow_detection") if det: off = det["ipc_magic_offset"] @@ -115,6 +192,15 @@ def verify() -> int: print(f"FAIL {name}: expected rejection, parsed successfully") failures += 1 + # Coverage floor. Protocol 1.1 is "writers emit bin, readers accept legacy + # FOREVER"; that dual-read guarantee is only proven while the fixture carries + # a vector in each encoding. Without this, deleting the bin twin — or the + # legacy vector that is the legacy-read proof — leaves the gate green. + for want in ("int-array", "bin"): + if want not in observed_encodings: + print(f"FAIL envelope-encoding coverage: no frame vector exercises the {want} envelope encoding") + failures += 1 + if failures: print(f"\n{failures} failure(s)") return 1 @@ -122,13 +208,77 @@ def verify() -> int: return 0 -def generate() -> int: +def _build_default_path_vector() -> tuple[dict, str]: + """Build the default-@cache-write vector from the installed cachekit wheel. + + Returns (vector_dict, envelope_encoding) where envelope_encoding is + "bin" (msgpack 0xc4/0xc5/0xc6, protocol 1.1 writers) or "int-array" + (legacy array-of-ints writers). Every frame is round-tripped through the + real cachekit-py deserialization path before being returned. + """ import msgpack # third-party; generation only from cachekit._rust_serializer import ByteStorage from cachekit.cache_handler import CacheSerializationHandler from cachekit.serializers.wrapper import SerializationWrapper + value = {"user_id": 42, "name": "cachekit", "active": True} + handler = CacheSerializationHandler(serializer_name="default") + frame = handler.serialize_data(value, cache_key="python-frame-vector") + _require(handler.deserialize_data(frame, cache_key="python-frame-vector") == value, "cachekit-py round-trip mismatch") + payload_mv, meta, ser_name = SerializationWrapper.unwrap(frame) + payload = bytes(payload_mv) + inner, fmt = ByteStorage("msgpack").retrieve(payload) + inner = bytes(inner) + _require(msgpack.unpackb(inner) == value and fmt == "msgpack", "ByteStorage.retrieve round-trip mismatch") + env = msgpack.unpackb(payload) # positional fixarray(4) + _require(isinstance(env, list) and len(env) == 4, "envelope is not a 4-element msgpack array") + # msgpack-python decodes bin as bytes and array-of-ints as list — the + # observed type IS the wire encoding of compressed_data. + encoding = "bin" if isinstance(env[0], bytes) else "int-array" + # Protocol 1.1 scopes the bin flip to compressed_data ONLY: checksum + # [u8;8] must stay an array of integers in every encoding. + _require(isinstance(env[1], list), "checksum drifted to msgpack bin (excluded from the protocol 1.1 flip)") + # `format` is excluded alongside `checksum`, and it feeds AAD construction — + # a drift to bin here must be a named invariant, not a json.dumps TypeError. + _require(isinstance(env[3], str), "format drifted off msgpack str (excluded from the protocol 1.1 flip)") + default_header, _ = parse_frame(frame) + _require(default_header["m"] == meta and default_header["s"] == ser_name, "frame header disagrees with unwrap metadata") + vector = { + "name": "default_saas_write_msgpack_bytestorage", + "description": ( + "Exact stored bytes for a default @cache write (StandardSerializer, integrity on): " + "CK v3 frame wrapping the ByteStorage envelope of the MessagePack-encoded value. " + "This is what any backend — including the SaaS — receives from cachekit-py in auto mode." + ), + "value_json": value, + "frame_hex": frame.hex(), + "expected_header": default_header, + "expected_payload_hex": payload.hex(), + "payload_envelope": { + "encoding": ( + "rmp_serde positional fixarray(4); compressed_data encodes as msgpack bin " + "(serde_bytes, protocol 1.1); checksum [u8;8] stays an array of integers" + if encoding == "bin" + else "rmp_serde::to_vec positional fixarray(4); Vec/[u8;8] fields encode as msgpack arrays of integers" + ), + "envelope_encoding": encoding, + "compressed_data_hex": bytes(env[0]).hex(), + "checksum_hex": bytes(env[1]).hex(), + "original_size": env[2], + "format": env[3], + "inner_msgpack_hex": inner.hex(), + }, + } + return vector, encoding + + +def generate() -> int: + import msgpack # third-party; generation only + + from cachekit.cache_handler import CacheSerializationHandler + from cachekit.serializers.wrapper import SerializationWrapper + import cachekit vectors: list[dict] = [] @@ -138,7 +288,7 @@ def generate() -> int: raw_meta = {"format": "msgpack", "compressed": False} raw_frame = SerializationWrapper.wrap(raw_payload, raw_meta, "default") p, m, s = SerializationWrapper.unwrap(raw_frame) - assert bytes(p) == raw_payload and m == raw_meta and s == "default" + _require(bytes(p) == raw_payload and m == raw_meta and s == "default", "SerializationWrapper round-trip mismatch") # expected_header comes from this tool's own independent parser, so the # vector pins what the frame actually contains (incl. the "v" field, which # cachekit-py's unwrap drops) rather than a hand-maintained copy. @@ -154,41 +304,16 @@ def generate() -> int: ) # 2. Full default-path SaaS write: value -> msgpack -> ByteStorage envelope -> CK frame. - value = {"user_id": 42, "name": "cachekit", "active": True} - handler = CacheSerializationHandler(serializer_name="default") - frame = handler.serialize_data(value, cache_key="python-frame-vector") - assert handler.deserialize_data(frame, cache_key="python-frame-vector") == value - payload_mv, meta, ser_name = SerializationWrapper.unwrap(frame) - payload = bytes(payload_mv) - inner, fmt = ByteStorage("msgpack").retrieve(payload) - inner = bytes(inner) - assert msgpack.unpackb(inner) == value and fmt == "msgpack" - env = msgpack.unpackb(payload) # positional fixarray(4); byte fields are int arrays - assert isinstance(env, list) and len(env) == 4 - default_header, _ = parse_frame(frame) - assert default_header["m"] == meta and default_header["s"] == ser_name - vectors.append( - { - "name": "default_saas_write_msgpack_bytestorage", - "description": ( - "Exact stored bytes for a default @cache write (StandardSerializer, integrity on): " - "CK v3 frame wrapping the ByteStorage envelope of the MessagePack-encoded value. " - "This is what any backend — including the SaaS — receives from cachekit-py in auto mode." - ), - "value_json": value, - "frame_hex": frame.hex(), - "expected_header": default_header, - "expected_payload_hex": payload.hex(), - "payload_envelope": { - "encoding": "rmp_serde::to_vec positional fixarray(4); Vec/[u8;8] fields encode as msgpack arrays of integers", - "compressed_data_hex": bytes(env[0]).hex(), - "checksum_hex": bytes(env[1]).hex(), - "original_size": env[2], - "format": env[3], - "inner_msgpack_hex": inner.hex(), - }, - } - ) + default_vector, encoding = _build_default_path_vector() + if encoding != "int-array": + print( + f"REFUSED: installed cachekit {cachekit.__version__} emits {encoding} envelopes; " + "'generate' cannot reproduce the legacy (array-of-ints) vectors. Use " + "'generate-bin-twin' instead. Nothing written.", + file=sys.stderr, + ) + return 1 + vectors.append(default_vector) # 3. Arrow path: frame wrapping [8-byte xxHash3-64][Arrow IPC file]. # Hard requirement for generation — writing the fixture without this vector @@ -201,12 +326,12 @@ def generate() -> int: df = pd.DataFrame({"id": [1, 2], "score": [1.5, 2.5]}) arrow_frame = arrow_handler.serialize_data(df, cache_key="python-frame-vector") rt = arrow_handler.deserialize_data(arrow_frame, cache_key="python-frame-vector") - assert rt.equals(df) + _require(rt.equals(df), "Arrow round-trip mismatch") a_payload_mv, a_meta, a_ser = SerializationWrapper.unwrap(arrow_frame) a_payload = bytes(a_payload_mv) - assert a_payload[8:14] == b"ARROW1" + _require(a_payload[8:14] == b"ARROW1", "Arrow IPC magic not at documented offset") arrow_header, _ = parse_frame(arrow_frame) - assert arrow_header["m"] == a_meta and arrow_header["s"] == a_ser + _require(arrow_header["m"] == a_meta and arrow_header["s"] == a_ser, "Arrow frame header disagrees with unwrap metadata") vectors.append( { "name": "arrow_dataframe_write", @@ -289,16 +414,99 @@ def generate() -> int: "frame_vectors": vectors, "error_vectors": error_vectors, } + # `generate` rewrites the whole fixture, so it can only ever be additive by + # accident. Refuse to write if that rewrite would drop a vector that is + # already committed — the encoding sniff above catches a post-1.1 wheel, but + # a pre-1.1 wheel passes it and would silently delete the *_bin twins (and + # any other vector this tool no longer knows how to build). Both directions + # close here, on the names actually present, rather than on a wheel version. + dropped = {v["name"] for v in _load_fixture()["frame_vectors"]} - {v["name"] for v in vectors} + if dropped: + print( + f"REFUSED: regenerating would drop committed vector(s): {', '.join(sorted(dropped))}. " + "Committed vectors are the source of truth four SDKs pin — nothing written.", + file=sys.stderr, + ) + return 1 VECTOR_PATH.write_text(json.dumps(doc, indent=2, sort_keys=False) + "\n") print(f"wrote {VECTOR_PATH} ({len(vectors)} frame vectors, {len(error_vectors)} error vectors)") return 0 +def generate_bin_twin() -> int: + """Append the protocol 1.1 bin-envelope twin of the default-path vector. + + Append-only (LAB-903): every existing vector is left byte-untouched — the + legacy vectors are the legacy-read proof and cannot be regenerated by a + post-0.4.0-core wheel. The twin gets its own generator provenance so the + fixture records which wheel produced which vector. + """ + import cachekit + + twin, encoding = _build_default_path_vector() + if encoding != "bin": + print( + f"installed cachekit {cachekit.__version__} still emits {encoding} envelopes; " + "generate-bin-twin needs a wheel carrying cachekit-core >= 0.4.0. Nothing written.", + file=sys.stderr, + ) + return 1 + + twin_name = "default_saas_write_msgpack_bytestorage_bin" + twin["name"] = twin_name + twin["description"] = ( + "Protocol 1.1 twin of default_saas_write_msgpack_bytestorage: same value, same " + "default @cache write path, but the ByteStorage envelope's compressed_data is " + "msgpack bin (serde_bytes) instead of an array of integers. Readers MUST accept " + "both encodings; the legacy encoding stays pinned by the legacy vector's bytes." + ) + twin["generator"] = ( + f"cachekit {cachekit.__version__} (PyPI wheel; Rust core via PyO3), generated by " + "tools/python-frame-reference.py generate-bin-twin" + ) + + doc = _load_fixture() + by_name = {v["name"]: v for v in doc["frame_vectors"]} + if twin_name in by_name: + # Append-only and idempotent: a no-op re-run is success, not failure. + print(f"{twin_name} already present; nothing written.", file=sys.stderr) + return 0 + + # The description above asserts the ENCODING is the only difference from the + # legacy vector. Prove it, rather than trusting the wheel: a wheel that also + # changed the LZ4 level, msgpack key order, or the frame header would append a + # vector that lies about what it isolates, into a fixture downstream SDKs pin. + legacy = by_name.get("default_saas_write_msgpack_bytestorage") + _require(legacy is not None, "legacy default-path vector missing — cannot prove the twin is a twin") + assert legacy is not None # narrowing for type checkers; _require already raised + _require(twin["value_json"] == legacy["value_json"], "twin value_json differs from the legacy vector") + _require(twin["expected_header"] == legacy["expected_header"], "twin frame header differs from the legacy vector") + for field in ("compressed_data_hex", "checksum_hex", "original_size", "format", "inner_msgpack_hex"): + _require( + twin["payload_envelope"][field] == legacy["payload_envelope"][field], + f"twin payload_envelope.{field} differs from the legacy vector — encoding must be the ONLY delta", + ) + doc["frame_vectors"].append(twin) + if not doc["generator"].startswith("legacy vectors:"): + doc["generator"] = ( + f"legacy vectors: {doc['generator']} (unchanged since); " + "*_bin twins carry their own per-vector 'generator' field" + ) + VECTOR_PATH.write_text(json.dumps(doc, indent=2, sort_keys=False) + "\n") + print(f"appended {twin_name} to {VECTOR_PATH} (cachekit {cachekit.__version__})") + return 0 + + if __name__ == "__main__": mode = sys.argv[1] if len(sys.argv) > 1 else "verify" + if mode in ("-h", "--help"): + print(__doc__) + sys.exit(0) if mode == "generate": sys.exit(generate()) + if mode == "generate-bin-twin": + sys.exit(generate_bin_twin()) if mode == "verify": sys.exit(verify()) - print(f"unsupported mode: {mode!r}; expected 'verify' or 'generate'", file=sys.stderr) + print(f"unsupported mode: {mode!r}; expected 'verify', 'generate', or 'generate-bin-twin'", file=sys.stderr) sys.exit(2)