Repository navigation
Releases
Foldlite DB 0.4.2
Grouped portable view reduction
- Add optional
reduce_groups: truesupport to portable view ABI v1 across the native Rust engine (foldlitedb-portable-engine) and browser worker runtime (portable_assets/host.jsandworker.js). Views opting in fold up to 32 groups per fresh Wasm instantiation via{"op": "reduce_groups", "groups": [...]}with automatic recursive binary splitting on 256 KiB ABI capacity, Wasmi fuel exhaustion, or explicit{"ok": {"split": true}}responses, amortizing linear-memory allocation across high-cardinality batches while preserving fresh-global isolation and identical per-key state/result checkpoints. Bundled views and existingFOLDLOG2/.cacheartifacts remain unchanged.
Foldlite DB 0.4.1
Highlights
This release publishes the engineering hardening, reproducible numeric reducers, and SDK/API improvements prepared in 0.4.0 and 0.3.3 (whose PyPI jobs were blocked by the retired macos-14 GitHub Actions runner image; wheels.yml now builds Apple Silicon wheels on macos-15).
- Reproducible in-repo numeric reducers (
native/reducer): Builds the bundledreducer.wasmmodule (mean/v1,envelope/v1,sum/v1,count/v1,finite-scalar-position/v1) reproducibly from in-repo Rust source with zero-buffering borrowed deserialization,scripts/build_reducer_wasm.py --check, generated third-party license notices, and domain-neutral terminology. - Hardened AssemblyScript SDK & portable/reducer hosts: Rejects lone UTF-16 surrogate escapes, trailing non-whitespace JSON tokens, and superlinear big-decimal literals in
views/assembly/sdk/(with rebuilt bundledmetrics,catalogue, anddocumentsWasm modules); enforces IEEE-754 double spellings inportable_assets/host.js, canonicalizes-0retained keys and aligns cursor errors inportable_assets/worker.js, and enforces pre-write buffer disjointness + post-call failure poisoning across Rust and JS reducer hosts. - Native Rust storage & PyO3 engine improvements: Allows
Database.migrateonFOLDLOG1logs to skip an incomplete torn final frame directly and open read-only logs in unwriteable directories; uses single-passDigestReader.cacheverification, zero-copy frame hashing, and 8 MiB:memory:frame limits; replaces English substring error guessing with structuredEngineError/ErrorKind; decouples per-batch callback commits from full-state serialization via incrementalcallback_updates(with a 1 GiB checkpoint ceiling) and batchedcanonical_keys. - Python API & companion module fixes: Disambiguates
Database.query(view_name, ...)/rows(view_name, ...)fromkey=..., supportsDatabase.get(id=None, *, position=None, source="default")with strict validation, wires upbatch_size/max_batch_bytesflushing inmaterialize_jsonl_viewand incremental suffix deltas inDatabase.projection, fixesread_jsonl_tailexact newline-boundary retention, validatessse()event names, wakes idlestream_eventsimmediately onSubscriber.close(),fsyncsrepair_jsonl_tailtruncations, and shipspy.typed.
Upgrading
Existing 0.2.0 and 0.3.x FOLDLOG2 document logs need no log migration. Opening an older database with View.bundled(...) automatically reconstructs its disposable .cache under the hardened 0.4.1 bundled modules and starts a fresh cursor generation; unchanged custom Wasm modules keep their existing checkpoints and cursors.
See the step-by-step upgrade guide and the consolidated changelog for full details.
Foldlite DB 0.4.0
Note: Superseded for PyPI publication by Foldlite DB 0.4.1, which updates the Apple Silicon wheel runner in
wheels.ymlfrom the retiredmacos-14GitHub Actions image tomacos-15.
Highlights
- Reproducible in-repo numeric reducers (
native/reducer): Builds the bundledreducer.wasmmodule (mean/v1,envelope/v1,sum/v1,count/v1,finite-scalar-position/v1) reproducibly from in-repo Rust source with zero-buffering borrowed deserialization,scripts/build_reducer_wasm.py --check, generated third-party license notices, and domain-neutral terminology. - Hardened AssemblyScript SDK & portable/reducer hosts: Rejects lone UTF-16 surrogate escapes, trailing non-whitespace JSON tokens, and superlinear big-decimal literals in
views/assembly/sdk/(with rebuilt bundledmetrics,catalogue, anddocumentsWasm modules); enforces IEEE-754 double spellings inportable_assets/host.js, canonicalizes-0retained keys and aligns cursor errors inportable_assets/worker.js, and enforces pre-write buffer disjointness + post-call failure poisoning across Rust and JS reducer hosts. - Native Rust storage & PyO3 engine improvements: Allows
Database.migrateonFOLDLOG1logs to skip an incomplete torn final frame directly and open read-only logs in unwriteable directories; uses single-passDigestReader.cacheverification, zero-copy frame hashing, and 8 MiB:memory:frame limits; replaces English substring error guessing with structuredEngineError/ErrorKind; decouples per-batch callback commits from full-state serialization via incrementalcallback_updates(with a 1 GiB checkpoint ceiling) and batchedcanonical_keys. - Python API & companion module fixes: Disambiguates
Database.query(view_name, ...)/rows(view_name, ...)fromkey=..., supportsDatabase.get(id=None, *, position=None, source="default")with strict validation, wires upbatch_size/max_batch_bytesflushing inmaterialize_jsonl_viewand incremental suffix deltas inDatabase.projection, fixesread_jsonl_tailexact newline-boundary retention, validatessse()event names, wakes idlestream_eventsimmediately onSubscriber.close(),fsyncsrepair_jsonl_tailtruncations, and shipspy.typed.
Upgrading
Existing 0.2.0 and 0.3.x FOLDLOG2 document logs need no log migration. Opening an older database with View.bundled(...) automatically reconstructs its disposable .cache under the hardened 0.4.0 bundled modules and starts a fresh cursor generation; unchanged custom Wasm modules keep their existing checkpoints and cursors.
See the step-by-step upgrade guide and the consolidated changelog for full details.
Foldlite DB 0.3.3
State machine and lifecycle hardening
- Validate callback
Viewdefinitions before opening or resetting the native storage engine so invalid callback views never truncate an existing log whenreset=Trueis passed, and makeView.moduleread-only. - Release native ownership without writing a checkpoint (
{"op": "close", "checkpoint": false}) whenDatabase(...)initialization aborts on a generation mismatch or callback registration error, preserving any existing valid.cachesidecar on disk; reject opening an existing non-empty untagged database with an explicitgeneration. - Consolidate Python callback view state into a cohesive
_RegisteredViewstructure with atomic rollback on failedregister(), enforce re-entrancy guards (_callback_evaluating) across lazy group finalization and incremental projection reduction, and pass rebuilt callback checkpoints into{"op": "rebuild"}soDatabase.rebuild()writes and syncs.cacheonce with accuratecache.bytesreporting. - Support an explicit
view=keyword argument onDatabase.query()andDatabase.rows(), reject ambiguous simultaneous positional and keyword key selectors, and prevent guest Wasm"View rejected operation: ..."messages from being misclassified asDocumentConflictorCursorExpired. - Resolve symlink targets (including dangling symlinks) before validating primary log paths so dangling symlinks cannot target reserved
.cacheor.locksidecars, allow read-onlyFOLDLOG1and migration source logs to open without write permissions or in-place tail truncation, skip reading obsolete checkpoints whenreset=True, and validate ABI input/output buffer disjointness in native and JS reducer hosts. - Preserve caller watermark dictionaries in
stream_events, emitgapwithout a falsereadyframe when a subscriber overflows during replay, terminate closed subscriber loops cleanly, reject booleanmax_bytesinread_jsonl_tail, and markAppendJournalfailed onsync()verification errors.
See the upgrade guide and consolidated changelog.
Foldlite DB 0.3.2
Foldlite DB 0.3.2 upgrades the native WebAssembly runtime to Wasmi 2.0.0 for faster portable view execution. Existing module bytes, document logs, caches, and the Python API remain compatible. No database migration is required.
The portable fuel budget is calibrated for Wasmi 2.0 accounting, including near-capacity 1,024-document batches. The release retains bounded memory and request sizes and adds regression coverage for runtime isolation, fuel exhaustion, and resource failures. Custom numeric reducer fuel overrides should be validated against the new accounting. Native third-party notices and installation/release documentation are updated.
See the changelog and upgrade guide.
Foldlite DB 0.3.1
Portable range and prefix queries are available in the native Python API, callback views, and browser worker. Keys use numeric and recursive JSON ordering; stable continuations bind the complete selection.
Upgrade: existing 0.2.0 logs and compatible caches require no migration. Start a fresh query traversal after upgrading. Existing snapshot/change cursors remain valid with a compatible cache. See the upgrade guide.
This release includes the range-query changes prepared for 0.3.0. 0.3.0 was
not published to PyPI: its Windows validation caught test fixture encoding and
open-log locking assumptions. Its GitHub tag is retained without moving it.
Queries
- Add ascending range queries with
start_key/end_keyand inclusive or
exclusive endpoints, plus string and structural arrayprefixselection. - Expose the same selectors in Python
query()/rows(), named callback views,
and the browser worker. Ranges return existing per-key results; query-time
reduction across selected keys is not implemented. - Order keys by JSON type, numbers numerically, strings by Unicode codepoint,
and compound keys recursively. This replaces serialized-JSON row ordering;
string collation is binary rather than CouchDB's ICU ordering. - Bind versioned query continuations to the complete selector and snapshot.
Preserve stable pages during ingestion and allow page-size changes. Reject
changed selectors and legacy query tokens explicitly. - Maintain a derived ordered native query index without changing canonical
group identity, ingestion order, or sequence allocation. Existing 0.2.0 logs
and compatible caches require no migration; snapshot/change cursors and
document pagination remain compatible. Start a fresh query traversal after
upgrading from 0.2.0.
Documentation
- Add this consolidated history and a dedicated upgrade guide with backup,
migration, verification, cutover, rollback, and compatibility steps. - Document range/prefix semantics, executable examples, and the 0.2.0-to-0.3.1
query compatibility steps. Include the changelog in source distributions.
Cross-platform verification
- Read the shared Unicode query fixture explicitly as UTF-8 and compare log
bytes after releasing the database's Windows ownership lock. - Run the Windows Python 3.11 installed-wheel/source-rebuild checks on every
pull request and mainline push, alongside the existing Linux checks.
Validation includes 340 Python tests, 22 real-Chromium tests, 61 Rust tests, 13 SDK tests, executable documentation/mobile checks, and a real 0.2.0 cache/cursor upgrade fixture. Windows Python 3.11 now runs in PR/mainline checks. The publishing workflow tests all five wheel platforms and the rebuilt source archive before uploading. All six PyPI distribution hashes match the tested GitHub artifacts. A fresh installation from PyPI passed range/prefix, compound-key ordering, and persistent snapshot continuation checks. See the attached release receipt for hashes and CI evidence.
Foldlite DB 0.3.0
Not published to PyPI. The release workflow correctly stopped when Windows validation found two test portability errors: default-locale decoding of the Unicode fixture and reading the exclusively locked log. The runtime compiled successfully. The corrections are published as 0.3.1; this 0.3.0 tag will remain unchanged.
Portable range and prefix queries are available in the native Python API, callback views, and browser worker. Key ordering is numeric and recursive across JSON types, and stable continuation tokens bind the complete selection.
Upgrade: existing 0.2.0 logs and compatible caches require no migration. Start a fresh query traversal after upgrading; existing snapshot/change cursors remain valid with a compatible cache. See the upgrade guide.
Queries
- Add ascending range queries with
start_key/end_keyand inclusive or
exclusive endpoints, plus string and structural arrayprefixselection. - Expose the same selectors in Python
query()/rows(), named callback views,
and the browser worker. Ranges return existing per-key results; query-time
reduction across selected keys is not implemented. - Order keys by JSON type, numbers numerically, strings by Unicode codepoint,
and compound keys recursively. This replaces serialized-JSON row ordering;
string collation is binary rather than CouchDB's ICU ordering. - Bind versioned query continuations to the complete selector and snapshot.
Preserve stable pages during ingestion and allow page-size changes. Reject
changed selectors and legacy query tokens explicitly. - Maintain a derived ordered native query index without changing canonical
group identity, ingestion order, or sequence allocation. Existing 0.2.0 logs
and compatible caches require no migration; snapshot/change cursors and
document pagination remain compatible. Start a fresh query traversal after
upgrading from 0.2.0.
Documentation
- Add this consolidated history and a dedicated upgrade guide with backup,
migration, verification, cutover, rollback, and compatibility steps. - Document range/prefix semantics, executable examples, and the 0.2.0-to-0.3.0
query compatibility steps. Include the changelog in source distributions.
Validation includes 340 Python tests, 22 real-Chromium tests, 61 Rust tests, 13 SDK tests, executable documentation and desktop/mobile checks, and a real 0.2.0 cache/cursor upgrade fixture. The release workflow builds and tests five native wheel platforms and the rebuilt source archive before PyPI publication. Distribution hashes and the publishing run will be attached as a receipt after verification.
Foldlite DB 0.2.0
Foldlite DB 0.2.0 unifies native and Python callback document acceptance, commits, and replay, and closes all ten findings in the engineering review.
- Native reset holds stable ownership locks; checkpoint paths cannot overwrite another database's log.
- New FOLDLOG2 frames persist predecessor hashes to detect omitted, reordered, and spliced frames.
- Every callback append API uses native preparation and deduplication. Maps, reducers, and finalizers complete before durable commit; disposable callback caches recover from original batches.
- External projection descriptors bind state and consumed progress together. Version/reference changes rebuild from the authoritative source.
- Native and browser hosts share metadata-source, payload-depth, and relocated Wasm output-buffer rules, with shared conformance fixtures.
- Core checks run before PR merges. Storage failure injection, lifecycle oracles, real legacy migration fixtures, and measured replay/memory/latency reports strengthen validation.
This minor release includes deliberate compatibility changes. Existing FOLDLOG1 logs remain readable and exportable, but new appends require verified copy migration:
from foldlitedb import Database
# Close the source owner first; the destination must not exist.
Database.migrate("old.fold", "migrated.fold")Migration preserves original batch boundaries and leaves the source unchanged. Keep the original backup until the migrated database has been verified with its view.
Primary filenames ending .cache/.lock or containing .cache. are reserved for sidecars. Source names beginning _ are retained metadata and do not contribute to views in either host. A primary callback now supplies query()/rows(); callback preview()/changes() explicitly reject unsupported operations. Python callbacks remain trusted deterministic code and require version/reference changes when their behavior changes.
Local validation passed 185 Python, 53 Rust, 17 Chromium, and 13 Node tests, plus source-rebuilt wheel checks, executable documentation, reproducible compiled modules, and 1k/10k/100k replay fixtures. The publishing workflow tests five native-wheel platforms and a rebuilt source archive, validates the exact distributions, and publishes those artifacts without rebuilding.
See the engineering review and resolution, API and migration guide, and PRs #6, #7, #8, #9, and #10.
v0.1.4
Security: upgrade wasmi to 0.47.2 for GHSA-g4v2-cjqp-rfmq (CVE-2025-66627), a linear-memory use-after-free when a module grows its memory. wasmi 0.42.0 through 0.47.1 are affected; earlier releases of this package (through 0.1.2) use wasmi 0.46.0. No engine API changes.
This release also includes the changes prepared for 0.1.3, which was not published separately:
- The native host rejects JSON integers that a double cannot represent exactly (int(float(n)) == n), matching the Python host. Previously 9007199254740993 was silently stored as 9007199254740992.0. Writes and query keys carrying such integers now fail instead of being rounded.
- ABI_BUFFER_BYTES, MAX_BATCH_DOCUMENTS, MAX_BATCH_EMISSIONS, MAX_LOG_FRAME_BYTES, MAX_JSON_DEPTH and MAX_SAFE_INTEGER are exported from the Rust crate root, and limit errors name the operation, size and limit without referring to a Python API.
- Engine::writer_poisoned() and Engine::is_writable() report writer health, stats include log.writer_poisoned, and close releases the log handle even when the final checkpoint fails, so a poisoned log can be reopened and retried in the same process.
- The portable views toolchain requires a Node runtime with the October security fixes (^22.23.4, ^24.22.1 or >=26.12.1).