Skip to content

Releases

Foldlite DB 0.4.2

Choose a tag to compare

@martyn martyn released this 15 Nov 13:39
227f70b

Grouped portable view reduction

  • Add optional reduce_groups: true support to portable view ABI v1 across the native Rust engine (foldlitedb-portable-engine) and browser worker runtime (portable_assets/host.js and worker.js). Views opting in fold up to 32 groups per fresh Wasm instantiation via {"op": "reduce_groups", "groups": [...]} with automatic recursive binary splitting on 256 KiB ABI capacity, Wasmi fuel exhaustion, or explicit {"ok": {"split": true}} responses, amortizing linear-memory allocation across high-cardinality batches while preserving fresh-global isolation and identical per-key state/result checkpoints. Bundled views and existing FOLDLOG2 / .cache artifacts remain unchanged.

Foldlite DB 0.4.1

Choose a tag to compare

@martyn martyn released this 10 Nov 19:48
7391132

Highlights

This release publishes the engineering hardening, reproducible numeric reducers, and SDK/API improvements prepared in 0.4.0 and 0.3.3 (whose PyPI jobs were blocked by the retired macos-14 GitHub Actions runner image; wheels.yml now builds Apple Silicon wheels on macos-15).

  • Reproducible in-repo numeric reducers (native/reducer): Builds the bundled reducer.wasm module (mean/v1, envelope/v1, sum/v1, count/v1, finite-scalar-position/v1) reproducibly from in-repo Rust source with zero-buffering borrowed deserialization, scripts/build_reducer_wasm.py --check, generated third-party license notices, and domain-neutral terminology.
  • Hardened AssemblyScript SDK & portable/reducer hosts: Rejects lone UTF-16 surrogate escapes, trailing non-whitespace JSON tokens, and superlinear big-decimal literals in views/assembly/sdk/ (with rebuilt bundled metrics, catalogue, and documents Wasm modules); enforces IEEE-754 double spellings in portable_assets/host.js, canonicalizes -0 retained keys and aligns cursor errors in portable_assets/worker.js, and enforces pre-write buffer disjointness + post-call failure poisoning across Rust and JS reducer hosts.
  • Native Rust storage & PyO3 engine improvements: Allows Database.migrate on FOLDLOG1 logs to skip an incomplete torn final frame directly and open read-only logs in unwriteable directories; uses single-pass DigestReader .cache verification, zero-copy frame hashing, and 8 MiB :memory: frame limits; replaces English substring error guessing with structured EngineError/ErrorKind; decouples per-batch callback commits from full-state serialization via incremental callback_updates (with a 1 GiB checkpoint ceiling) and batched canonical_keys.
  • Python API & companion module fixes: Disambiguates Database.query(view_name, ...) / rows(view_name, ...) from key=..., supports Database.get(id=None, *, position=None, source="default") with strict validation, wires up batch_size/max_batch_bytes flushing in materialize_jsonl_view and incremental suffix deltas in Database.projection, fixes read_jsonl_tail exact newline-boundary retention, validates sse() event names, wakes idle stream_events immediately on Subscriber.close(), fsyncs repair_jsonl_tail truncations, and ships py.typed.

Upgrading

Existing 0.2.0 and 0.3.x FOLDLOG2 document logs need no log migration. Opening an older database with View.bundled(...) automatically reconstructs its disposable .cache under the hardened 0.4.1 bundled modules and starts a fresh cursor generation; unchanged custom Wasm modules keep their existing checkpoints and cursors.

See the step-by-step upgrade guide and the consolidated changelog for full details.

Foldlite DB 0.4.0

Choose a tag to compare

@martyn martyn released this 10 Nov 19:20
26d3fce

Note: Superseded for PyPI publication by Foldlite DB 0.4.1, which updates the Apple Silicon wheel runner in wheels.yml from the retired macos-14 GitHub Actions image to macos-15.

Highlights

  • Reproducible in-repo numeric reducers (native/reducer): Builds the bundled reducer.wasm module (mean/v1, envelope/v1, sum/v1, count/v1, finite-scalar-position/v1) reproducibly from in-repo Rust source with zero-buffering borrowed deserialization, scripts/build_reducer_wasm.py --check, generated third-party license notices, and domain-neutral terminology.
  • Hardened AssemblyScript SDK & portable/reducer hosts: Rejects lone UTF-16 surrogate escapes, trailing non-whitespace JSON tokens, and superlinear big-decimal literals in views/assembly/sdk/ (with rebuilt bundled metrics, catalogue, and documents Wasm modules); enforces IEEE-754 double spellings in portable_assets/host.js, canonicalizes -0 retained keys and aligns cursor errors in portable_assets/worker.js, and enforces pre-write buffer disjointness + post-call failure poisoning across Rust and JS reducer hosts.
  • Native Rust storage & PyO3 engine improvements: Allows Database.migrate on FOLDLOG1 logs to skip an incomplete torn final frame directly and open read-only logs in unwriteable directories; uses single-pass DigestReader .cache verification, zero-copy frame hashing, and 8 MiB :memory: frame limits; replaces English substring error guessing with structured EngineError/ErrorKind; decouples per-batch callback commits from full-state serialization via incremental callback_updates (with a 1 GiB checkpoint ceiling) and batched canonical_keys.
  • Python API & companion module fixes: Disambiguates Database.query(view_name, ...) / rows(view_name, ...) from key=..., supports Database.get(id=None, *, position=None, source="default") with strict validation, wires up batch_size/max_batch_bytes flushing in materialize_jsonl_view and incremental suffix deltas in Database.projection, fixes read_jsonl_tail exact newline-boundary retention, validates sse() event names, wakes idle stream_events immediately on Subscriber.close(), fsyncs repair_jsonl_tail truncations, and ships py.typed.

Upgrading

Existing 0.2.0 and 0.3.x FOLDLOG2 document logs need no log migration. Opening an older database with View.bundled(...) automatically reconstructs its disposable .cache under the hardened 0.4.0 bundled modules and starts a fresh cursor generation; unchanged custom Wasm modules keep their existing checkpoints and cursors.

See the step-by-step upgrade guide and the consolidated changelog for full details.

Foldlite DB 0.3.3

Choose a tag to compare

@martyn martyn released this 10 Nov 15:22

State machine and lifecycle hardening

  • Validate callback View definitions before opening or resetting the native storage engine so invalid callback views never truncate an existing log when reset=True is passed, and make View.module read-only.
  • Release native ownership without writing a checkpoint ({"op": "close", "checkpoint": false}) when Database(...) initialization aborts on a generation mismatch or callback registration error, preserving any existing valid .cache sidecar on disk; reject opening an existing non-empty untagged database with an explicit generation.
  • Consolidate Python callback view state into a cohesive _RegisteredView structure with atomic rollback on failed register(), enforce re-entrancy guards (_callback_evaluating) across lazy group finalization and incremental projection reduction, and pass rebuilt callback checkpoints into {"op": "rebuild"} so Database.rebuild() writes and syncs .cache once with accurate cache.bytes reporting.
  • Support an explicit view= keyword argument on Database.query() and Database.rows(), reject ambiguous simultaneous positional and keyword key selectors, and prevent guest Wasm "View rejected operation: ..." messages from being misclassified as DocumentConflict or CursorExpired.
  • Resolve symlink targets (including dangling symlinks) before validating primary log paths so dangling symlinks cannot target reserved .cache or .lock sidecars, allow read-only FOLDLOG1 and migration source logs to open without write permissions or in-place tail truncation, skip reading obsolete checkpoints when reset=True, and validate ABI input/output buffer disjointness in native and JS reducer hosts.
  • Preserve caller watermark dictionaries in stream_events, emit gap without a false ready frame when a subscriber overflows during replay, terminate closed subscriber loops cleanly, reject boolean max_bytes in read_jsonl_tail, and mark AppendJournal failed on sync() verification errors.

See the upgrade guide and consolidated changelog.

Foldlite DB 0.3.2

Choose a tag to compare

@martyn martyn released this 03 Nov 16:00
91557f3

Foldlite DB 0.3.2 upgrades the native WebAssembly runtime to Wasmi 2.0.0 for faster portable view execution. Existing module bytes, document logs, caches, and the Python API remain compatible. No database migration is required.

The portable fuel budget is calibrated for Wasmi 2.0 accounting, including near-capacity 1,024-document batches. The release retains bounded memory and request sizes and adds regression coverage for runtime isolation, fuel exhaustion, and resource failures. Custom numeric reducer fuel overrides should be validated against the new accounting. Native third-party notices and installation/release documentation are updated.

See the changelog and upgrade guide.

Foldlite DB 0.3.1

Choose a tag to compare

@martyn martyn released this 03 Nov 06:08
9751c0b

Portable range and prefix queries are available in the native Python API, callback views, and browser worker. Keys use numeric and recursive JSON ordering; stable continuations bind the complete selection.

Upgrade: existing 0.2.0 logs and compatible caches require no migration. Start a fresh query traversal after upgrading. Existing snapshot/change cursors remain valid with a compatible cache. See the upgrade guide.

This release includes the range-query changes prepared for 0.3.0. 0.3.0 was
not published to PyPI:
its Windows validation caught test fixture encoding and
open-log locking assumptions. Its GitHub tag is retained without moving it.

Queries

  • Add ascending range queries with start_key/end_key and inclusive or
    exclusive endpoints, plus string and structural array prefix selection.
  • Expose the same selectors in Python query()/rows(), named callback views,
    and the browser worker. Ranges return existing per-key results; query-time
    reduction across selected keys is not implemented.
  • Order keys by JSON type, numbers numerically, strings by Unicode codepoint,
    and compound keys recursively. This replaces serialized-JSON row ordering;
    string collation is binary rather than CouchDB's ICU ordering.
  • Bind versioned query continuations to the complete selector and snapshot.
    Preserve stable pages during ingestion and allow page-size changes. Reject
    changed selectors and legacy query tokens explicitly.
  • Maintain a derived ordered native query index without changing canonical
    group identity, ingestion order, or sequence allocation. Existing 0.2.0 logs
    and compatible caches require no migration; snapshot/change cursors and
    document pagination remain compatible. Start a fresh query traversal after
    upgrading from 0.2.0.

Documentation

  • Add this consolidated history and a dedicated upgrade guide with backup,
    migration, verification, cutover, rollback, and compatibility steps.
  • Document range/prefix semantics, executable examples, and the 0.2.0-to-0.3.1
    query compatibility steps. Include the changelog in source distributions.

Cross-platform verification

  • Read the shared Unicode query fixture explicitly as UTF-8 and compare log
    bytes after releasing the database's Windows ownership lock.
  • Run the Windows Python 3.11 installed-wheel/source-rebuild checks on every
    pull request and mainline push, alongside the existing Linux checks.

Validation includes 340 Python tests, 22 real-Chromium tests, 61 Rust tests, 13 SDK tests, executable documentation/mobile checks, and a real 0.2.0 cache/cursor upgrade fixture. Windows Python 3.11 now runs in PR/mainline checks. The publishing workflow tests all five wheel platforms and the rebuilt source archive before uploading. All six PyPI distribution hashes match the tested GitHub artifacts. A fresh installation from PyPI passed range/prefix, compound-key ordering, and persistent snapshot continuation checks. See the attached release receipt for hashes and CI evidence.

Foldlite DB 0.3.0

Choose a tag to compare

@martyn martyn released this 03 Nov 05:54
99ff520

Not published to PyPI. The release workflow correctly stopped when Windows validation found two test portability errors: default-locale decoding of the Unicode fixture and reading the exclusively locked log. The runtime compiled successfully. The corrections are published as 0.3.1; this 0.3.0 tag will remain unchanged.

Portable range and prefix queries are available in the native Python API, callback views, and browser worker. Key ordering is numeric and recursive across JSON types, and stable continuation tokens bind the complete selection.

Upgrade: existing 0.2.0 logs and compatible caches require no migration. Start a fresh query traversal after upgrading; existing snapshot/change cursors remain valid with a compatible cache. See the upgrade guide.

Queries

  • Add ascending range queries with start_key/end_key and inclusive or
    exclusive endpoints, plus string and structural array prefix selection.
  • Expose the same selectors in Python query()/rows(), named callback views,
    and the browser worker. Ranges return existing per-key results; query-time
    reduction across selected keys is not implemented.
  • Order keys by JSON type, numbers numerically, strings by Unicode codepoint,
    and compound keys recursively. This replaces serialized-JSON row ordering;
    string collation is binary rather than CouchDB's ICU ordering.
  • Bind versioned query continuations to the complete selector and snapshot.
    Preserve stable pages during ingestion and allow page-size changes. Reject
    changed selectors and legacy query tokens explicitly.
  • Maintain a derived ordered native query index without changing canonical
    group identity, ingestion order, or sequence allocation. Existing 0.2.0 logs
    and compatible caches require no migration; snapshot/change cursors and
    document pagination remain compatible. Start a fresh query traversal after
    upgrading from 0.2.0.

Documentation

  • Add this consolidated history and a dedicated upgrade guide with backup,
    migration, verification, cutover, rollback, and compatibility steps.
  • Document range/prefix semantics, executable examples, and the 0.2.0-to-0.3.0
    query compatibility steps. Include the changelog in source distributions.

Validation includes 340 Python tests, 22 real-Chromium tests, 61 Rust tests, 13 SDK tests, executable documentation and desktop/mobile checks, and a real 0.2.0 cache/cursor upgrade fixture. The release workflow builds and tests five native wheel platforms and the rebuilt source archive before PyPI publication. Distribution hashes and the publishing run will be attached as a receipt after verification.

Foldlite DB 0.2.0

Choose a tag to compare

@martyn martyn released this 03 Nov 03:44
bf44dda

Foldlite DB 0.2.0 unifies native and Python callback document acceptance, commits, and replay, and closes all ten findings in the engineering review.

  • Native reset holds stable ownership locks; checkpoint paths cannot overwrite another database's log.
  • New FOLDLOG2 frames persist predecessor hashes to detect omitted, reordered, and spliced frames.
  • Every callback append API uses native preparation and deduplication. Maps, reducers, and finalizers complete before durable commit; disposable callback caches recover from original batches.
  • External projection descriptors bind state and consumed progress together. Version/reference changes rebuild from the authoritative source.
  • Native and browser hosts share metadata-source, payload-depth, and relocated Wasm output-buffer rules, with shared conformance fixtures.
  • Core checks run before PR merges. Storage failure injection, lifecycle oracles, real legacy migration fixtures, and measured replay/memory/latency reports strengthen validation.

This minor release includes deliberate compatibility changes. Existing FOLDLOG1 logs remain readable and exportable, but new appends require verified copy migration:

from foldlitedb import Database

# Close the source owner first; the destination must not exist.
Database.migrate("old.fold", "migrated.fold")

Migration preserves original batch boundaries and leaves the source unchanged. Keep the original backup until the migrated database has been verified with its view.

Primary filenames ending .cache/.lock or containing .cache. are reserved for sidecars. Source names beginning _ are retained metadata and do not contribute to views in either host. A primary callback now supplies query()/rows(); callback preview()/changes() explicitly reject unsupported operations. Python callbacks remain trusted deterministic code and require version/reference changes when their behavior changes.

Local validation passed 185 Python, 53 Rust, 17 Chromium, and 13 Node tests, plus source-rebuilt wheel checks, executable documentation, reproducible compiled modules, and 1k/10k/100k replay fixtures. The publishing workflow tests five native-wheel platforms and a rebuilt source archive, validates the exact distributions, and publishes those artifacts without rebuilding.

See the engineering review and resolution, API and migration guide, and PRs #6, #7, #8, #9, and #10.

v0.1.4

Choose a tag to compare

@martyn martyn released this 02 Nov 15:09
a7d209c

Security: upgrade wasmi to 0.47.2 for GHSA-g4v2-cjqp-rfmq (CVE-2025-66627), a linear-memory use-after-free when a module grows its memory. wasmi 0.42.0 through 0.47.1 are affected; earlier releases of this package (through 0.1.2) use wasmi 0.46.0. No engine API changes.

This release also includes the changes prepared for 0.1.3, which was not published separately:

  • The native host rejects JSON integers that a double cannot represent exactly (int(float(n)) == n), matching the Python host. Previously 9007199254740993 was silently stored as 9007199254740992.0. Writes and query keys carrying such integers now fail instead of being rounded.
  • ABI_BUFFER_BYTES, MAX_BATCH_DOCUMENTS, MAX_BATCH_EMISSIONS, MAX_LOG_FRAME_BYTES, MAX_JSON_DEPTH and MAX_SAFE_INTEGER are exported from the Rust crate root, and limit errors name the operation, size and limit without referring to a Python API.
  • Engine::writer_poisoned() and Engine::is_writable() report writer health, stats include log.writer_poisoned, and close releases the log handle even when the final checkpoint fails, so a poisoned log can be reopened and retried in the same process.
  • The portable views toolchain requires a Node runtime with the October security fixes (^22.23.4, ^24.22.1 or >=26.12.1).

v0.1.2

Choose a tag to compare

@martyn martyn released this 20 Oct 04:46

Batch projection view group reductions across pages, avoid duplicating raw source frames in derived .fold logs, and cache finalized group results across queries.