Skip to content

chunkshop 0.3.2 (last Postgres-only release)

Choose a tag to compare

@TheYonk TheYonk released this 05 May 10:32
· 573 commits to main since this release

The last release of the Postgres-only line. 0.4.x introduces modular
sink backends — MariaDB, Clickhouse, SQLite, with PG kept at full parity.

Highlights

  • Universal if_oversize: ChunkerConfig field on every chunker config
    in both Python and Rust. Routes any chunk whose embedded_content or
    original_content exceeds the effective ceiling through a fallback
    chunker. Chains up to 5 levels deep (deeper raises an explicit error).
  • fixed_overlap.max_chars (optional) — char-bounded as well as
    word-bounded.
  • Wrapper effective ceilingneighbor_expand / summary_embed /
    hierarchical_summary resolve their ceiling as
    cfg.max_chars > base.max_chars > None. Wrappers inherit by default.
  • Dedup'd WARN-once-per-cell when if_oversize is unset and an
    oversize chunk would be emitted. Names the chunker, the ceiling, and a
    copy-paste suggestion. No log spam.
  • Coarse-row exemption on hierarchical_summary — coarse rows
    (one-per-group) are skipped from the check by design.
  • Rust semantic chunker now logs tracing::warn! on hard-split,
    matching Python's semantic.py:120. Parity gap closed.
  • Recursion guardif_oversize chains beyond depth 5 raise
    OversizeRecursionError (Python) / Error::OversizeRecursion (Rust).
  • NEW docs/samples/if-oversize/ — runnable demo showing both the
    WARN behavior (no fallback) and the fallback chain (with fallback).
  • docs/chunkers.md oversize-behavior table refreshed with a concrete
    Setting if_oversize section.

Install

pip install chunkshop==0.3.2
cargo install chunkshop-rs --version 0.3.2

What's next: 0.4.x — modular backends

Postgres stops being load-bearing in the sink layer. MariaDB is the
first sibling backend to ship; PG stays at full parity. Spec:
docs/spec/v4.0-modular-backends.md (see commit 4b22380).

Full changelog: https://github.com/yonk-labs/chunkshop/blob/v0.3.2/CHANGELOG.md
Compare: v0.3.1...v0.3.2