v1.0.9

What's Changed

• Fix pickling issue in BaseTokenizer by @chonknick in #153
• Improve test coverage for OverlapRefinery by @chonknick in #154
• Feat: Enhance the tests for OverlapRefinery, Tokenizers and SlumberChunker by @chonknick in #155
• Feat: Enhance the tests for SPDMChunker, SemanticChunker, NeuralChunker and Visualizer by @chonknick in #156
• Feat: Add cloud CodeChunker + tests by @chonknick in #157
• Fix: CodeChunker fails multiprocessing due to pickling issues; Run it sequentially by @chonknick in #158
• Add GeminiEmbeddings Support by @chonknick in #159
• Fix: OverlapRefinery with mode="recursive" fails with level gets too low by @chonknick in #161
• Fix: OverlapRefinery is caching too much; revert context_size calculation caching by @chonknick in #162
• Feat: Create Cython functions for split and merge basic ops for chunking! by @chonknick in #163
• Feat: Add CPython optimized methods for split and merge ops for performance boosts! by @chonknick in #164
• Migrate Docs from chonkie-docs repo to chonkie for easier management + maintainance by @chonknick in #149
• Migrate DOCS.md by @chonknick in #165
• Update DOCS by @chonknick in #166
• Update DOCS.md + Add example for JSONPorter by @chonknick in #167
• Feat: Add support for PsycopgHandshake by @chonknick in #171
• Feat: Add PsycopgHandshake for initial support of pgvector by @chonknick in #172
• Update semantic.py to fix check if any of the split token counts are greater than the max chunk size by @geosmart in #175
• Fix: Refactor PyscopgHandshake to PgvectorHandshake via vecs by @chonknick in #174
• Update DOCS.md + Fix SemanticChunker bug for all comparison by @chonknick in #176
• Update version to 1.0.9 in pyproject.toml for the next release. by @chonknick in #177
• Feat: Add initial support for experimental.CodeChunker — improved code chunking for few languages by @chonknick in #178
• Fix: cython module build failing during CD — Add cython build info to pyproject.toml by @chonknick in #180
• Fix: Use cibuildwheel to build and publish wheels to PyPI by @chonknick in #182
• Fix: Update the CD script to use uv instead in the hopes of finally publishing the v1.0.9 by @chonknick in #183

New Contributors

• @geosmart made their first contribution in #175

Full Changelog: v1.0.8...v1.0.9