v1.0.6

✨ Highlights

1. Welcome Chonkie's very own agentic chunker, SlumberChunker! Requires the genie optional install to work. Genie is Chonkie's Generative Inference Engines, that allow any generative models or API to easily plug-in with Chonkie. Currently, the genie optional install installs dependencies for GeminiGenie — and will require a GEMINI_API_KEY to work properly.

pip install "chonkie[genie]"

# Import
from chonkie import SlumberChunker

# Initialize
chunker  = SlumberChunker(verbose=True) # set verbose to True, since it takes a while~

# CHONK!
chunker(text) 

2. A fully neural approach to chunking, NeuralChunker! Requires the neural optional install to work. This uses a BERT-like model that's fine-tuned for chunking, making it really fast and high-quality. Second only to SlumberChunker in terms of chunk quality.

pip install "chonkie[neural]"

# import 
from chonkie import NeuralChunker

# initialize
chunker = NeuralChunker()

# CHONK!
chunks = chunker(text) 

3. Added auto language detection for CodeChunker! Now you can just pass in code to it without having to specify the language before hand. It will detect the language by itself. While the latency for detection is minimal (sub-millisecond), it does affect performance, so please specify the language if you care.

# Import
from chonkie import CodeChunker

# Initialize the "auto" CodeChunker
chunker = CodeChunker() # No need to specify, "auto" by default

# CHONK!
chunks = chunker(code) 

4. Added Genies! Since this version features our first generative feature, the SlumberChunker, we added the Genie to work with it as well as future generative features. Genie's are chonkie's way to handle multiple APIs and model interfaces working together for chunking~ The first Genie to be added is the GeminiGenie — which works with Gemini models. Requires the genie optional install.

pip install "chonkie[genie]"

# Import 
from chonkie import GeminiGenie

# Init
genie = GeminiGenie(api_key=YOUR_API_KEY)

# generate
genie.generate("Hi!") 

# generate JSON
genie.generate_json("Hi", JSON_SCHEMA) 

What's Changed

• Feat : update test workflow to run on prs even before merging by @not-lain in #69
• Feat: Run CI/CD on PRs as well by @chonknick in #70
• Feat: Add support for auto-detecting language in CodeChunker via Magika by @chonknick in #71
• Feat: Add initial support for SlumberChunker by @chonknick in #73
• Feat: Add auto language support for CodeChunker + Add initial support for SlumberChunker by @chonknick in #74
• Update version to 1.0.6a0 in pyproject.toml and init.py for upcom… by @chonknick in #75
• Fix: Resolve pickling error in BaseTokenizer by replacing lambda by @Harsha-Karimikonda in #76
• Fix: SlumberChunker's splitting algorithm from RecursiveChunker to custom + Pickling issue in BaseTokenizer by @chonknick in #78
• Feat: Add initial support for the NeuralChunker by @chonknick in #77
• Feat: Add support for NeuralChunker + Bump version to v1.0.6 by @chonknick in #79
• Update README.md by @chonknick in #80

New Contributors

• @not-lain made their first contribution in #69

Full Changelog: v1.0.5...v1.0.6