v1.0.4

✨ Highlights

1. Welcome our newest chunker to the family: CodeChunker! CodeChunker is specialized to handle code files and can gain structural understanding of the code before chunking each portion separately~ Supports 100+ programming languages! Let's check out the usage~

Firstly, install the code chunker dependencies via:

pip install "chonkie[code]"

and then simply run it like any other chunker~

# Import
from chonkie import CodeChunker

# Init
chunker = CodeChunker(language="python")

# Get some python code
code = ...

# CHONK!
chunks = chunker(code) 

2. Added support for JinaAI embeddings with JinaEmbeddings — allowing for use with SemanticChunker and SDPMChunker!

Install it via the following command:

pip install "chonkie[jina]"

and use it like this~

# Import 
from chonkie import JinaEmbeddings 
from chonkie import SemanticChunker

# Optionally import Visaulizer for visalizations
from chonkie import Visualizer()

# Init
viz = Visualizer()
embeddings = JinaEmbeddings()
chunker = SemanticChunker(embeddings) 


# Get some text
text = ...

# CHONK!
chunks = chunker(text) 

# Optional: Visualize!
viz(chunks) 

3. Added support for the OverlapRefinery which allows you to add overlap context to your chunks~ It's available in the default install and can be used with any chunker. Just chunk with a chunker and pass your chunks through the refinery!

# import
from chonkie import RecursiveChunker, OverlapRefinery

# Init
chunker = RecursiveChunker()
refinery = OverlapRefinery("gpt2") # By default initializes with "character" tokenizer, can pass in "gpt2" to match the chunker

# Get some text 
text = ...

# CHONK and Refine!
chunks = chunker(text) 
chunks = refinery(chunks)

4. Added support for the EmbeddingsRefinery which allows you to run the chunks through a embedding model and have the embeddings available for downstream loading in a vector database. Similar to the OverlapRefinery just pass in the chunks from a Chunker into a EmbeddingsRefinery object loaded with the appropriate embedding model and each Chunk will then be loaded with .embedding value which can be used downstream.

What's Changed

• Add initial support for chunking code via CodeChunker by @chonknick in #53
• Add Initial support for code chunking via CodeChunker by @chonknick in #54
• Feat: Add Jina AI Embeddings support by @Harsha-Karimikonda in #35
• Feat: Add support for JinaEmbeddings + OverlapRefinery + EmbeddingsRefinery by @chonknick in #57
• Fix: Paths for the Chonkie Cloud chunkers; No module named 'chonkie.cloud.chunkers' error by @chonknick in #58
• Fix: Attempt fixing chonkie.cloud path bug + update README.md to have integrations by @chonknick in #60

New Contributors

• @Harsha-Karimikonda made their first contribution in #35

Full Changelog: v1.0.3...v1.0.4