v1.0.3

✨ Highlights

• The new Chonkie Visualizer is here! You can now view your chunks, understand chunk quality and debug your chunker with visual feedback~ Use the print method to print rich text on your terminal or use the save method to save a highlighted html on your device! It's very simple to use, just pass in your chunks~

from chonkie import Visualizer

viz = Visualizer()

# Print the chunks on the terminal with .print or directly call the Visualizer object too
viz.print(chunks) 

# Save the HTML file
viz.save("chonkie.html", chunks)

• Chonkie now adds support for Recipes which allow you to use multilingual chunking out of the box, as well as document specific chunking methods. Initial support starts with: en, hi, zh, jp and ko, while document type markdown is supported too. Use it via the from_recipe class method with any chunker that takes delimiters or RecursiveRules.

from chonkie import RecursiveChunker

# Initialize the recursive chunker to chunk Markdown
chunker = RecursiveChunker.from_recipe("markdown", lang="en")

# Initialize the recursive chunker to chunk Hindi texts
chunker = RecursiveChunker.from_recipe(lang="hi")

• Additional fixes for performance enhancements in RecursiveChunker, SentenceChunker, and WordTokenizer

What's Changed

• Fix: Refactor SentenceChunker to remove estimate + feedback by @chonknick in #23
• Fix: RecursiveRules.from_dict() showing Key error: 'levels' does not exist because of .pop() by @chonknick in #24
• Fix: Remove .find for indexing in the RecursiveChunker for better efficiency by @Pratik960 in #26
• Feat: Add Initial support for Recipes and multilingual CHONKs! by @chonknick in #27
• Fix: RecursiveChunker whitespace splitting is not reconstructable (missing spaces) + (#26, #27) by @chonknick in #28
• Update version to 1.0.3a1 in pyproject.toml and init.py for Chonkie by @chonknick in #29
• Fix: WordTokenizer.count_tokens should use .tokenize instead of .encode by @chonknick in #36
• Add Chonkie Vizard — easy chunk visualization with Visualizer by @chonknick in #39
• Feat: Add Chonkie Vizard to main! Visualizer class for easy chunk visualizations by @chonknick in #40
• Fix: Add chonkie.utils module to package list in pyproject.toml by @chonknick in #41
• Fix: chonkie.cloud does not contain chunkers error + bump up version to v1.0.3 by @chonknick in #43

New Contributors

• @Pratik960 made their first contribution in #26

Full Changelog: v1.0.2...v1.0.3