v0.2.0

rag-chunk v0.2.0 Release Notes

Summary

Adds token-based chunking via tiktoken, model selection, richer CLI output, improved docs, and expanded tests. Backward compatible.

Added

• --use-tiktoken for token-precise chunking
• --tiktoken-model (default: gpt-3.5-turbo)
• Optional install: pip install rag-chunk[tiktoken]
• Rich table output (auto if rich installed)
• README: tokenization section and usage examples
• Unit tests for token-based paths

Improved

• CLI help clarifies words vs tokens
• Refactored strategy runner (fewer locals, pylint clean)
• Lint compliance across modules

Fixed

• Line length, broad exception, naming, and local variable warnings
• Optional extras warning (prepared for publish)

Usage

Word-based:

rag-chunk analyze examples/ --strategy fixed-size --chunk-size 300