0.5.1

Readium 0.5.1 Release Notes

Overview

Readium 0.5.1 introduces significant enhancements to token tree functionality, streamlines the CLI interface, updates dependencies, and simplifies the codebase. This release focuses on improving the analysis capabilities while making the tool more maintainable and user-friendly.

Key Improvements

Token Tree Enhancements

• Added comprehensive token tree functionality to both CLI and Python API
• Integrated token count display by file and directory structure
• Included token tree by default in standard output
• Implemented the tiktoken library for reliable tokenization of documentation content

CLI Interface Improvements

• Added new --tokens flag for displaying token metrics
• Introduced dedicated subcommand for showing only the token tree
• Simplified handling of arguments like --exclude-dir
• Enhanced validation for user inputs to prevent common errors
• Improved error messaging for better troubleshooting

Dependency Management

• Made tiktoken a core dependency rather than optional
• Removed the unnecessary tokenizers optional dependency group
• Upgraded pypdf to version 4.3.1 for better compatibility

Codebase Simplification

• Removed unused print_error utility and related imports
• Streamlined __all__ definitions in core modules
• Improved code organization for better maintainability
• Enhanced type annotations for better code quality

Cleanup

• Removed the deprecated pr.sh script to reduce repository clutter
• Eliminated redundant code and simplified implementation patterns

Conclusion

Readium 0.5.1 significantly enhances the tool's ability to analyze documentation through token metrics while improving the developer and user experience through a more streamlined interface and simplified codebase. These improvements make Readium more powerful for extracting insights from documentation across repositories, directories, and URLs.