Skip to content

v1.19.0 - Enhanced Pagination & Token Estimation

Choose a tag to compare

View all tags
@afterxleep afterxleep released this 12 Sep 17:38
· 22 commits to main since this release
v1.19.0
2e120dc

🚀 Version 1.19.0 - Enhanced Pagination & Token Estimation

✨ Major Features

🧠 Intelligent Token Estimation

  • NEW: TokenEstimator utility class with sophisticated tokenization patterns
  • Realistic Accuracy: 2.7-3.7 chars/token vs previous naive 4:1 ratio
  • Content-Aware: Different estimation for code blocks, markdown, technical terms, URLs
  • Reusable: Available across all services for consistent token counting

📄 Advanced Pagination System

  • Text-Level Pagination: Chunks large documents at optimal token boundaries
  • Smart Chunking: Respects word boundaries, preserves formatting
  • Agent-Friendly: Clear navigation headers and pagination instructions
  • Token Compliance: All responses stay within 25K token limits

🔧 Enhanced MCP Tools

  • get_global_rules: Now supports page parameter for large rule sets
  • read_specific_document: Automatic pagination for large documents
  • Backward Compatible: Existing calls without page parameter still work

🛠️ Technical Improvements

PaginationService Enhancements

  • Professional Chunking: Token-aware text splitting with proper boundaries
  • Navigation Headers: Pagination guidance for AI agents
  • Performance: Efficient chunking with minimal overhead

Core Integration

  • index.js: Updated to use realistic token counting
  • Services: Consistent TokenEstimator usage across codebase
  • Tests: Comprehensive test coverage for all new functionality

For Developers

// NEW: Import TokenEstimator for accurate token counting
import { TokenEstimator } from './src/utils/TokenEstimator.js';

// Use realistic token estimation
const tokens = TokenEstimator.estimateTokens(text);
const exceedsLimit = TokenEstimator.exceedsLimit(text, 25000);

Full Changelog: v1.18.0...v1.19.0

Assets 2
Loading