Feat/versioning system#2
Merged
Merged
Conversation
Extends cl100k_base and o200k_base tokenizers with specialized tokens for building chat models, reasoning systems, and autonomous agents. Token IDs are allocated in non-conflicting ranges (cl100k: 100277-100330, o200k: 200019-200072). Token categories: - Conversation structure (system, user, assistant, ChatML delimiters) - Chain-of-Thought reasoning (think blocks for DeepSeek/o1-style reasoning) - ReAct agent loops (plan, step, act, observe) - Tool/function calling (function, result, error with retry handling) - Code execution (Jupyter-style code/output/language blocks) - RAG citations (context, quote, cite, source attribution) - Memory/state (memory store and recall for stateful agents) - Control flow (pad, stop, sep) - Multimodal (image, audio, video placeholders) - Document structure (title, section, summary for semantic parsing) Implementation: - Rust constants with comprehensive documentation in src/core/tokenizer.rs - Python bindings expose AgentTokens classes with token ID mappings - Full module integration across core, Python bindings, and public API - Updated README with feature descriptions and supported model details - New docs/special_tokens.md with design rationale and usage examples
Add unified versioning system that uses .version file as the single source of truth for package versions. The system validates git tags against the base version and automatically converts versions to platform-specific formats during CI builds. Key components: - .version file stores base semantic version (0.2.0) - scripts/update_version.sh validates tags and updates Cargo.toml/pyproject.toml - Release workflow validates tags before building packages - Supports prerelease tags: alpha, beta, rc (case-insensitive) - Converts versions for each platform (Cargo semver vs PyPI PEP 440) - Fails with clear errors on tag/version mismatches This ensures version consistency across Rust and Python packages while supporting both stable and prerelease versions through git tags.
Replace runtime test assertions on constants with compile-time const block assertions to satisfy Clippy lint (assertions_on_constants). The verification logic remains unchanged but now executes at compile time rather than test time, providing earlier detection of constant constraint violations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: implement tag-based versioning system with single source of truth
Add unified versioning system that uses .version file as the single source
of truth for package versions. The system validates git tags against the base
version and automatically converts versions to platform-specific formats
during CI builds.
Key components:
This ensures version consistency across Rust and Python packages while
supporting both stable and prerelease versions through git tags.