Skip to content

v0.2.2pre1

Pre-release
Pre-release

Choose a tag to compare

View all tags
@taku910 taku910 released this 02 Jul 00:57
v0.2.2pre1
e0cce7d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release Notes v0.2.2 (pre1)

This release contains major updates, including the migration of the Python wrapper to pybind11, native NumPy support, enhanced offset mapping, performance optimizations, security hardening, and internal code modernization.

1. Python Wrapper (pybind11 Migration & Features)

  1. SWIG to pybind11 Migration: Fully migrated the Python wrapper from SWIG to pybind11, enabling better performance, safer memory management, and cleaner code integration (#1266, Commit 528dc9e).
  2. Native NumPy Support:
    • Supported zero-copy decoding for NumPy arrays using IntSpanOrVector (Commit c5b0ce0).
    • Respected buffer strides in CastToVectorInt to avoid Out-Of-Bounds (OOB) reads when processing NumPy arrays (#1275, Commit a0daee2).
  3. Enhanced Offset Mapping:
    • Added new offset mapping features supporting both Unicode character offsets and raw byte offsets.
    • Added bounds checking to prevent OOB reads in offset mapping (Commit c5b0ce0).
    • Fixed crash issues and OOB spans in alignment conversions (#1231, #1211).
  4. Type Stubs (PEP 561): Added inline type stubs (.pyi files) and packaged PEP 561 marker to provide auto-completion and static type safety in IDEs (#1258, Commit 8b7f97b).
  5. Parallel Encoding API: Introduced parallel_encode and ThreadPool APIs, allowing Python users to tokenize single large documents across multiple CPU cores in parallel (#1254, Commit 3c37c39).
  6. Direct Proto Loading: Added support for loading processors and normalizers directly from in-memory Proto instances (Commit 12788ba).
  7. Concurrency Enhancements: Added GIL release during C++ execution to improve multi-threaded Python performance (#1251, Commit 0a044ae).
  8. BOS/EOS Behavior Hardening: Aligned BOS/EOS adding behavior with C++ constraints, raising explicit ValueError instead of silent failures or crashes (Commit 780618a).

2. Performance Optimizations

  1. BPE Training Optimization: Ported O(log n) lazy priority queue update optimization to BPE trainer to speed up training (Commit 8db9878).
  2. Encode/Decode Optimizations: Decoding/Encoding optimization for integer/string input case by skipping SentencePieceText conversion, resulting in ~2x speedup (Commit 4b9231b).
  3. Unigram Pruning: Improved unigram pruning using token-normalized frequency calculations (#1135, Commit 1cd0c59).

3. Security & Hardening

  1. Trie Validation at Load: Added verification of darts-clone trie offsets when loading models to prevent segfaults on corrupted or malicious model files (#1224, Commit 6b2f05d).
  2. Bounds Checking in Maps: Added bounds checks to piece accessors and validated value offsets in DecompileCharsMap (#1265, Commit 82b7c6d).
  3. Crash Prevention: Replaced internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs (Commit 66b2587).
  4. Null Character Disallowment: Explicitly disallow vocabulary pieces containing null characters (Commit eb3ba49).
  5. NBest Timeout: Implemented a global timeout for NBest A* search to prevent hangs on pathological inputs (Commit ba57749).

4. C++ API & Internals Modernization

  1. Abseil Migration: Extensively migrated from custom utilities to Abseil primitives (#1270, #1237, #1236):
    • Migrated sentencepiece::util::Status to absl::Status (Commit 39279bf).
    • Switched to absl::log (deprecating --minloglevel and introducing --quiet flag, Commit 7d25ca7).
    • Replaced custom random generators with absl::random (Commit 678a450).
  2. C++17 Requirement: Modernized the codebase to require C++17 (reverted from C++20 for compatibility, Commit 71de2ab).
  3. Normalizer Maps: Supported loading normalizer from raw maps in C++ (#1273, Commit e476e5e).

5. Build & Packaging

  1. Deprecate Win32 Wheels: Stopped support and distribution of 32-bit Windows wheels (win32) (Commit 24fd00c).
  2. Modernized Python builds: Switched from calling setup.py directly to the standard pypa/build PEP 517 frontend (#1167, Commit f300c7d).
  3. PEP 639 License: Added PEP 639 license metadata to pyproject.toml (#1173).
  4. Alpine & CI Updates: Upgraded build runners and updated alpine Docker images.

6. Documentation

  1. API Cheat Sheet: Added comparison tables, zero-copy, and sampling comparison guides.
  2. Refactored Options: Fully reorganized options.md and special_symbols.md with verified code examples.
Assets 64
Loading