Skip to content

v0.6.5

v0.6.5: feat: Make Tokenizer cloneable and introduce TokenBuf
@o24s o24s tagged this 21 Oct 12:16 
This commit fixes the design limitation described in [upstream issue #99](https://github.com/daac-tools/vibrato/issues/99) by removing the lifetime parameter from `Worker`. It also introduces `TokenBuf` and refactors `Tokenizer` to improve flexibility.

1. Remove Lifetime from `Worker`
   `Worker<'t>` no longer borrows a `Tokenizer`. It now owns a clone, removing the lifetime parameter.

2. Make `Tokenizer` Cloneable
   - Internally uses `Arc<Dictionary>` for efficient shared ownership.
   - Implements `Clone` (cheap `Arc` copy).
   - Adds `Tokenizer::new(Dictionary)` and `Tokenizer::with_shared_dictionary(Arc<Dictionary>)` for flexible initialization.

3. Introduce `TokenBuf` for Owned Token Data
   - `Token<'w>` remains a lightweight borrowed view.
   - `TokenBuf` owns its data (`String` fields) and has a `'static` lifetime, suitable for storing or sending tokens across threads.
   - Adds `Token::to_buf()` for explicit conversion, following the `Path::to_path_buf()` pattern.
Assets 2
Loading