From 719286513f6dece74ce7b2d8875d19963c773e15 Mon Sep 17 00:00:00 2001 From: Luke Francl Date: Wed, 18 Dec 2024 17:09:41 -0800 Subject: [PATCH] Add "So many tokens, so little time" This is a blog post about a new byte-pair tokenizer written in Rust. --- draft/2024-12-18-this-week-in-rust.md | 1 + 1 file changed, 1 insertion(+) diff --git a/draft/2024-12-18-this-week-in-rust.md b/draft/2024-12-18-this-week-in-rust.md index ca4abe9b2..e940bd653 100644 --- a/draft/2024-12-18-this-week-in-rust.md +++ b/draft/2024-12-18-this-week-in-rust.md @@ -51,6 +51,7 @@ and just ask the editors to select the category. ### Miscellaneous * [Rust social status update 2024.12](https://rust.code-maven.com/rust-update-2024-12-17) +* [So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer](https://github.blog/ai-and-ml/llms/so-many-tokens-so-little-time-introducing-a-faster-more-flexible-byte-pair-tokenizer/) ## Crate of the Week