Skip to content

Native artifact: libtokenizers 1.27.0 static lib, Windows x64 (GNU, source-built)

Pre-release
Pre-release

Choose a tag to compare

View all tags
@NestorCanales NestorCanales released this 17 Jul 18:52
tokenizers-1.27.0-windows-x64
bc9dadc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Static tokenizer library (libtokenizers.a) for Windows x64, built from the official daulet/tokenizers v1.27.0 source tag (MIT). No upstream Windows prebuilt exists — the project publishes prebuilt archives for macOS and Linux only — so the local-embeddings epic builds and hosts it here, pinned by SHA-256 in assets/manifest.json like every other artifact.

Built on Windows with the GNU Rust toolchain (matches the MinGW gcc that CGo uses to link the Go binary — the MSVC toolchain would produce an incompatible .lib):

rustup default stable-x86_64-pc-windows-gnu
git clone --depth 1 --branch v1.27.0 https://github.com/daulet/tokenizers
cd tokenizers && cargo build --release
# artifact: target/release/libtokenizers_ffi.a  (renamed to libtokenizers.a to match -ltokenizers)

Toolchain: rustc 1.97.1 (GNU), Go 1.26.5, MinGW gcc 16.1.0. Full procedure + link-flag notes in Documentation/windows-build.md.

Assets 3
Loading