v0.0.8

Release v0.0.8

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Docker

docker run --rm -it ghcr.io/agentstation/tokenizer:0.0.8 llama3 encode "Hello, world!"

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.8

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Shell Completions

Generate shell completions for your shell:

# Bash
tokenizer completion bash > /etc/bash_completion.d/tokenizer

# Zsh
tokenizer completion zsh > "${fpath[1]}/_tokenizer"

# Fish
tokenizer completion fish > ~/.config/fish/completions/tokenizer.fish

Changelog

Bug fixes

• 23b9593: fix: remove deprecated conflicts_with from goreleaser config (@jackspirou)

Others

• 0dd15a6: add svg (@jackspirou)
• 501e035: refactor: consolidate to single memory-efficient scanner implementation (@jackspirou)
• e0e3421: refactor: improve demo with better branding and features (@jackspirou)

Checksums

7da583461172e431e1c4e3ee9d7472d220050578e61f16bb587fb82fab16018d  tokenizer_0.0.8_darwin_arm64.tar.gz
d6d94b14a0473f911b447f60bac0f5a404fe49b886226b893b9159f038313b08  tokenizer_0.0.8_darwin_arm64.tar.gz.sbom.json
86ae838a8ed6d3c9bc9ef0ff433504d7afe747cf198e50d083d84a1c5a765696  tokenizer_0.0.8_darwin_x86_64.tar.gz
5a33d3cc52dc0f6b4a73aa8d5f37502a4e95e446afaa75fc563c799b844e061e  tokenizer_0.0.8_darwin_x86_64.tar.gz.sbom.json
b7c0e722dbb8319519f4af3d0ebb40db150c604c179e161249f751aae5ed44ac  tokenizer_0.0.8_freebsd_arm64.tar.gz
a1ffac8c1cc9de70ac8d0e24135b62c3429ff2aa67a546eebe5cedb4efacfead  tokenizer_0.0.8_freebsd_arm64.tar.gz.sbom.json
34f87a5e44f1e73e0ae29873a35b23c139e345a8b6959bcf9fc7981f87181532  tokenizer_0.0.8_freebsd_x86_64.tar.gz
409b82302d957c1babd63c98c87258297d94a8f4f40c7189fd6b4a8be814fd89  tokenizer_0.0.8_freebsd_x86_64.tar.gz.sbom.json
d39e2c4edcf39480bcc1fd27184f01755e9aa19442b140a2a80ab85c46cc0c60  tokenizer_0.0.8_linux_arm64.tar.gz
2f4477188d9ea78932b6083e832523287a35f7f69f08c333459065f16e2b83c9  tokenizer_0.0.8_linux_arm64.tar.gz.sbom.json
5bd43f98e1770c4ab695ff101837c576f4dc437fb0537de36cc9610914ddd493  tokenizer_0.0.8_linux_armv6.tar.gz
3739695c9e0e92ff92a70285ef6d7389a2efef706a2039258f374a56ee56ca10  tokenizer_0.0.8_linux_armv6.tar.gz.sbom.json
28d2f6a2fedd093d0a7da6587397f9e4a4ea0395cdca8b13aa77e2116a7a9b25  tokenizer_0.0.8_linux_armv7.tar.gz
bd52f152bb950a156275f57841fe844715a7d355528079c728e413b392ff7f56  tokenizer_0.0.8_linux_armv7.tar.gz.sbom.json
46200e1f0d23870376e3cfb4ffb194e2ab611a553c5a1657fb7800c9081da185  tokenizer_0.0.8_linux_x86_64.tar.gz
f30775b31f18b54745c7530915a1a134780d046a94e1e2fbde745fa146777743  tokenizer_0.0.8_linux_x86_64.tar.gz.sbom.json
d019605c3cf691d51b63afa026b088ec437e76f9d3ad41797580ff871426b7e1  tokenizer_0.0.8_windows_x86_64.zip
801f3a0dcb555acf9224eee03f2c9186346464bca449e76bb1a3272c5cd6be5c  tokenizer_0.0.8_windows_x86_64.zip.sbom.json