This repository was archived by the owner on Oct 2, 2023. It is now read-only.
Releases: cohere-ai/tokenizer
Releases · cohere-ai/tokenizer
v1.1.2
v1.1.1
v1.0.4
v1.0.3
v1.0.2
v1.0.1
1.0.1
v1.0.0
v0.1.0
Initial release! Repository made public.
Changelog
- c676e1e Merge pull request #17 from cohere-ai/tokenizer-build
- fb1fdea Merge pull request #24 from cohere-ai/remove-excess-folders
- f80af56 oopsies didn't add the folders to remove ig?
- e0f0134 Merge pull request #23 from cohere-ai/change-tokenizer-folder
- d3d451f Merge pull request #22 from cohere-ai/tokenizer-logo-delete
- 3600b4e changed to vocab
- f6e576d Update README.md
- 3a90599 whoops didn't add these
- f354316 changed name of folder
- c2fddff Update README.md
- d0f5b57 Update README.md
- 5a56628 Update README.md
- f5819b2 Update README.md
- 168ff4e delete cokenizer :(
- 885dcf1 Merge pull request #14 from cohere-ai/readme-initial-draft
- 996fbfa Update README.md
- 0820d22 removed installation & deprecated functions
- f5904ec Create README.md
- 5fd3d45 Update README.md
- 35661d5 revert
- df411b5 gorelease
- 944ccdd Update README.md
- ce9c711 fix
- 85fe254 Merge pull request #16 from cohere-ai/documentingSpeed
- 5956745 removed 500mb and 1GB test (take wayyyyy too long to run & worthless)
- 62448a4 Merge pull request #20 from cohere-ai/privatizing
- 236e91e changed all non-struct / encode / decode functions to private
- 013d906 Merge pull request #18 from cohere-ai/revert-15-getTokens
- 30a2788 Revert "added GetTokens and ListTokens"
- ca17157 Merge pull request #15 from cohere-ai/getTokens
- ecd0b71 fix no main function error
- 5e6dd48 add goreleaser
- f3305b0 refactored some code (added a default benchmark encoder constructor) and a test to make sure the two "constructors" for the encoders create the same fields
- 3e4d91f added benchmarks for 1k tokens encoding + decoding
- a6ad988 changed constructor
- fcf41a8 move badge thingy
- 38f97e3 yay image wokring
- 97a96c1 cokenizer image test 1
- 93281fd cokenizer image :)
- f229b85 prepping to add image
- e3b00d5 newline so import shows in code block
- 2678b2e added GetTokens and ListTokens
- 1336b08 updating README
- 67d08d6 Merge pull request #11 from cohere-ai/testing
- e7e12d4 oops
- 734aae0 newline
- 647fa6c Change char
- 2bee5af vendored
- 6bd3934 added linting and fixed build error
- cc8157d added testing ci
- 6c0b2f2 uncomment
- d9801b4 remove print
- 69e1360 fix
- f23fd64 fix tests
- 638d00c Merge pull request #10 from cohere-ai/refactor-and-cleanup
- a4cd33d formatting fix
- ec25dcf oops
- d8a72bf added test and took out normalization
- 8921daf revert
- e149059 made punctuation an import
- ef10cd2 fixed comment
- f2a8698 Basic refactoring + cleanup
- ca4ab09 init
- e9b2e85 Update README.md
- 8c3993c Initial commit