v0.1.0

Initial release! Repository made public.

Changelog

• c676e1e Merge pull request #17 from cohere-ai/tokenizer-build
• fb1fdea Merge pull request #24 from cohere-ai/remove-excess-folders
• f80af56 oopsies didn't add the folders to remove ig?
• e0f0134 Merge pull request #23 from cohere-ai/change-tokenizer-folder
• d3d451f Merge pull request #22 from cohere-ai/tokenizer-logo-delete
• 3600b4e changed to vocab
• f6e576d Update README.md
• 3a90599 whoops didn't add these
• f354316 changed name of folder
• c2fddff Update README.md
• d0f5b57 Update README.md
• 5a56628 Update README.md
• f5819b2 Update README.md
• 168ff4e delete cokenizer :(
• 885dcf1 Merge pull request #14 from cohere-ai/readme-initial-draft
• 996fbfa Update README.md
• 0820d22 removed installation & deprecated functions
• f5904ec Create README.md
• 5fd3d45 Update README.md
• 35661d5 revert
• df411b5 gorelease
• 944ccdd Update README.md
• ce9c711 fix
• 85fe254 Merge pull request #16 from cohere-ai/documentingSpeed
• 5956745 removed 500mb and 1GB test (take wayyyyy too long to run & worthless)
• 62448a4 Merge pull request #20 from cohere-ai/privatizing
• 236e91e changed all non-struct / encode / decode functions to private
• 013d906 Merge pull request #18 from cohere-ai/revert-15-getTokens
• 30a2788 Revert "added GetTokens and ListTokens"
• ca17157 Merge pull request #15 from cohere-ai/getTokens
• ecd0b71 fix no main function error
• 5e6dd48 add goreleaser
• f3305b0 refactored some code (added a default benchmark encoder constructor) and a test to make sure the two "constructors" for the encoders create the same fields
• 3e4d91f added benchmarks for 1k tokens encoding + decoding
• a6ad988 changed constructor
• fcf41a8 move badge thingy
• 38f97e3 yay image wokring
• 97a96c1 cokenizer image test 1
• 93281fd cokenizer image :)
• f229b85 prepping to add image
• e3b00d5 newline so import shows in code block
• 2678b2e added GetTokens and ListTokens
• 1336b08 updating README
• 67d08d6 Merge pull request #11 from cohere-ai/testing
• e7e12d4 oops
• 734aae0 newline
• 647fa6c Change char
• 2bee5af vendored
• 6bd3934 added linting and fixed build error
• cc8157d added testing ci
• 6c0b2f2 uncomment
• d9801b4 remove print
• 69e1360 fix
• f23fd64 fix tests
• 638d00c Merge pull request #10 from cohere-ai/refactor-and-cleanup
• a4cd33d formatting fix
• ec25dcf oops
• d8a72bf added test and took out normalization
• 8921daf revert
• e149059 made punctuation an import
• ef10cd2 fixed comment
• f2a8698 Basic refactoring + cleanup
• ca4ab09 init
• e9b2e85 Update README.md
• 8c3993c Initial commit