NPM package

[atgctg]

Would it be possible to add a wasm target and make tiktoken available for Node.js projects?

I'm currently relying on gpt-3-encoder but would prefer to use tiktoken for performance reasons.

[hauntsaninja]

Thanks for the suggestion! I'm not currently planning on implementing this, but it is likely that at some point we will.

If other people encountering this also have this feature request, please thumbs up the original post.

Note that the third party gpt-3-encoder library will not work exactly right for any of the Codex or GPT-3.5 series models, including code-cushman-001, code-davinci-002, text-davinci-002, text-davinci-003, etc. It will also not work at all for e.g. the recent embeddings models, like text-embedding-ada-002

[hauntsaninja]

Just published a new version of tiktoken that includes a mapping from model to tokeniser. Anything not using r50k is liable to be incorrect (sometimes subtly, sometimes majorly, sometimes majorly but you won't notice) with the third party gpt-3-encoder library: https://github.com/openai/tiktoken/blob/main/tiktoken/model.py#L7

[dqbd]

Hello, I've been working on JS bindings for tiktoken, found here: https://github.com/dqbd/tiktoken. Core methods are implemented, some methods are missing for now.

You can install it as such:

npm install @dqbd/tiktoken

If it is desired, I could try to wrangle the changes and create an upstream PR.

Note: Couldn't secure the tiktoken NPM package name, as it is currently owned by @gmpetrov

[ceifa]

Hey, I also made a simple alternative for tiktoken on Node.js! A lot of features are missing too, but I plan to make a 1:1 library with the python version.

Link: https://github.com/ceifa/tiktoken-node

Because it relies on node addon instead of webassembly, it is 5-6x faster than @dqbd approach.