Use a JS based tokenizer for token counting

[dqbd]

The idea behind this PR is to reduce the friction when adopting LangchainJS by rewriting the tokenizer library into JS and fetching ranks on demand from a CDN (https://tiktoken.pages.dev). As a side effect, this will reduce the bundle size (necessary for edge functions, see #809, #62), at the expense of slower tokenization (which should be a non-issue for pure token counting).

Tokenization behavior should match the @dqbd/tiktoken, which has been entirely removed.

Supersedes #847 and #1146

• Move the tests to ensure correct tokenization
• Remove WASM tiktoken entirely in count_tokens (weak dependencies are not well supported afaict)
• Serve other encoders from CDN (w/ cache)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview	May 17, 2023 1:21pm

[nfcampos]

Looks great!

[nfcampos]

Let's also update the test-exports packages to remove any custom config we had put in there for wasm etc

[nfcampos]

We can probably remove this file completely?

[anzemur]

@nfcampos @dqbd I think this is not quite right - the unpackaged size of js-tiktoken is almost 10MB and this package is listed as the dependency. The usage of the library in langchainjs is implemented in a way that the encoders are always loaded from the CDN the first time they are used, so the actual encoder files are never used - this means that we have 10MB of overhead here. The "light" version of js-tiktoken should only include the code required for running the tokenizers, and the user should provide the actual encoder - from local files or from CDN.

The size of langchainjs itself is almost 5MB and with all of the dependencies, this can cause troubles in environments where package size is limited (ex. AWS Lambda).

[dqbd]

Hey @anzemur! As mentioned in dqbd/tiktoken#68 but applicable for LangchainJS as well, the best course of action is to add an intermediate build step for tree shaking and bundling dependencies, such as esbuild or rollup (as documented here). That should address your concerns with the bundle size as well as increase your performance of Lambda functions