new GPTTokens() sometime fails with RuntimeError: unreachable

[takahser]

When using this package, sometimes there are runtime errors in the downstream @dqbd/tiktoken dependency.

Code snippet:

new GPTTokens({
  model: 'gpt-3.5-turbo',
  messages: [
    {
      role: 'user',
      content: JSON.stringify(query),
    },
  ],
});

Error:

/Users/xxx/repos/gpt-pr-comment-summary/crawler/node_modules/@dqbd/tiktoken/tiktoken_bg.cjs:262
            wasm.tiktoken_encode(retptr, this.ptr, ptr0, len0, addHeapObject(allowed_special), addHeapObject(disallowed_special));
                 ^
RuntimeError: unreachable
    at wasm://wasm/00b5f812:wasm-function[563]:0x6a72a
    at wasm://wasm/00b5f812:wasm-function[665]:0x6fd7a
    at wasm://wasm/00b5f812:wasm-function[756]:0x70f7f
    at wasm://wasm/00b5f812:wasm-function[237]:0x5c43a
    at wasm://wasm/00b5f812:wasm-function[200]:0x4db89
    at wasm://wasm/00b5f812:wasm-function[34]:0x1f78a
    at wasm://wasm/00b5f812:wasm-function[159]:0x48dc3
    at Tiktoken.encode (/Users/xxx/repos/gpt-pr-comment-summary/crawler/node_modules/@dqbd/tiktoken/tiktoken_bg.cjs:262:18)
    at GPTTokens.num_tokens_from_messages (/Users/xxx/repos/gpt-pr-comment-summary/crawler/node_modules/gpt-tokens/index.js:162:40)
    at GPTTokens.num_tokens_from_messages (/Users/xxx/repos/gpt-pr-comment-summary/crawler/node_modules/gpt-tokens/index.js:126:25)

[RuslanMikailov]

+1

[Cainier]

Looks like a memory leak issue, I fixed this bug in v1.0.7

#6