@sveltejs/adapter-cloudflare: cloudflare worker (pages) with vite: "default" is not exported by module

[happysalada]

hi, I've tried the full vite and cloudflare worker setup, however vite seems to have a problem on the wasm file

error during build:
RollupError: "default" is not exported by "node_modules/.pnpm/@dqbd+tiktoken@1.0.7/node_modules/@dqbd/tiktoken/lite/tiktoken_bg.wasm", imported by "src/routes/+page.server.ts".
    at error (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:2125:30)
    at Module.error (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:13452:16)
    at Module.traceVariable (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:13863:29)
    at ModuleScope.findVariable (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:12418:39)
    at ReturnValueScope.findVariable (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:6966:38)
    at ChildScope.findVariable (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:6966:38)
    at BlockScope.findVariable (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:6966:38)
    at ReturnValueScope.findVariable (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:6966:38)
    at Identifier.bind (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:8116:40)
    at CallExpression.bind (file:///Users/raphael/dev/glad-chat/node_modules/.pnpm/rollup@3.21.5/node_modules/rollup/dist/es/shared/node-entry.js:5735:28)
 ELIFECYCLE  Command failed with exit code 1.

just to make sure, I've got both

		wasm(),
		topLevelAwait(),

in my vite config
and I have got the following in my server page

import { Tiktoken, init } from "@dqbd/tiktoken/lite/init";
import wasm from "@dqbd/tiktoken/lite/tiktoken_bg.wasm";
import model from "@dqbd/tiktoken/encoders/cl100k_base.json";

await init((imports) => WebAssembly.instantiate(wasm, imports));
	  const encoder = new Tiktoken(
	      model.bpe_ranks,
	      model.special_tokens,
	      model.pat_str
	    );

not having the init line actually creates an error that the wasm wasn't initialized properly.

(I'm using sveltekit and deploying on cloudflare pages, just for context).

[matthewrobb]

I too am having this issue with the exact same situation

[happysalada]

@matthewrobb actually after more digging it seems to be a problem with the vite wasm plugin, ive opened an issue there if you want to check it out.

[happysalada]

The documentation on this repo is just wrong

[dqbd]

Hi! Sorry for the delay.

The combination of SvelteKit + Cloudflare Pages / Workers has not been properly tested yet. Will reflect that in README.md.

@happysalada, @matthewrobb could you please share a repository with a reproducible case? Even small things such as version of the packages may be the root of the issue.

If that is not possible, consider using js-tiktoken instead, which is a pure JS port instead of WASM bindings. This does come at a certain performance penalty (at the moment around 5x slower, future PRs will reduce the gap to 2.5x), but will work just fine for these untested environments.

---

Here are some preliminary notes when finding the cause of the issue.

SvelteKit + Cloudflare is a tricky combination. When using tiktoken/lite/init + tiktoken/lite/tiktoken_bg.wasm with vite-plugin-wasm, Vite inlines the WASM file and instantiates it directly AND with missing properties (memory etc.). This does go against current init() function, which assumes a WASM file to be a WebAssembly.Module instead.

On the other way, when using tiktoken/lite directly and importing WASM file is done by the package itself, the WASM import results into a WebAssembly.Module is found in default key, requiring calling init() inside the package.

[happysalada]

here is the repo https://github.com/happysalada/glad-chat
I've updated the dependencies to latest.
running pnpm run build
should get you the error.

I've tried import * as wasm from ... but that seems to mess the worker such that it returns a 405 on any request.

I've tried using titoken/lite directly and in that case the project builds fine, but once deployed it errors out with (log) TypeError: malloc is not a function
(my test for that is at happysalada/glad-chat@cec73b1)

[happysalada]

@dqbd did you have a chance to have a look ?

[happysalada]

small update that since I'll be making commits to the repo
here is the commit where the problem can be reproduced https://github.com/happysalada/glad-chat/tree/68d6d2bfa1895a73881ee8f292f2565a3be58c4b

[dqbd]

Hello! Can reproduce the issue, but still need more time to see whether it is something I can do on the package side. Meanwhile updated the README.md to reflect the incompatibility issue.

[happysalada]

After reporting on the vite wasm plugin Menci/vite-plugin-wasm#33 (comment)
The feedback I had was that it should be a simple import of Tiktoken since wasm files don't have a default import.
I tried it on this branch https://github.com/happysalada/glad-chat/blob/wasm/src/routes/%2Bpage.server.ts#L7
and the error I'm getting is
"TypeError: malloc is not a function"
hopefully that's helpful.

[SunnyGPT]

Hello! Can reproduce the issue, but still need more time to see whether it is something I can do on the package side. Meanwhile updated the README.md to reflect the incompatibility issue.

Can I pay you to integrate it with cloudflare worker? Because I don't know programming, it is too difficult to deploy, so I look forward to your reply.

[dqbd]

Hi! For now, consider using js-tiktoken instead, which makes more sense in size-constrainted edge environments. Feel free to reopen if needed

[niieani]

Hey folks, if you're still looking for a tokenizer that works well on edge workers, you can try gpt-tokenizer. Latest version has been designed to work great in size/resource-constrained environments, like edge/Cloudflare Workers (see benchmarks).

import tokenizer from "gpt-tokenizer/model/gpt-4o"

tokenizer.encode('hello, this is a test') // returns `[24912, 11, 495, 382, 261, 1746]`