Integrating Kalosm into a Next.js App

[armelhbobdad]

👋 Hello @ealmloff and floneum maintainers,

I’m exploring how to embed the Kalosm Rust crate (the core of Floneum’s AI stack) directly into a Next.js application’s server-side code. I’ve identified two main paths:

1. Native Node Addon via NAPI-RS

   • Pros: stable ABI across Node.js versions, minimal runtime overhead
   • Cons: requires a native build step and proper handling of .node files

2. WebAssembly Module via wasm-pack

   • Pros: pure WASM—no native binaries, easy npm packaging
   • Cons: initial load latency for module instantiation

I’d love to gather your thoughts on:

• Preferred integration method: Do you recommend NAPI-RS, WASM, or another approach (e.g., Neon, Vercel custom runtime)?
• Model asset management: Any best practices for bundling or loading the quantized model files at runtime?
• Performance considerations: Have you benchmarked Kalosm’s inference overhead in Node.js, and if so, could you share any insights?
• TypeScript ergonomics: Which approach offers the smoothest .d.ts support for Kalosm bindings?

Thank you for any guidance or examples you can share! 🙏

[ealmloff]

Kalosm doesn't compile to wasm yet, so you need to create native bindings if you want to use it in Node. In the future, the new fusor runtime should enable WebGpu support for web apps. Kalosm will handle downloading the models at runtime if you point to huggingface or you can load a model you have already downloaded. I haven't done anything with node bindings before so I'm not sure how they effect performance or what the typing interface looks like

[armelhbobdad]

Thanks so much for the clarification! 🙏
It would be really nice if future versions with Fusor included some documentation or examples on how to integrate it within the Node.js ecosystem, especially with frameworks like Next.js. That would help a lot of folks building local-first or full-stack JS apps tap into Kalosm more easily. Excited to see where it goes! 🚀