Feature request: JS CDN file for web LLM

[AshD]

For projects that don't use NPM, Is it possible to create a Javascript version that is deployed on a CDN?

This will make it easier to integrate with ASP.NET projects that don't use NPM.

Thanks,
Ash

[flatsiedatsie]

You can try this one:

https://esm.sh/@mlc-ai/web-llm@0.2.8?bundle-deps

[gaiborjosue]

Check this out @AshD: https://www.jsdelivr.com/package/npm/@mlc-ai/web-llm

[AwokeKnowing]

@gaiborjosue can you give the minimal js example using that? not sure what to import or call.

that file starts with "import require$$3 from"perf_hooks";" but i don't have any 'perf_hooks' module. To clarify, people get stuff from cdn, so that they don't require a build step. I should just import something from a cdn url and then call the functions it exports, in pure plain javascript. For something so fundamental as an LLM, this makes perfect sense to bundle as a library you just import and call.

edit:
so far what I did:

1. download the file to web-llm.js
2. delete import require$$3 from 'perf_hooks'; from the beginning of the file
3. add return Date at top of the getPerformance function

Now i can run the example on the site:

import * as webllm from "./web-llm.js";

async function main() {
  const initProgressCallback = (report: webllm.InitProgressReport) => {
    const label = document.getElementById("init-label");
    label.innerText = report.text;
  };
  const selectedModel = "Llama-3-8B-Instruct-q4f32_1";
  const engine: webllm.EngineInterface = await webllm.CreateEngine(
    selectedModel,
    /*engineConfig=*/{ initProgressCallback: initProgressCallback }
  );

  const reply0 = await engine.chat.completions.create({
    messages: [{ "role": "user", "content": "Tell me about Pittsburgh." }]
  });
  console.log(reply0);
  console.log(await engine.runtimeStatsText());
}

main();

[flatsiedatsie]

I've ran into the same issue where I found it surprisingly difficult integrate WebLLM in a project. I wrote a cargo-cult-ish modification of the source code to get around this. Your solution would be cleaner.

add return Date at top of the getPerformance function

I can't find that function in the file? Where should I modify that?

[gaiborjosue]

Oh wow, thanks so much @AwokeKnowing! I was running into the same issue in the morning. Thanks for posting the solution:)

[gaiborjosue]

Hi @flatsiedatsie the file @AwokeKnowing is talking about is: https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.35/lib/index.js

Edit:

We have hosted the edited CDN at: https://mpsych.github.io/cdn/web-llm/web-llm-cdn.js so you can use that directly without the need of manually downloading and updating the code :)

[CharlieFRuan]

Thanks for all your work and contribution! A similar solution is included in npm 0.2.36, and the following code snippet in https://jsfiddle.net/ should work with no setups:

// See https://www.npmjs.com/package/@mlc-ai/web-llm documentation.
import * as webllm from 'https://esm.run/@mlc-ai/web-llm';

async function main() {
  const initProgressCallback = (report) => {
    console.log(report.text);
  };
  const selectedModel = "TinyLlama-1.1B-Chat-v0.4-q4f16_1-1k";
  const engine = await webllm.CreateEngine(
    selectedModel,
    {
      initProgressCallback: initProgressCallback
    }
  );

  const reply0 = await engine.chat.completions.create({
    messages: [{
      "role": "user",
      "content": "Tell me about Pittsburgh."
    }]
  });
  console.log(reply0);
  console.log(await engine.runtimeStatsText());
}

main();

[flatsiedatsie]

I just re-implemented WebLLM in my project for the third time :-) But this time without any hacks, and with an easier upgrade path. Everything works as advertised. Thank you!