Skip to content

feature: llm output tokens batching #628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Oct 1, 2025
Merged

feature: llm output tokens batching #628

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8afb4ee
initial draft of token batching
Sep 23, 2025
5b015e9
reused runner.stats for token data
Sep 24, 2025
5736da8
added token baatching to llms
Sep 26, 2025
26c512e
small refactor, added docs
Sep 26, 2025
cd47f96
fixed bug where first token was emitted before batch
Sep 26, 2025
85f9328
review changes
Oct 1, 2025
a60bb33
review changes
Oct 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 33 additions & 15 deletions docs/docs/02-hooks/01-natural-language-processing/useLLM.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,20 +60,21 @@ For more information on loading resources, take a look at [loading models](../..

### Returns

| Field | Type | Description |
| ------------------ | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `generate()` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
| `interrupt()` | `() => void` | Function to interrupt the current inference. |
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model. |
| `token` | `string` | The most recently generated token. |
| `isReady` | `boolean` | Indicates whether the model is ready. |
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response. |
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
| `configure` | `({ chatConfig?: Partial<ChatConfig>, toolsConfig?: ToolsConfig }) => void` | Configures chat and tool calling. See more details in [configuring the model](#configuring-the-model). |
| `sendMessage` | `(message: string) => Promise<void>` | Function to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
| `messageHistory` | `Message[]` | History containing all messages in conversation. This field is updated after model responds to `sendMessage`. |
| Field | Type | Description |
| ------------------------ | -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `generate()` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
| `interrupt()` | `() => void` | Function to interrupt the current inference. |
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model. |
| `token` | `string` | The most recently generated token. |
| `isReady` | `boolean` | Indicates whether the model is ready. |
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response. |
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
| `configure` | `({chatConfig?: Partial<ChatConfig>, toolsConfig?: ToolsConfig, generationConfig?: GenerationConfig}) => void` | Configures chat and tool calling. See more details in [configuring the model](#configuring-the-model). |
| `sendMessage` | `(message: string) => Promise<void>` | Function to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
| `messageHistory` | `Message[]` | History containing all messages in conversation. This field is updated after model responds to `sendMessage`. |
| `getGeneratedTokenCount` | `() => number` | Returns the number of tokens generated in the last response. |

<details>
<summary>Type definitions</summary>
Expand Down Expand Up @@ -102,9 +103,11 @@ interface LLMType {
configure: ({
chatConfig,
toolsConfig,
generationConfig,
}: {
chatConfig?: Partial<ChatConfig>;
toolsConfig?: ToolsConfig;
generationConfig?: GenerationConfig;
}) => void;
generate: (messages: Message[], tools?: LLMTool[]) => Promise<void>;
sendMessage: (message: string) => Promise<void>;
Expand Down Expand Up @@ -138,6 +141,11 @@ interface ToolCall {
arguments: Object;
}

interface GenerationConfig {
outputTokenBatchSize: number;
batchTimeInterval: number;
}

type LLMTool = Object;
```

Expand All @@ -147,7 +155,7 @@ type LLMTool = Object;

You can use functions returned from this hooks in two manners:

1. Functional/pure - we will not keep any state for you. You'll need to keep conversation history and handle function calling yourself. Use `generate` (and rarely `forward`) and `response`. Note that you don't need to run `configure` to use those. Furthermore, it will not have any effect on those functions.
1. Functional/pure - we will not keep any state for you. You'll need to keep conversation history and handle function calling yourself. Use `generate` (and rarely `forward`) and `response`. Note that you don't need to run `configure` to use those. Furthermore, `chatConfig` and `toolsConfig` will not have any effect on those functions.

2. Managed/stateful - we will manage conversation state. Tool calls will be parsed and called automatically after passing appropriate callbacks. See more at [managed LLM chat](#managed-llm-chat).

Expand Down Expand Up @@ -267,6 +275,12 @@ To configure model (i.e. change system prompt, load initial conversation history

- **`displayToolCalls`** - If set to true, JSON tool calls will be displayed in chat. If false, only answers will be displayed.

**`generationConfig`** - Object configuring generation settings, currently only output token batching.

- **`outputTokenBatchSize`** - Soft upper limit on the number of tokens in each token batch (in certain cases there can be more tokens in given batch, i.e. when the batch would end with special emoji join character).

- **`batchTimeInterval`** - Upper limit on the time interval between consecutive token batches.

### Sending a message

In order to send a message to the model, one can use the following code:
Expand Down Expand Up @@ -459,6 +473,10 @@ The response should include JSON:
}
```

## Token Batching

Depending on selected model and the user's device generation speed can be above 60 tokens per second. If the `tokenCallback` triggers rerenders and is invoked on every single token it can significantly decrease the app's performance. To alleviate this and help improve performance we've implemented token batching. To configure this you need to call `configure` method and pass `generationConfig`. Inside you can set two parameters `outputTokenBatchSize` and `batchTimeInterval`. They set the size of the batch before tokens are emitted and the maximum time interval between consecutive batches respectively. Each batch is emitted if either `timeInterval` elapses since last batch or `countInterval` number of tokens are generated. This allows for smooth generation even if model lags during generation. Default parameters are set to 10 tokens and 80ms for time interval (~12 batches per second).

## Available models

| Model Family | Sizes | Quantized |
Expand Down
Loading