You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/docs/features/ai/backend-integration.mdx
+9-4Lines changed: 9 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,12 +11,16 @@ The most common (and recommended) setup to integrate BlockNote AI with an LLM is
11
11
## Default setup (Vercel AI SDK)
12
12
13
13
The example below closely follows the [basic example from the Vercel AI SDK](https://ai-sdk.dev/docs/ai-sdk-ui/chatbot#example) for Next.js.
14
-
The only difference is that we're retrieving the BlockNote tools from the request body and using the `toolDefinitionsToToolSet` function to convert them to AI SDK tools. The LLM will now be able to invoke these tools to make modifications to the BlockNote document as requested by the user. The tool calls are forwarded to the client application where they're handled automatically by the AI Extension.
14
+
The only difference is that we're retrieving the BlockNote tools from the request body and using the `toolDefinitionsToToolSet` function to convert them to AI SDK tools. We also forward the serialized document state (selection, cursor, block IDs) that BlockNote adds to every user message by calling `injectDocumentStateMessages`. The LLM will now be able to invoke these tools to make modifications to the BlockNote document as requested by the user. The tool calls are forwarded to the client application where they're handled automatically by the AI Extension.
@@ -103,4 +108,4 @@ You can connect BlockNote AI features with more advanced AI pipelines. You can i
103
108
with BlockNote AI, [get in touch](/about).
104
109
</Callout>
105
110
106
-
- By default, BlockNote AI composes the LLM request (messages) based on the user's prompt and passes these to your backend. See [this example](https://github.com/TypeCellOS/BlockNote/blob/main/examples/09-ai/07-server-promptbuilder/src/App.tsx) for an example where composing the LLM request (prompt building) is delegated to the server.
111
+
- By default, BlockNote AI sends the entire LLM chat history to the backend. See [the server persistence example](https://github.com/TypeCellOS/BlockNote/tree/main/examples/09-ai/07-server-persistence) for a pattern where the backend stores chat and only the latest message is sent to the backend.
@@ -104,6 +109,12 @@ export async function POST(req: Request) {
104
109
}
105
110
```
106
111
112
+
This follows the regular `streamText` pattern of the AI SDK, with 3 exceptions:
113
+
114
+
- the BlockNote document state is extracted from message metadata and injected into the messages, using `injectDocumentStateMessages`
115
+
- BlockNote client-side tool definitions are extracted from the request body and passed to the LLM using `toolDefinitionsToToolSet`
116
+
- The system prompt is set to the default BlockNote system prompt (`aiDocumentFormats.html.systemPrompt`). You can override or extend the system prompt. If you do so, make sure your modified system prompt still explains the AI on how to modify the BlockNote document.
117
+
107
118
See [Backend integrations](/docs/features/ai/backend-integration) for more information on how to integrate BlockNote AI with your backend.
* Returns a read-only zustand store with the state of the AI Menu
74
+
* Returns a read-only Tanstack Store with the state of the AI Menu
68
75
*/
69
-
get store():ReadonlyStoreApi<{
76
+
get store():Store<{
70
77
aiMenuState:
71
78
| ({
72
79
/**
@@ -91,10 +98,10 @@ class AIExtension {
91
98
}>;
92
99
93
100
/**
94
-
* Returns a zustand store with the global configuration of the AI Extension.
101
+
* Returns a Tanstack Store with the global configuration of the AI Extension.
95
102
* These options are used by default across all LLM calls when calling {@linkinvokeAI}
96
103
*/
97
-
readonly options:StoreApi<AIRequestHelpers>;
104
+
readonly options:Store<AIRequestHelpers>;
98
105
99
106
/** Open the AI menu at a specific block */
100
107
openAIMenuAtBlock(blockID:string):void;
@@ -118,7 +125,7 @@ class AIExtension {
118
125
}
119
126
```
120
127
121
-
### `InvokeAIOptions`
128
+
### `InvokeAI`
122
129
123
130
Requests to an LLM are made by calling `invokeAI` on the `AIExtension` object. This takes an `InvokeAIOptions` object as an argument.
124
131
@@ -138,6 +145,8 @@ type InvokeAIOptions = {
138
145
} &AIRequestHelpers; // Optionally override helpers per request
139
146
```
140
147
148
+
Because `InvokeAIOptions` extends `AIRequestHelpers`, you can override these options on a per-call basis without changing the global extension configuration.
149
+
141
150
## `getStreamToolsProvider`
142
151
143
152
When an LLM is called, it needs to interpret the document and invoke operations to modify it. Use a format's `getStreamToolsProvider` to obtain the tools the LLM may call while editing. In most cases, use `aiDocumentFormats.html.getStreamToolsProvider(...)`.
@@ -159,85 +168,72 @@ type getStreamToolsProvider = (
159
168
) =>StreamToolsProvider;
160
169
```
161
170
162
-
## `AIRequest` and `AIRequestSender` (advanced)
171
+
## Document state builders (advanced)
163
172
164
-
The AIRequest models a single AI operation against the editor (prompt, selection, tools). The AIRequestSender is responsible for submitting that request to the AI SDK layer.
173
+
When BlockNote AI sends a request it also forwards a serialized snapshot of the editor. LLMs use this document state to understand document, cursor position and active selection. The `DocumentStateBuilder` type defines how that snapshot is produced:
174
+
175
+
```typescript
176
+
typeDocumentStateBuilder<T> = (
177
+
aiRequest:Omit<AIRequest, "documentState">,
178
+
) =>Promise<
179
+
| {
180
+
selection:false;
181
+
blocks:BlocksWithCursor<T>[];
182
+
isEmptyDocument:boolean;
183
+
}
184
+
| {
185
+
selection:true;
186
+
selectedBlocks: { id:string; block:T }[];
187
+
blocks: { block:T }[];
188
+
isEmptyDocument:boolean;
189
+
}
190
+
>;
191
+
```
192
+
193
+
By default, `aiDocumentFormats.html.defaultDocumentStateBuilder` is used.
194
+
195
+
## `AIRequest` (advanced)
196
+
197
+
`buildAIRequest` returns everything BlockNote AI needs to execute an AI call:
165
198
166
199
```typescript
167
200
typeAIRequest= {
168
201
editor:BlockNoteEditor;
169
-
chat:Chat<UIMessage>;
170
-
userPrompt:string;
171
202
selectedBlocks?:Block[];
172
203
emptyCursorBlockToDelete?:string;
173
204
streamTools:StreamTool<any>[];
174
-
};
175
-
176
-
typeAIRequestSender= {
177
-
sendAIRequest: (
178
-
aiRequest:AIRequest,
179
-
options:ChatRequestOptions,
180
-
) =>Promise<void>;
205
+
documentState:DocumentState<any>;
206
+
onStart: () =>void;
181
207
};
182
208
```
183
209
184
-
The default `AIRequestSender` used is `defaultAIRequestSender(aiDocumentFormats.html.defaultPromptBuilder, aiDocumentFormats.html.defaultPromptInputDataBuilder)`. It takes an AIRequest and the default prompt builder (see below) to construct the updated messages array and submits this to the AI SDK.
185
-
186
-
## PromptBuilder (advanced)
210
+
## `sendMessageWithAIRequest` (advanced)
187
211
188
-
A `PromptBuilder` allows you to fine-tune the messages sent to the LLM. A `PromptBuilder` mutates the AI SDK `UIMessage[]` in place based on the user prompt and document-specific input data. Input data is produced by a paired `PromptInputDataBuilder`.
189
-
190
-
We recommend forking the [default PromptBuilder](https://github.com/TypeCellOS/BlockNote/blob/main/packages/xl-ai/src/api/formats/html-blocks/defaultHTMLPromptBuilder.ts) as a starting point.
212
+
Use `sendMessageWithAIRequest` when you need to manually call the LLM without updating the state of the BlockNote AI menu.
213
+
For example, you could use this when you want to submit LLM requests from a different context (e.g.: a chat window).
214
+
`sendMessageWithAIRequest` is similar to `chat.sendMessages`, but it attaches the `documentState` to the outgoing message metadata, configures tool streaming, and forwards tool definitions (JSON Schemas) to your backend.
191
215
192
216
```typescript
193
-
// Mutates the messages based on format-specific input data
194
-
exporttypePromptBuilder<E> = (
195
-
messages:UIMessage[],
196
-
inputData:E,
197
-
) =>Promise<void>;
198
-
199
-
// Builds the input data passed to the PromptBuilder from a BlockNote AIRequest
The `invokeAI` function automatically passes the default options set in the `AIExtension` to the LLM request. It also handles the LLM response and updates the state of the AI menu accordingly.
225
+
## `buildAIRequest` (advanced)
213
226
214
-
For advanced use cases, you can also directly use the lower-level `buildAIRequest` and `executeAIRequest` functions to issue an LLM request directly.
215
-
216
-
### `buildAIRequest`
217
-
218
-
Use buildAIRequest to assemble an AIRequest from editor state and configuration.
227
+
Use `buildAIRequest` to assemble an `AIRequest` from editor state if you are bypassing `invokeAI` and call `sendMessageWithAIRequest` directly.
0 commit comments