Skip to content

Commit 75339cf

Browse files
authored
refactor(ai): streamline prompt building on server (#2198)
* initial checkin * fix * misc fixes * fixes * fix test * docs * fix test * fix lockfile * fix error handling * fix lint * fix example * small fixes * fix lock
1 parent b486637 commit 75339cf

File tree

306 files changed

+4501
-5009
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

306 files changed

+4501
-5009
lines changed

docs/content/docs/features/ai/backend-integration.mdx

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,16 @@ The most common (and recommended) setup to integrate BlockNote AI with an LLM is
1111
## Default setup (Vercel AI SDK)
1212

1313
The example below closely follows the [basic example from the Vercel AI SDK](https://ai-sdk.dev/docs/ai-sdk-ui/chatbot#example) for Next.js.
14-
The only difference is that we're retrieving the BlockNote tools from the request body and using the `toolDefinitionsToToolSet` function to convert them to AI SDK tools. The LLM will now be able to invoke these tools to make modifications to the BlockNote document as requested by the user. The tool calls are forwarded to the client application where they're handled automatically by the AI Extension.
14+
The only difference is that we're retrieving the BlockNote tools from the request body and using the `toolDefinitionsToToolSet` function to convert them to AI SDK tools. We also forward the serialized document state (selection, cursor, block IDs) that BlockNote adds to every user message by calling `injectDocumentStateMessages`. The LLM will now be able to invoke these tools to make modifications to the BlockNote document as requested by the user. The tool calls are forwarded to the client application where they're handled automatically by the AI Extension.
1515

1616
```ts app/api/chat/route.ts
1717
import { openai } from "@ai-sdk/openai";
1818
import { convertToModelMessages, streamText } from "ai";
19-
import { toolDefinitionsToToolSet } from "@blocknote/xl-ai";
19+
import {
20+
aiDocumentFormats,
21+
injectDocumentStateMessages,
22+
toolDefinitionsToToolSet,
23+
} from "@blocknote/xl-ai/server";
2024

2125
// Allow streaming responses up to 30 seconds
2226
export const maxDuration = 30;
@@ -26,7 +30,8 @@ export async function POST(req: Request) {
2630

2731
const result = streamText({
2832
model: openai("gpt-4.1"), // see https://ai-sdk.dev/docs/foundations/providers-and-models
29-
messages: convertToModelMessages(messages),
33+
system: aiDocumentFormats.html.systemPrompt,
34+
messages: convertToModelMessages(injectDocumentStateMessages(messages)),
3035
tools: toolDefinitionsToToolSet(toolDefinitions),
3136
toolChoice: "required",
3237
});
@@ -103,4 +108,4 @@ You can connect BlockNote AI features with more advanced AI pipelines. You can i
103108
with BlockNote AI, [get in touch](/about).
104109
</Callout>
105110

106-
- By default, BlockNote AI composes the LLM request (messages) based on the user's prompt and passes these to your backend. See [this example](https://github.com/TypeCellOS/BlockNote/blob/main/examples/09-ai/07-server-promptbuilder/src/App.tsx) for an example where composing the LLM request (prompt building) is delegated to the server.
111+
- By default, BlockNote AI sends the entire LLM chat history to the backend. See [the server persistence example](https://github.com/TypeCellOS/BlockNote/tree/main/examples/09-ai/07-server-persistence) for a pattern where the backend stores chat and only the latest message is sent to the backend.

docs/content/docs/features/ai/getting-started.mdx

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,11 @@ This example follows the [basic example from the AI SDK](https://ai-sdk.dev/docs
8585
```ts app/api/chat/route.ts
8686
import { openai } from "@ai-sdk/openai";
8787
import { convertToModelMessages, streamText } from "ai";
88-
import { toolDefinitionsToToolSet } from "@blocknote/xl-ai";
88+
import {
89+
aiDocumentFormats,
90+
injectDocumentStateMessages,
91+
toolDefinitionsToToolSet,
92+
} from "@blocknote/xl-ai/server";
8993

9094
// Allow streaming responses up to 30 seconds
9195
export const maxDuration = 30;
@@ -95,7 +99,8 @@ export async function POST(req: Request) {
9599

96100
const result = streamText({
97101
model: openai("gpt-4.1"), // see https://ai-sdk.dev/docs/foundations/providers-and-models
98-
messages: convertToModelMessages(messages),
102+
system: aiDocumentFormats.html.systemPrompt,
103+
messages: convertToModelMessages(injectDocumentStateMessages(messages)),
99104
tools: toolDefinitionsToToolSet(toolDefinitions),
100105
toolChoice: "required",
101106
});
@@ -104,6 +109,12 @@ export async function POST(req: Request) {
104109
}
105110
```
106111

112+
This follows the regular `streamText` pattern of the AI SDK, with 3 exceptions:
113+
114+
- the BlockNote document state is extracted from message metadata and injected into the messages, using `injectDocumentStateMessages`
115+
- BlockNote client-side tool definitions are extracted from the request body and passed to the LLM using `toolDefinitionsToToolSet`
116+
- The system prompt is set to the default BlockNote system prompt (`aiDocumentFormats.html.systemPrompt`). You can override or extend the system prompt. If you do so, make sure your modified system prompt still explains the AI on how to modify the BlockNote document.
117+
107118
See [Backend integrations](/docs/features/ai/backend-integration) for more information on how to integrate BlockNote AI with your backend.
108119

109120
# Full Example

docs/content/docs/features/ai/reference.mdx

Lines changed: 61 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@ type AIRequestHelpers = {
3030
*/
3131
transport?: ChatTransport<UIMessage>;
3232

33+
/**
34+
* Use the ChatProvider to customize how the AI SDK Chat instance is created.
35+
* For example, when you want to reuse an existing Chat instance used in the rest of your application.
36+
*
37+
* @note you cannot use both `chatProvider` and `transport` together.
38+
*/
39+
chatProvider?: () => Chat<UIMessage>;
40+
3341
/**
3442
* Customize which stream tools are available to the LLM.
3543
*/
@@ -43,12 +51,11 @@ type AIRequestHelpers = {
4351
chatRequestOptions?: ChatRequestOptions;
4452

4553
/**
46-
* Responsible for submitting a BlockNote AIRequest to the AI SDK.
47-
* Use to transform the messages sent to the LLM.
54+
* Build the serializable document state that will be forwarded to the backend.
4855
*
49-
* @default defaultAIRequestSender(aiDocumentFormats.html.defaultPromptBuilder, aiDocumentFormats.html.defaultPromptInputDataBuilder)
56+
* @default aiDocumentFormats.html.defaultDocumentStateBuilder
5057
*/
51-
aiRequestSender?: AIRequestSender;
58+
documentStateBuilder?: DocumentStateBuilder<any>;
5259
};
5360
```
5461

@@ -64,9 +71,9 @@ class AIExtension {
6471
invokeAI(opts: InvokeAIOptions): Promise<void>;
6572

6673
/**
67-
* Returns a read-only zustand store with the state of the AI Menu
74+
* Returns a read-only Tanstack Store with the state of the AI Menu
6875
*/
69-
get store(): ReadonlyStoreApi<{
76+
get store(): Store<{
7077
aiMenuState:
7178
| ({
7279
/**
@@ -91,10 +98,10 @@ class AIExtension {
9198
}>;
9299

93100
/**
94-
* Returns a zustand store with the global configuration of the AI Extension.
101+
* Returns a Tanstack Store with the global configuration of the AI Extension.
95102
* These options are used by default across all LLM calls when calling {@link invokeAI}
96103
*/
97-
readonly options: StoreApi<AIRequestHelpers>;
104+
readonly options: Store<AIRequestHelpers>;
98105

99106
/** Open the AI menu at a specific block */
100107
openAIMenuAtBlock(blockID: string): void;
@@ -118,7 +125,7 @@ class AIExtension {
118125
}
119126
```
120127

121-
### `InvokeAIOptions`
128+
### `InvokeAI`
122129

123130
Requests to an LLM are made by calling `invokeAI` on the `AIExtension` object. This takes an `InvokeAIOptions` object as an argument.
124131

@@ -138,6 +145,8 @@ type InvokeAIOptions = {
138145
} & AIRequestHelpers; // Optionally override helpers per request
139146
```
140147

148+
Because `InvokeAIOptions` extends `AIRequestHelpers`, you can override these options on a per-call basis without changing the global extension configuration.
149+
141150
## `getStreamToolsProvider`
142151

143152
When an LLM is called, it needs to interpret the document and invoke operations to modify it. Use a format's `getStreamToolsProvider` to obtain the tools the LLM may call while editing. In most cases, use `aiDocumentFormats.html.getStreamToolsProvider(...)`.
@@ -159,85 +168,72 @@ type getStreamToolsProvider = (
159168
) => StreamToolsProvider;
160169
```
161170

162-
## `AIRequest` and `AIRequestSender` (advanced)
171+
## Document state builders (advanced)
163172

164-
The AIRequest models a single AI operation against the editor (prompt, selection, tools). The AIRequestSender is responsible for submitting that request to the AI SDK layer.
173+
When BlockNote AI sends a request it also forwards a serialized snapshot of the editor. LLMs use this document state to understand document, cursor position and active selection. The `DocumentStateBuilder` type defines how that snapshot is produced:
174+
175+
```typescript
176+
type DocumentStateBuilder<T> = (
177+
aiRequest: Omit<AIRequest, "documentState">,
178+
) => Promise<
179+
| {
180+
selection: false;
181+
blocks: BlocksWithCursor<T>[];
182+
isEmptyDocument: boolean;
183+
}
184+
| {
185+
selection: true;
186+
selectedBlocks: { id: string; block: T }[];
187+
blocks: { block: T }[];
188+
isEmptyDocument: boolean;
189+
}
190+
>;
191+
```
192+
193+
By default, `aiDocumentFormats.html.defaultDocumentStateBuilder` is used.
194+
195+
## `AIRequest` (advanced)
196+
197+
`buildAIRequest` returns everything BlockNote AI needs to execute an AI call:
165198

166199
```typescript
167200
type AIRequest = {
168201
editor: BlockNoteEditor;
169-
chat: Chat<UIMessage>;
170-
userPrompt: string;
171202
selectedBlocks?: Block[];
172203
emptyCursorBlockToDelete?: string;
173204
streamTools: StreamTool<any>[];
174-
};
175-
176-
type AIRequestSender = {
177-
sendAIRequest: (
178-
aiRequest: AIRequest,
179-
options: ChatRequestOptions,
180-
) => Promise<void>;
205+
documentState: DocumentState<any>;
206+
onStart: () => void;
181207
};
182208
```
183209

184-
The default `AIRequestSender` used is `defaultAIRequestSender(aiDocumentFormats.html.defaultPromptBuilder, aiDocumentFormats.html.defaultPromptInputDataBuilder)`. It takes an AIRequest and the default prompt builder (see below) to construct the updated messages array and submits this to the AI SDK.
185-
186-
## PromptBuilder (advanced)
210+
## `sendMessageWithAIRequest` (advanced)
187211

188-
A `PromptBuilder` allows you to fine-tune the messages sent to the LLM. A `PromptBuilder` mutates the AI SDK `UIMessage[]` in place based on the user prompt and document-specific input data. Input data is produced by a paired `PromptInputDataBuilder`.
189-
190-
We recommend forking the [default PromptBuilder](https://github.com/TypeCellOS/BlockNote/blob/main/packages/xl-ai/src/api/formats/html-blocks/defaultHTMLPromptBuilder.ts) as a starting point.
212+
Use `sendMessageWithAIRequest` when you need to manually call the LLM without updating the state of the BlockNote AI menu.
213+
For example, you could use this when you want to submit LLM requests from a different context (e.g.: a chat window).
214+
`sendMessageWithAIRequest` is similar to `chat.sendMessages`, but it attaches the `documentState` to the outgoing message metadata, configures tool streaming, and forwards tool definitions (JSON Schemas) to your backend.
191215

192216
```typescript
193-
// Mutates the messages based on format-specific input data
194-
export type PromptBuilder<E> = (
195-
messages: UIMessage[],
196-
inputData: E,
197-
) => Promise<void>;
198-
199-
// Builds the input data passed to the PromptBuilder from a BlockNote AIRequest
200-
export type PromptInputDataBuilder<E> = (aiRequest: AIRequest) => Promise<E>;
201-
202-
// Create an AIRequestSender from your custom builders.
203-
// This lets you plug your PromptBuilder into the request pipeline used by invokeAI/executeAIRequest.
204-
function defaultAIRequestSender<E>(
205-
promptBuilder: PromptBuilder<E>,
206-
promptInputDataBuilder: PromptInputDataBuilder<E>,
207-
): AIRequestSender;
217+
async function sendMessageWithAIRequest(
218+
chat: Chat<UIMessage>,
219+
aiRequest: AIRequest,
220+
message?: Parameters<Chat<UIMessage>["sendMessage"]>[0],
221+
options?: Parameters<Chat<UIMessage>["sendMessage"]>[1],
222+
): Promise<Result<void>>;
208223
```
209224

210-
## Lower-level functions (advanced)
211-
212-
The `invokeAI` function automatically passes the default options set in the `AIExtension` to the LLM request. It also handles the LLM response and updates the state of the AI menu accordingly.
225+
## `buildAIRequest` (advanced)
213226

214-
For advanced use cases, you can also directly use the lower-level `buildAIRequest` and `executeAIRequest` functions to issue an LLM request directly.
215-
216-
### `buildAIRequest`
217-
218-
Use buildAIRequest to assemble an AIRequest from editor state and configuration.
227+
Use `buildAIRequest` to assemble an `AIRequest` from editor state if you are bypassing `invokeAI` and call `sendMessageWithAIRequest` directly.
219228

220229
```typescript
221-
function buildAIRequest(opts: {
230+
async function buildAIRequest(opts: {
222231
editor: BlockNoteEditor;
223-
chat: Chat<UIMessage>;
224-
userPrompt: string;
225232
useSelection?: boolean;
226233
deleteEmptyCursorBlock?: boolean;
227234
streamToolsProvider?: StreamToolsProvider<any, any>;
235+
documentStateBuilder?: DocumentStateBuilder<any>;
228236
onBlockUpdated?: (blockId: string) => void;
229-
}): AIRequest;
230-
```
231-
232-
### `executeAIRequest`
233-
234-
Use executeAIRequest to send it with an AIRequestSender and process streaming tool calls.
235-
236-
```typescript
237-
function executeAIRequest(opts: {
238-
aiRequest: AIRequest;
239-
sender: AIRequestSender;
240-
chatRequestOptions?: ChatRequestOptions;
241237
onStart?: () => void;
242-
}): Promise<void>;
238+
}): Promise<AIRequest>;
243239
```

docs/package.json

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@
7171
"@vercel/analytics": "^1.5.0",
7272
"@vercel/og": "^0.6.8",
7373
"@y-sweet/react": "^0.6.3",
74-
"ai": "^5.0.45",
74+
"ai": "^5.0.102",
7575
"babel-plugin-react-compiler": "19.1.0-rc.2",
7676
"better-auth": "^1.3.27",
7777
"better-sqlite3": "^11.10.0",
@@ -103,8 +103,7 @@
103103
"twoslash": "^0.3.4",
104104
"y-partykit": "^0.0.25",
105105
"yjs": "^13.6.27",
106-
"zod": "^3.25.76",
107-
"zustand": "^5.0.3"
106+
"zod": "^3.25.76"
108107
},
109108
"devDependencies": {
110109
"@blocknote/ariakit": "workspace:*",
@@ -142,4 +141,4 @@
142141
"y-partykit": "^0.0.33",
143142
"yjs": "^13.6.27"
144143
}
145-
}
144+
}

examples/09-ai/01-minimal/.bnexample.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,6 @@
66
"dependencies": {
77
"@blocknote/xl-ai": "latest",
88
"@mantine/core": "^8.3.4",
9-
"ai": "^5.0.45"
9+
"ai": "^5.0.102"
1010
}
1111
}

examples/09-ai/01-minimal/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
"react": "^19.2.0",
2323
"react-dom": "^19.2.0",
2424
"@blocknote/xl-ai": "latest",
25-
"ai": "^5.0.45"
25+
"ai": "^5.0.102"
2626
},
2727
"devDependencies": {
2828
"@types/react": "^19.2.2",

examples/09-ai/02-playground/.bnexample.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,6 @@
66
"dependencies": {
77
"@blocknote/xl-ai": "latest",
88
"@mantine/core": "^8.3.4",
9-
"ai": "^5.0.45"
9+
"ai": "^5.0.102"
1010
}
1111
}

examples/09-ai/02-playground/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
"react": "^19.2.0",
2323
"react-dom": "^19.2.0",
2424
"@blocknote/xl-ai": "latest",
25-
"ai": "^5.0.45"
25+
"ai": "^5.0.102"
2626
},
2727
"devDependencies": {
2828
"@types/react": "^19.2.2",

examples/09-ai/03-custom-ai-menu-items/.bnexample.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"dependencies": {
77
"@blocknote/xl-ai": "latest",
88
"@mantine/core": "^8.3.4",
9-
"ai": "^5.0.45",
9+
"ai": "^5.0.102",
1010
"react-icons": "^5.2.1"
1111
}
1212
}

examples/09-ai/03-custom-ai-menu-items/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
"react": "^19.2.0",
2323
"react-dom": "^19.2.0",
2424
"@blocknote/xl-ai": "latest",
25-
"ai": "^5.0.45",
25+
"ai": "^5.0.102",
2626
"react-icons": "^5.2.1"
2727
},
2828
"devDependencies": {

0 commit comments

Comments
 (0)