Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 201 additions & 0 deletions aiprompts/openai-request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
# OpenAI Request Input Field Structure (On-the-Wire Format)

This document describes the actual JSON structure sent to the OpenAI API in the `input` field of [`OpenAIRequest`](../pkg/aiusechat/openai/openai-convertmessage.go:111).

## Overview

The `input` field is a JSON array containing one of three object types:

1. **Messages** (user/assistant) - `OpenAIMessage` objects
2. **Function Calls** (tool invocations) - `OpenAIFunctionCallInput` objects
3. **Function Call Results** (tool outputs) - `OpenAIFunctionCallOutputInput` objects

These are converted from [`OpenAIChatMessage`](../pkg/aiusechat/openai/openai-backend.go:46-52) internal format and cleaned before transmission ([see lines 485-494](../pkg/aiusechat/openai/openai-backend.go:485-494)).

## 1. Message Objects (User/Assistant)

User and assistant messages sent as [`OpenAIMessage`](../pkg/aiusechat/openai/openai-backend.go:54-57):

```json
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Hello, analyze this image"
},
{
"type": "input_image",
"image_url": "data:image/png;base64,iVBORw0KG..."
}
]
}
```

**Key Points:**
- `role`: Always `"user"` or `"assistant"`
- `content`: **Always an array** of content blocks (never a plain string)

### Content Block Types

#### Text Block
```json
{
"type": "input_text",
"text": "message content here"
}
```

#### Image Block
```json
{
"type": "input_image",
"image_url": "data:image/png;base64,..."
}
```
- Can be a data URL or https:// URL
- `filename` field is **removed** during cleaning

#### PDF File Block
```json
{
"type": "input_file",
"file_data": "JVBERi0xLjQKJeLjz9M...",
"filename": "document.pdf"
}
```
- `file_data`: Base64-encoded PDF content

#### Function Call Block (in assistant messages)
```json
{
"type": "function_call",
"call_id": "call_abc123",
"name": "search_files",
"arguments": {"query": "test"}
}
```

## 2. Function Call Objects (Tool Invocations)

Tool calls from the model sent as [`OpenAIFunctionCallInput`](../pkg/aiusechat/openai/openai-backend.go:59-67):

```json
{
"type": "function_call",
"call_id": "call_abc123",
"name": "search_files",
"arguments": "{\"query\":\"test\",\"path\":\"./src\"}"
}
```

**Key Points:**
- `type`: Always `"function_call"`
- `call_id`: Unique identifier generated by model
- `name`: Function name to execute
- `arguments`: JSON-encoded string of parameters
- `status`: Optional (`"in_progress"`, `"completed"`, `"incomplete"`)
- Internal `toolusedata` field is **removed** during cleaning

## 3. Function Call Output Objects (Tool Results)

Tool execution results sent as [`OpenAIFunctionCallOutputInput`](../pkg/aiusechat/openai/openai-backend.go:69-75):

```json
{
"type": "function_call_output",
"call_id": "call_abc123",
"output": "Found 3 files matching query"
}
```

**Key Points:**
- `type`: Always `"function_call_output"`
- `call_id`: Must match the original function call's `call_id`
- `output`: Can be text, image array, or error object

### Output Value Types

#### Text Output
```json
{
"type": "function_call_output",
"call_id": "call_abc123",
"output": "Result text here"
}
```

#### Image Output
```json
{
"type": "function_call_output",
"call_id": "call_abc123",
"output": [
{
"type": "input_image",
"image_url": "data:image/png;base64,..."
}
]
}
```

#### Error Output
```json
{
"type": "function_call_output",
"call_id": "call_abc123",
"output": "{\"ok\":\"false\",\"error\":\"File not found\"}"
}
```
- Error output is a JSON-encoded string containing `ok` and `error` fields

## Complete Example

```json
{
"model": "gpt-4o",
"input": [
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "What files are in src/?"
}
]
},
{
"type": "function_call",
"call_id": "call_xyz789",
"name": "list_files",
"arguments": "{\"path\":\"src/\"}"
},
{
"type": "function_call_output",
"call_id": "call_xyz789",
"output": "main.go\nutil.go\nconfig.go"
},
{
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "The src/ directory contains 3 files: main.go, util.go, and config.go"
}
]
}
],
"stream": true,
"max_output_tokens": 4096
}
```

## Cleaning Process

Before transmission, internal fields are removed ([cleanup code](../pkg/aiusechat/openai/openai-backend.go:485-494)):

- **Messages**: `previewurl` field removed, `filename` removed from `input_image` blocks
- **Function Calls**: `toolusedata` field removed
- **Function Outputs**: Sent as-is (no cleaning needed)

This ensures the API receives only the fields it expects.
17 changes: 15 additions & 2 deletions frontend/builder/builder-apppanel.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ import { TabRpcClient } from "@/app/store/wshrpcutil";
import { BuilderAppPanelModel, type TabType } from "@/builder/store/builder-apppanel-model";
import { BuilderFocusManager } from "@/builder/store/builder-focusmanager";
import { BuilderCodeTab } from "@/builder/tabs/builder-codetab";
import { BuilderConfigDataTab } from "@/builder/tabs/builder-configdatatab";
import { BuilderFilesTab, DeleteFileModal, RenameFileModal } from "@/builder/tabs/builder-filestab";
import { BuilderPreviewTab } from "@/builder/tabs/builder-previewtab";
import { BuilderEnvTab } from "@/builder/tabs/builder-secrettab";
import { BuilderSecretTab } from "@/builder/tabs/builder-secrettab";
import { builderAppHasSelection } from "@/builder/utils/builder-focus-utils";
import { ErrorBoundary } from "@/element/errorboundary";
import { atoms } from "@/store/global";
Expand Down Expand Up @@ -310,6 +311,13 @@ const BuilderAppPanel = memo(() => {
isAppFocused={isAppFocused}
onClick={() => handleTabClick("code")}
/>
<TabButton
label="Config/Data"
tabType="configdata"
isActive={activeTab === "configdata"}
isAppFocused={isAppFocused}
onClick={() => handleTabClick("configdata")}
/>
<TabButton
label="Files"
tabType="files"
Expand Down Expand Up @@ -363,7 +371,12 @@ const BuilderAppPanel = memo(() => {
</div>
<div className="w-full h-full" style={{ display: activeTab === "secrets" ? "block" : "none" }}>
<ErrorBoundary>
<BuilderEnvTab />
<BuilderSecretTab />
</ErrorBoundary>
</div>
<div className="w-full h-full" style={{ display: activeTab === "configdata" ? "block" : "none" }}>
<ErrorBoundary>
<BuilderConfigDataTab />
</ErrorBoundary>
</div>
</div>
Expand Down
14 changes: 12 additions & 2 deletions frontend/builder/store/builder-apppanel-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { base64ToString, stringToBase64 } from "@/util/util";
import { atom, type Atom, type PrimitiveAtom } from "jotai";
import { debounce } from "throttle-debounce";

export type TabType = "preview" | "files" | "code" | "secrets";
export type TabType = "preview" | "files" | "code" | "secrets" | "configdata";

export type EnvVar = {
name: string;
Expand Down Expand Up @@ -121,6 +121,16 @@ export class BuilderAppPanelModel {
}
}

updateSecretBindings(newBindings: { [key: string]: string }) {
const currentStatus = globalStore.get(this.builderStatusAtom);
if (currentStatus) {
globalStore.set(this.builderStatusAtom, {
...currentStatus,
secretbindings: newBindings,
});
}
}

async loadEnvVars(builderId: string) {
try {
const rtInfo = await RpcApi.GetRTInfoCommand(TabRpcClient, {
Expand Down Expand Up @@ -215,7 +225,7 @@ export class BuilderAppPanelModel {
async switchBuilderApp() {
const builderId = globalStore.get(atoms.builderId);
try {
await RpcApi.StopBuilderCommand(TabRpcClient, builderId);
await RpcApi.DeleteBuilderCommand(TabRpcClient, builderId);
await new Promise((resolve) => setTimeout(resolve, 500));
await RpcApi.SetRTInfoCommand(TabRpcClient, {
oref: WOS.makeORef("builder", builderId),
Expand Down
Loading
Loading