Refactor: Unify special tool handling for parallel execution

[FL4TLiN3]

Summary

Special tools (think, readPdfFile, readImageFile) currently block parallel tool call execution. When an LLM returns multiple tool calls including a special tool, only the special tool is executed and others are dropped.

Background

• Past constraint: Tool results couldn't include base64-encoded PDF/image data directly
• Current state: Vercel AI SDK 5.0+ and Anthropic API (Feb 2025) now support PDF/images in tool results
• Perstack version: Using AI SDK v5.0.104, which includes this support

Current Implementation Issues

1. calling-tool.ts: Special tools (think, readPdfFile, readImageFile) are prioritized and return early, dropping other parallel tool calls
2. resolvingPdfFileLogic: Adds PDF to userMessage instead of toolMessage (legacy workaround)
3. createToolMessage: Type doesn't include FileInlinePart for PDFs
4. toolResultPartToCoreToolResultPart: No handling for fileInlinePart

Proposed Changes

1. Update message handling for PDF support

• Extend createToolMessage type to support FileInlinePart
• Update toolResultPartToCoreToolResultPart to convert fileInlinePart to media type
• Modify resolvingPdfFileLogic to include PDF in toolMessage (not userMessage)

2. Simplify tool execution flow

Before:

attemptCompletion → single execution (others ignored)
think → single execution (others ignored)
readPdfFile → single execution (others ignored)
readImageFile → single execution (others ignored)
MCP tools → parallel execution
delegate/interactive → sequential after MCP

After:

attemptCompletion → single execution (others ignored, completion means done)
All other MCP tools (think, readPdfFile, readImageFile, etc.) → parallel execution
delegate/interactive → sequential after MCP parallel execution

3. Consider state machine simplification

With unified tool result handling, ResolvingPdfFile, ResolvingImageFile, and ResolvingThought states may be consolidatable into ResolvingToolResult.

Tasks

• Extend createToolMessage type to include FileInlinePart
• Update toolResultPartToCoreToolResultPart for fileInlinePart
• Refactor resolvingPdfFileLogic to use toolMessage
• Modify calling-tool.ts to parallelize all MCP tools except attemptCompletion
• Update tests
• Evaluate state machine simplification (may be separate PR)

Related

• PR Add: Parallel tool call support #62 (Parallel tool call support)
• Vercel AI SDK PR #9733 (PDF/image support in tool results)

[github-actions]

I've analyzed issue #64 and the codebase. Here's a comprehensive overview of the current implementation and the proposed refactoring:

Current State

1. Special Tool Handling (packages/runtime/src/states/calling-tool.ts)

Currently, special tools (think, readPdfFile, readImageFile, attemptCompletion) are handled with separate resolve functions that return early, preventing parallel execution with other tools:

if (skillName === "@perstack/base") {
  if (toolName === "think") {
    return resolveThought(setting, checkpoint, { toolResult })
  }
  if (toolName === "attemptCompletion") {
    // ... handles completion
  }
  if (toolName === "readPdfFile") {
    return resolvePdfFile(setting, checkpoint, { toolResult })
  }
  if (toolName === "readImageFile") {
    return resolveImageFile(setting, checkpoint, { toolResult })
  }
}

2. PDF File Handling (packages/runtime/src/states/resolving-pdf-file.ts)

PDFs are currently added to a separate user message instead of the tool result:

return finishToolCall(setting, checkpoint, {
  newMessages: [
    createToolMessage([...]),  // Tool result with text
    createUserMessage(files),  // PDF in separate user message (legacy workaround)
  ],
})

3. Type Limitations (packages/core/src/schemas/message-part.ts)

The ToolResultPart interface only supports TextPart and ImageInlinePart, not FileInlinePart:

export interface ToolResultPart extends BasePart {
  type: "toolResultPart"
  toolCallId: string
  toolName: string
  contents: (TextPart | ImageInlinePart)[]  // ❌ Missing FileInlinePart
  isError?: boolean
}

Proposed Changes

1. Extend Type Support

Update ToolResultPart to include FileInlinePart:

contents: (TextPart | ImageInlinePart | FileInlinePart)[]

2. Update Message Creation (packages/runtime/src/messages/message.ts)

The createToolMessage function needs to accept FileInlinePart:

export function createToolMessage(
  contents: Array<
    Omit<ToolResultPart, "id" | "contents"> & {
      contents: Array<Omit<TextPart, "id"> | Omit<ImageInlinePart, "id"> | Omit<FileInlinePart, "id">>
    }
  >,
): ToolMessage

3. Refactor PDF/Image Handling

Modify resolvingPdfFileLogic to include files in the tool result instead of a separate user message:

return finishToolCall(setting, checkpoint, {
  newMessages: [
    createToolMessage([
      {
        type: "toolResultPart",
        toolCallId: id,
        toolName,
        contents: files,  // Include PDFs directly in tool result
      },
    ]),
  ],
})

4. Simplify Tool Execution Flow (calling-tool.ts)

Only attemptCompletion should block parallel execution:

if (skillName === "@perstack/base" && toolName === "attemptCompletion") {
  if (hasRemainingTodos(toolResult)) {
    return resolveToolResult(setting, checkpoint, { toolResult })
  }
  return attemptCompletion(setting, checkpoint, { toolResult })
}
// All other tools (including think, readPdfFile, readImageFile) use standard flow
return resolveToolResult(setting, checkpoint, { toolResult })

Benefits

1. Unified handling: All MCP tools (except attemptCompletion) can execute in parallel
2. Cleaner architecture: Eliminates special-case logic for think, readPdfFile, readImageFile
3. Better performance: More tools can run concurrently
4. Standards compliance: Uses modern AI SDK capabilities for file handling in tool results

Implementation Notes

• The toolResultPartToCoreToolResultPart function in message.ts already handles media types generically (line 250), so it should work with FileInlinePart once types are updated
• Consider whether ResolvingPdfFile, ResolvingImageFile, and ResolvingThought states can be consolidated into ResolvingToolResult
• Ensure all tests are updated to reflect the new message structure

This refactoring aligns with the capabilities introduced in Vercel AI SDK 5.0+ and will enable more efficient parallel tool execution.

---

🤖 Bot Activity

💭 I need to answer issue #64. Let me start by fetching the issue information to understand what the qu...
⚡ Exec: gh api repos/perstack-ai/perstack/issues/64/comments --jq
💭 I see there's only one comment which is from the bot itself (processing message). This means the que...
⚡ Exec: gh issue view 64
💭 Now I understand the issue. This is issue #64 which is about refactoring special tool handling to en...
⚡ Exec: gh api repos/perstack-ai/perstack/issues/64/comments
💭 I see - there's only one comment which is the bot's own processing message. This means the issue was...
⚡ Exec: find . -name calling-tool.ts
📖 Reading: ./packages/runtime/src/states/calling-tool.ts
💭 Good! I can see the current implementation in calling-tool.ts. The special tools (think, readPdfFile...
⚡ Exec: find . -name *.ts
📖 Reading: ./packages/runtime/src/states/resolving-pdf-file.ts
💭 Perfect! I can see the issue clearly now. In `resolvingPdfFileLogic`, the PDF file is being added to...
⚡ Exec: find . -path */messages/message.ts
📖 Reading: ./packages/runtime/src/messages/message.ts
💭 Excellent! Now I can see the full picture:

1. **`createToolMessage` type**: Currently only accepts ...
⚡ Exec: find ./packages/core -name *.ts
📖 Reading: ./packages/core/src/schemas/message-part.ts
💭 Perfect! Now I can see the issue clearly. In the `ToolResultPart` interface (line 145-154), the `con...
⚡ Exec: find . -name *.ts
📖 Reading: ./packages/runtime/src/states/generating-tool-call.ts
💭 I can see that in `generating-tool-call.ts`, the code currently only handles the first tool call (`t...
⚡ Exec: gh pr view 62
💭 Great! Now I understand the full context. PR #62 implements parallel tool call support, but it still...
✨ Generating answer...
✅ Done
---