Skip to content

@effect/ai-openrouter streamText fails when a toolkit is present #6128

@fabstorres

Description

@fabstorres

What version of Effect is running?

effect: 3.20.0, @effect/ai: 0.34.0, @effect/ai-openrouter: 0.9.0

What steps can reproduce the bug?

If the toolkit option is removed from LanguageModel.streamText, the program runs successfully.

The failure only occurs when a toolkit is present.

import { LanguageModel, Prompt, Tool, Toolkit } from "@effect/ai";
import { OpenRouterClient, OpenRouterLanguageModel } from "@effect/ai-openrouter";
import { FetchHttpClient } from "@effect/platform";
import { Config, Effect, Layer, Schema, Stream } from "effect";

const StepFun = OpenRouterLanguageModel.model("stepfun/step-3.5-flash:free");

const SafeUser = Schema.Struct({
  firstName: Schema.String,
  ageRange: Schema.Literal("18-24", "25-34", "35-44", "45+"),
});

const GetUser = Tool.make("GetUser", {
  description: "Get non-sensitive information about the user",
  success: SafeUser,
  failure: Schema.Never,
});

const GetUserTools = Toolkit.make(GetUser);

const GetUserHandlers = GetUserTools.toLayer({
  GetUser: () =>
    Effect.succeed({
      firstName: "John Doe",
      ageRange: "25-34",
    }),
});

const program = LanguageModel.streamText({
  prompt: Prompt.make("Use GetUser tool and tell me about the current user."),
  toolkit: GetUserTools,
}).pipe(Stream.runDrain, Effect.provide(StepFun));

const OpenRouter = OpenRouterClient.layerConfig({
  apiKey: Config.redacted("OPENROUTER_API_KEY"),
}).pipe(Layer.provide(FetchHttpClient.layer));

program.pipe(Effect.provide([OpenRouter, GetUserHandlers]), Effect.runPromise);

What is the expected behavior?

LanguageModel.streamText should work when a Toolkit is provided and allow the model to call tools during streaming without throwing a MalformedOutput error.

What do you see instead?

(FiberFailure) MalformedOutput: { "module": "OpenRouterClient", "method": "streamRequest", "description": undefined, "cause": (parseJson <-> (@effect/ai-openrouter/ChatStreamingResponseChunk (Encoded side) <-> @effect/ai-openrouter/ChatStreamingResponseChunk))

Relevant part of the cause:
["type"] is missing

Actual streamed tool call chunk:
[{"index":0,"function":{"arguments":"{}"}}]

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions