Skip to content

@effect/ai-amazon-bedrock: streaming token usage is always empty #6237

@kn0ll

Description

@kn0ll

What version of Effect is running?

3.21.2

What steps can reproduce the bug?

import * as LanguageModel from "@effect/ai/LanguageModel"
import * as AmazonBedrockClient from "@effect/ai-amazon-bedrock/AmazonBedrockClient"
import * as AmazonBedrockLanguageModel from "@effect/ai-amazon-bedrock/AmazonBedrockLanguageModel"
import * as NodeHttpClient from "@effect/platform-node/NodeHttpClient"
import { Effect, Layer, Redacted, Stream } from "effect"

const MainLive = AmazonBedrockLanguageModel.layer({
  model: "anthropic.claude-3-haiku-20240307-v1:0"
}).pipe(
  Layer.provide(AmazonBedrockClient.layer({
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: Redacted.make(process.env.AWS_SECRET_ACCESS_KEY!),
    region: "us-east-1"
  })),
  Layer.provide(NodeHttpClient.layerUndici)
)

const program = LanguageModel.streamText({ prompt: "Say hello in one sentence." }).pipe(
  Stream.runFold(undefined as any, (_acc, part) => part.type === "finish" ? part : _acc),
  Effect.andThen((finish) => {
    console.log(`inputTokens: ${finish.usage.inputTokens}, outputTokens: ${finish.usage.outputTokens}`)
  }),
  Effect.scoped,
  Effect.provide(MainLive)
)

Effect.runPromise(program)

What is the expected behavior?

The finish part should contain real token counts:

inputTokens: number, outputTokens: number

What do you see instead?

Token counts are always undefined:

inputTokens: undefined, outputTokens: undefined

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions