Skip to content

v1.40.0

Choose a tag to compare

@myzie myzie released this 19 Oct 01:19
· 17 commits to main since this release
6175785

Highlights

Streaming calls now recover from a provider that hangs. Before this release,
a stream gave up on the first timeout even though the caller's context was
still live, so one Anthropic request that hung for five minutes failed the
whole turn. An attempt that times out is now abandoned and retried like any
other transport error.

Streams are also bounded by two new limits in place of the 300-second total
client timeout, which cut off healthy streams that ran past five minutes. A
first-event limit abandons and retries an attempt that has sent nothing yet:
one minute on Anthropic and OpenAI, whose streams open with an event before
the model starts work, and five minutes on Chat Completions endpoints. It is
off for Google, because Gemini can send nothing, not even headers, for several
minutes while it thinks. An idle limit of five minutes ends a stream that goes
quiet partway through. Set either with WithStreamTimeouts on the provider.
http.Client.Timeout still applies to Generate.

A turn that ends because the provider timed out is now reported as
provider_error or stream_interrupted, not deadline. deadline means the
caller's own context or ResponseTimeout expired.

Most code needs no changes. A custom provider built on
providers.NewRetryingStreamIterator must update its StreamFactory to take
the attempt's context.Context and make its request with it.

Pull requests

  • fix(providers): retry stream timeouts and bound streams by first-event and idle limits (#341)

Changelog

Added

  • Stream timeouts. WithStreamTimeouts(firstEvent, idle) on the Anthropic,
    OpenAI, OpenAI Completions and Google providers. An attempt with no event by
    firstEvent is retried (Anthropic and OpenAI default 1m, Completions 5m,
    Google off); a stream quiet for idle (5m) ends.

Changed

  • Streams ignore http.Client.Timeout. It covered reading the body, so a
    healthy stream past 5 minutes was cut off; it now applies to Generate only.
  • providers.StreamFactory takes the attempt's context.Context. Custom
    providers must make their request with it so a timed-out attempt is cancelled.

Fixed

  • Streams retry a timed-out attempt while the caller's context is live,
    instead of failing on the first attempt. This covers the first-event limit
    and transport timeouts such as http.Transport.ResponseHeaderTimeout.
  • A provider timeout is no longer reported as the turn's deadline.
    TurnReasonDeadline now requires the invocation's context to have expired;
    otherwise it is provider_error or stream_interrupted.