ThinkWatch v3.10.0

Images

• ghcr.io/thinkwatchproject/think-watch-server:3.10.0
• ghcr.io/thinkwatchproject/think-watch-web:3.10.0

Helm

helm install think-watch ./think-watch-3.10.0.tgz \
  --namespace thinkwatch --create-namespace

---

Upstreams that fail for the moment before any of the answer reached the
client are now tried again instead of the error going straight back, and
a model can set the reasoning effort its requests carry, with the
gateway adapting when an upstream refuses a level. The thinkwatch-core
crates move from v0.81.0 to v0.83.1.

One new setting (gateway.retry_wait_secs), one new column
(models.reasoning_effort, added on startup) and one new metric; no
route or setting is removed.

Read before upgrading

• models gains a nullable column, reasoning_effort. The schema
  applied at startup adds it (ADD COLUMN IF NOT EXISTS); existing
  models keep it empty and their requests go out unchanged.
• A reasoning level an upstream refuses is changed and sent again, for
  every model. This applies whether or not the model has a reasoning
  setting and cannot be turned off: when an upstream answers 400 because
  of the reasoning level a request carries, the request goes out once
  more with a level the upstream accepts, or without one, and the change
  is remembered for that provider and model for 7 days. The level sent
  can differ from the one the caller asked for, in either direction.
• A request whose upstreams fail for the moment can take up to two
  minutes to fail. When every route of a model failed with an overload
  or a server error, or could not be reached, before any of the answer
  reached the client, the gateway now waits and sends the request again
  instead of returning the error at once, for up to
  gateway.retry_wait_secs (120 seconds by default) after the request
  arrived. Clients and load balancers with a shorter timeout than that may
  give up first. Set gateway.retry_wait_secs to 0 to keep the earlier
  behaviour.
• A whole answer that runs into the request timeout is now an error
  with a log row. A non-streaming request's upstream attempt is cut
  2 seconds before the gateway's 120-second request timeout (counted from
  when the request reached the gateway port), and the client gets the
  last upstream error, or a 504 ProviderTimeout when there was none,
  with its gateway_logs row, instead of an empty 408. A retry starts
  only when its wait and another attempt as long as the last one fit
  before then.
• A stream's opening events arrive with its first content. While a
  stream can still be sent again, what the upstream sends before any
  content (Anthropic message_start and ping, Responses
  response.created and response.in_progress, a Chat chunk with only
  the role) is held back and goes out together with the first content.
  While it is held, and while the gateway waits to send again, the client
  receives an SSE comment (: keep-alive) after every 15 seconds without
  data — except Gemini clients, which never receive one. The status is
  200 as before.

Added

• A model can set its reasoning (models.reasoning_effort, the
  Reasoning control in the model editor): off takes reasoning out of
  requests that have it on, and a level (low, medium, high,
  xhigh, max) changes the effort of requests whose reasoning is on.
  Requests without reasoning, such as title generation, go out
  unchanged. The setting is written the way each hop's upstream format
  writes it: with off, OpenAI Chat and Responses upstreams receive
  none, Gemini 2.5 Flash a thinking budget of 0, and Anthropic and
  Bedrock upstreams no thinking; Gemini models that cannot stop thinking
  (2.5 Pro, Gemini 3) get no thinking configuration. The response cache
  keys on the request together with the setting.
• A reasoning level the upstream refuses is sent again once with one it
  accepts. On a 400 about the reasoning level, the request goes once
  more to the same route with the level the error names (the lowest one
  it lists when none was refused), or without one when it names none.
  This is not counted as a retry, though its time falls within the retry
  window. When that is answered, the choice is remembered in Redis per
  provider, upstream model and request format for 7 days, so later
  requests send what worked straight away; a provider's memory is
  cleared when its URL or credentials change, a route relearns its
  protocol, or the provider is deleted. A hop whose reasoning level was
  changed records "effort": {asked, sent, reason} on its
  gateway_logs row.
• Transient upstream failures are retried. When every candidate route
  has been tried and the last one failed for the moment — 500, 502, 503,
  504 or 529, a connection that could not be made or broke before the
  answer's head, a stream that broke off or reported an overload or a
  server error (overloaded_error, api_error, server_error,
  server_is_overloaded, slow_down, Gemini UNAVAILABLE) before any
  content — and none of the answer reached the client, the request waits
  2, 4, 8, then 16 seconds (each give or take 20%), or as long as the
  upstream said in retry-after-ms or Retry-After (a zero wait is
  ignored, and a shorter one than 250 ms is taken as 250 ms), and goes out again
  to the routes that failed that way. A model with one route, and a
  stream, goes back to the same route. Each attempt is picked as before:
  the strategy, the circuit breaker, the route caps, session affinity
  and the provider holding a conversation's encrypted history all apply,
  and each failed attempt counts against its route's breaker as it did.
  Retries stop when the next wait would end past the deadline; an attempt
  already under way is not cut short. Rate limits and spent quota (429),
  refused credentials (401, 403), any other 4xx, timeouts, the gateway's
  own limits, budgets and policies, and a client that left are never
  retried, and neither is a stream after part of its answer went out.
• gateway.retry_wait_secs (Settings → Gateway → Routing strategy):
  how long after a request arrives it may still be sent again, 0 to 600
  seconds, default 120; 0 turns retries off.
• gateway_provider_retry_total{provider}: attempts sent again after
  a transient failure, by the provider they went to, next to
  gateway_provider_fallback_total.
• The gateway_logs row of a request that was sent again records
  "retries": n in its detail, whether it then succeeded or failed.

Changed

• A provider whose base URL cannot be made into a request fails as
  ProviderError instead of NetworkError, and is not retried.
• An error a stream reports as OpenAI's server_error is logged as the
  upstream's 500 (ProviderHttpError), as the same error answered with
  HTTP 500 is, instead of 502 (ProviderError).