ThinkWatch v3.2.0

Images

• ghcr.io/thinkwatchproject/think-watch-server:3.2.0
• ghcr.io/thinkwatchproject/think-watch-web:3.2.0

Helm

helm install think-watch ./think-watch-3.2.0.tgz \
  --namespace thinkwatch --create-namespace

---

Redis Cluster and Redis over TLS now work, so managed Redis services can be
used. Usage limits work as documented: token limits refuse, an API key's
limits count on that key, and a refused request counts against nothing.
Server instances can start together against one database, a restart no
longer clears captured bodies older than 30 days, and the Helm network
policy allows the ports the database URLs name. The thinkwatch-core crates
stay at v0.62.0.

Read before upgrading

• Rate-limit windows start empty. Rate-limit counters move to new Redis
  keys (one hash per counter, tagged so that Redis Cluster can run them), and
  the counts from before the upgrade are not carried over: every window starts
  empty and fills from the first request after the upgrade. The old keys
  expire by themselves within two window lengths. Users' budget counters are
  kept; API keys' are not (see below).
• Route health starts fresh. A route's samples, circuit breaker and
  lifetime request count move to new keys for the same reason, so every route
  starts closed with nothing counted. The old lifetime counters never expire;
  redis-cli --scan --pattern 'route_health:[0-9a-f]*' | xargs redis-cli del
  removes them (the new keys start route_health:{).
• An API key's own budgets start again. 3.1.0 counted a key's budget on
  its owner's counter (budget:user:<user id>:…); 3.2.0 counts it on the
  key's own (budget:api_key_lineage:<lineage id>:…), which starts at zero.
  A key with a monthly budget of its own can spend all of it again in the
  rest of the month.
• Limits are looser while the rollout runs. Pods of 3.1.0 and 3.2.0
  count rate limits and route health on different keys, so each sees only
  its own version's requests: limits let more through, and circuit breakers
  can disagree, until the last 3.1.0 pod is gone.
• Token limits start refusing. A tokens rate limit never refused a
  request before. It now does once its window is full, so a deployment with
  token limits will see 429s where it saw none.
• A key's limits no longer replace its owner's. A rate limit or budget set
  on an API key used to take the place of the owner's limit for the same
  window or period. Both now apply, each on its own counter: a key's limits can
  narrow what its owner may do through that key, never widen it. A key given a
  higher limit than its owner to give it more room needs the owner's limit
  raised instead.
• rediss:// connects over TLS. 3.1.0 connected to a rediss:// URL
  over plain TCP. A rediss:// URL pointing at a port without TLS now fails
  at start: point it at the TLS port, or write redis://.
  • The certificate must name the host in a subjectAltName. One that names
    it only in its CN, which redis-cli accepts, is refused.
  • Cluster nodes that announce IP addresses their certificates don't name
    can't be reached.
• budget_unavailable in dashboards. With Redis down and
  security.rate_limit_fail_closed on, a request that has a budget is
  refused as budget_unavailable, not rate_limiter_unavailable: budgets
  are checked first.
• Upgrade with an ordinary rollout. Earlier versions don't take the lock
  that now makes instances set up the schema one at a time, so don't restart
  instances of the old version while the first one of this version starts.
• No transaction-mode pooler in front of Postgres. Schema setup now holds a
  Postgres session-level advisory lock, which a pooler in transaction mode
  (PgBouncer pool_mode = transaction) can leave held, and every later start
  then waits for it: point DATABASE_URL at Postgres itself or at a pooler in
  session mode (the Helm chart's README has the details).

Fixed

• Token limits refuse requests. A tokens rate limit never refused
  anything, and stopped counting once a request would have taken it past its
  limit. A request is now refused once the window's recorded usage reaches the
  limit, and every request's tokens are recorded after it, even past the limit
  — a window can overshoot by what was in flight when it filled.
• Several request limits at once. With two or more requests limits on a
  user (per minute and per hour, say), every request that passed was counted
  twice, and only one of the limits could refuse; with
  security.rate_limit_fail_closed on, every request was refused as
  rate_limiter_unavailable.
• An API key's limits count on that key. They were counted on its owner's
  counter, which every key of the owner shared, and the usage the console
  reads for a key (/api/admin/limits/api_key/{id}/usage) was always 0. Each
  key now has counters of its own, rate limits and budgets alike, for the
  gateway and the MCP gateway, and its usage shows what it used.
• A refused request counts against nothing. A request refused by a spent
  budget, or by one rate limit after another had passed, was still counted
  against the request limits. Budgets are now checked first and every rate
  limit in one step, so a refused request leaves every counter as it was.
• Retry-After says when to retry. A 429 from the gateway's own limits
  said Retry-After: 30 whatever the limit. It now gives the seconds until the
  window has room for another request, or until a spent budget's period ends
  (the next midnight, Monday or 1st of the month, UTC). A spent budget also
  sends x-should-retry: false, so the OpenAI and Anthropic SDKs don't retry it
  by themselves. The body stays in the caller's API format.
• Redis Cluster. The rate-limit, route-health and quota scripts touched
  keys of several hash slots, which a Redis Cluster refuses: on a cluster, rate
  limits silently stopped applying (or refused every request with
  security.rate_limit_fail_closed), circuit breakers never tripped, and cache
  invalidation reached one node only. Every key a script touches now shares a
  hash tag, pattern deletes scan every node, and the Helm chart's README
  describes a redis-cluster:// URL.
• Redis over TLS. The server was built without TLS for Redis: a
  rediss:// URL was used as plain TCP, so against a Redis that requires
  TLS — ElastiCache with in-transit encryption, Upstash, Azure Cache for
  Redis and Redis Cloud among them — the server did not start
  (Failed to connect to Redis: Protocol Error: Expected string.).
  rediss:// and rediss-cluster:// URLs now connect over TLS and check
  the certificate against the system's CAs, as upstream HTTPS does. For a
  Redis whose certificate a private CA signed, REDIS_CA_CERT names a PEM
  file with that CA, which is then trusted alone; the Helm chart sets it
  from a Secret given in redis.caSecret. The chart's README describes
  both.
• Several instances starting at once. Server instances starting together
  against one database — a Helm replicaCount above 1, a rolling upgrade, an
  autoscaler adding pods — applied the schema side by side, and all but one
  could exit with Database migration failed: apply db/schema.sql: … deadlock detected (on an empty database: duplicate key value violates unique constraint "pg_extension_name_index"). With ClickHouse, the rollups that an
  instance fills from the logs when it finds them empty (cost_rollup_hourly,
  provider_health_5m, mcp_server_call_counts) could be filled by each of
  them, counting every request once per instance on the cost pages, the
  dashboard and the MCP server list. Instances now set up Postgres and
  ClickHouse one at a time, under Postgres advisory locks: the others wait,
  logging Another instance is setting up the database schema; waiting for it to finish, then find it done. An instance that dies holding a lock releases
  it with its connection. The Helm chart's startup probe allows a pod 125 s
  to start instead of 35 s, set in startupProbe (the chart's README says
  when to raise it), and an attempt to reach a ClickHouse that doesn't answer
  gives up after 5 s instead of the system's TCP timeout.
• Captured bodies kept as long as configured. With ClickHouse and
  audit.body_retention_days above 30, every server start could clear the
  captured request and response bodies older than 30 days
  (gateway_logs.request_body / response_body, mcp_logs.tool_arguments
  / tool_result; the rows themselves stayed). The start-up table setup set
  those columns' TTL to 30 days each time, and ClickHouse applies a TTL to
  the data already stored as soon as it is set, before the server put the
  configured TTL back a moment later. The setup now gives these columns a
  TTL only when it creates them, so a restart leaves the configured one in
  place. Bodies already cleared cannot be recovered. The log tables' own
  TTLs (data.retention_days_*) were not affected.
• Helm network policy and databases on other ports. With
  networkPolicy.enabled, the server could reach PostgreSQL only on 5432,
  Redis on 6379 and ClickHouse on 8123, whatever their externalUrl said,
  so a database on another port was blocked — Azure Cache for Redis over TLS
  (6380), a managed Postgres or a ClickHouse on a port of its own: the
  server could not start, or started without writing to ClickHouse. The
  allowed ports now follow postgres.externalUrl, redis.externalUrl and
  clickhouse.externalUrl: every port a URL names, and the client's default
  for its scheme where it names none. networkPolicy.extraEgress adds egress
  rules as written, for ports no URL names (Redis Cluster nodes announcing
  other ports, an upstream or MCP server on a port other than 443). The
  chart's README describes both. Port 9000, ClickHouse's native protocol,
  is no longer allowed: the server reaches ClickHouse over plain HTTP only
  (an http:// URL; HTTPS is not supported). An S3 endpoint on 9000
  (RustFS, MinIO) configured outside the chart needs a rule in
  networkPolicy.extraEgress.