ThinkWatch v3.2.1

Images

• ghcr.io/thinkwatchproject/think-watch-server:3.2.1
• ghcr.io/thinkwatchproject/think-watch-web:3.2.1

Helm

helm install think-watch ./think-watch-3.2.1.tgz \
  --namespace thinkwatch --create-namespace

---

The Helm chart and the Compose file now pull the published images: they
named ghcr.io/thinkwatch/…, which this project does not publish to, so a
default install could not pull. A TOTP code now logs in once, and log rows
are no longer lost while ClickHouse is unreachable. Five reads from
ClickHouse returned wrong or missing data and now return what is stored:
the SLO latency percentiles, a route's latency history, the times on traces
and log bodies, a trace's app events, and the log lists for a range written
with a T; and a failed read is now an error or a logged warning instead of
an empty answer. The server also reads from ClickHouse 26.9 and later, the
console shows its own version again, and the dependencies move to current
releases, several of them for security advisories. The thinkwatch-core
crates move from v0.62.0 to v0.65.0, which makes outbound redaction and the
content filter faster without changing what they find.

No setting, schema or API route changes, and instances of 3.2.0 and 3.2.1
can run side by side during a rollout.

Read before upgrading

• Image names in your own files. A values file or --set flag of your
  own that names ghcr.io/thinkwatch/think-watch-server or …-web keeps
  overriding the chart's new default, and a copy of
  deploy/docker-compose.yml keeps the old name: change them to
  ghcr.io/thinkwatchproject/…. A pinned image tag has no v: 3.2.1,
  not v3.2.1.

Fixed

• Image names. The Helm chart (image.server.repository,
  image.web.repository) and deploy/docker-compose.yml named
  ghcr.io/thinkwatch/think-watch-server and …-web, which this project
  does not publish to and which refuses anonymous pulls, so helm install
  with the default values and docker compose pull failed. Releases are
  published to ghcr.io/thinkwatchproject/think-watch-server and …-web,
  and both files now name them. The tag examples in the chart's README and
  values-production.yaml.example wrote v0.2.0; published tags have no
  v, and the examples now say 3.2.1.
• SLO latency percentiles. GET /api/admin/slo reported p50, p95 and
  p99 latencies next to zero: the percentiles kept the column's integer
  type and were read as floating-point numbers. They now give the
  latencies in milliseconds.
• Route latency history. The latency sparkline on the routing page
  (GET /api/admin/models/{model_id}/route-history) was always empty: the
  query's column types differed from the ones the server read, and the
  failure fell back to no data. It now shows each minute's p50 and p95, and
  a gap for a minute whose requests all failed before a latency was
  recorded.
• Times on traces and log bodies. The trace view and the gateway and
  MCP log body views put the month name where the minutes belong
  (2026-10-06T04:October:52), because %M in ClickHouse's
  formatDateTime is the month name. They now read 04:31:52.
• App events on traces. A trace never listed app log events: the query
  that finds them compared a text column with a time, ClickHouse refused
  it, and the refusal showed as no events.
• ClickHouse 26.9 and later. ClickHouse 26.9 compresses query results
  with ZSTD unless the client names a method, and the server's client
  reads only LZ4, so every read failed with incorrect magic number. The
  server now names LZ4 on every query, which every ClickHouse version
  supports. Query results are also checked against the types the server
  reads them into, so a query whose types drift fails instead of returning
  garbage; every query was checked against ClickHouse 26.8 and 26.9.
• Log rows kept through a ClickHouse outage. The log writer is meant to
  keep a batch that ClickHouse did not take and retry it on the next tick,
  up to 1000 entries per table. A batch that failed while being sent —
  ClickHouse restarting or unreachable, the usual case — had already been
  emptied, so its gateway, MCP, audit, access or app log rows were lost
  while the server logged retaining N entries for retry. They are now
  kept and retried. A batch that ClickHouse stored but whose answer never
  arrived is written a second time.
• Time ranges on the log lists. On the gateway, MCP, audit, app and
  access log lists, a from written YYYY-MM-DDTHH:MM:SS, the form the
  API's error message suggests, dropped every row of that day from
  items while total still counted them: the list compared the times
  as text. The console sends YYYY-MM-DD HH:MM:SS and was not affected.
• ClickHouse failures no longer read as no data. The trace view, the
  usage-license meter (/api/admin/usage-license) and the per-period series
  and previous-period totals of the usage and cost statistics turned a
  failed ClickHouse query into an empty or zero answer, with nothing in the
  log. They now answer 500 and log the error. The dashboard's request and
  active-key tiles and the call counts on the MCP server list still answer
  without the part that failed, so the page loads, and now log a warning
  naming the query.
• The console's version. The sidebar of the web console showed
  v0.0.0 in every released web image, because the image build could not
  see the version file it reads. It now shows the release.

Security

• A TOTP code logs in once. A code accepted at login could log in again,
  with the password, for as long as it stayed valid: up to 90 seconds with
  the step of clock drift allowed either side, so a code read over a
  shoulder or a shared screen was enough. The last time step accepted for
  each user is now kept in Redis (totp_last_step:<user id>, for three
  minutes), and a code whose step is not later is refused, the code that
  turned TOTP on included. Logging in twice within one 30-second step needs
  the next code; one step of clock drift is still accepted. A refused
  replay is audited as auth.totp_failed with "replayed": true.
  Instances of 3.2.0 do not check, so this holds once the rollout is done.
• rand 0.10.3 (RUSTSEC-2026-0097)
  and xxhash-rust 0.8.19
  (GHSA-6g2r-675j-hx59).
  Neither affects how ThinkWatch uses the crate: it installs no log
  logger that draws random numbers, and it hashes with xxh3's default
  secret only.
• The web console's build tools: brace-expansion 2.1.7 and 5.0.12, and
  ip-address 10.7.3 (Dependabot alerts on dependencies of eslint and the
  shadcn CLI). None of them is part of the console that is served.

Changed

• thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard)
  v0.62.0 → v0.65.0. Secret scanning and the content filter do much less
  work per request: they search bytes directly, skip plain ASCII where no
  rule can match it, and lowercase text in runs. Core tests each fast path
  against the code it replaced on randomly generated text, so they find
  exactly what they found before.
• The bundled ClickHouse (Helm clickhouse.image, deploy/docker-compose.yml)
  moves from 26.3 to 26.8, both long-term-support releases. An upgrade
  restarts it on 26.8 with its data. ClickHouse does not promise that an
  older version reads what a newer one wrote, so to roll back to 3.2.0
  later, keep clickhouse.image at 26.8.
• Dependencies move to current releases, among them sqlx 0.9, clickhouse
  0.15, tower-http 0.7, utoipa 6, jsonwebtoken 11, argon2 0.6, aes-gcm 0.11,
  p256 0.14 and totp-rs 6. Password hashes, encrypted secrets, issued
  tokens, request signatures and TOTP codes from 3.2.0 keep verifying, and
  what 3.2.1 writes verifies on 3.2.0; known-answer tests cover each.
• Tests now hold two things that could otherwise break unnoticed: a login
  for an address with no account still spends the time of a password check
  (so its timing does not tell which addresses have accounts), and a column
  added to a ClickHouse log table comes with a default (without one, the
  previous release's inserts into that table fail during a rolling upgrade
  or after a rollback).
• The server image is built with Rust 1.99, and the web image serves from
  nginx 1.31 (was 1.29), built with Node 24.21 and pnpm 12. Building from
  source needs Rust 1.94.1 or newer (was 1.85).