Skip to content

ThinkWatch Core 0.68.0

Choose a tag to compare

@github-actions github-actions released this 10 Oct 06:12
· 26 commits to main since this release
0955f99

This release counts the network traffic between the gateway and its upstreams, lets a running request's content be watched live, records where each security hit was and what was done about it, and fixes Codex on a ChatGPT account (its finished turns were recorded as cancelled and its longer sessions had no usage) and upstreams that write "error": null in every streamed chunk.

Upgrade notes

  • The control-plane protocol version (CONTROL_API_VERSION) goes from 43 to 44. ThinkWatch Lite connects only to a core with the same protocol version: ThinkWatch Lite 2026.10.11 includes 0.67.1 (protocol 43) and does not connect to 0.68.0. A server used with it stays on 0.67.1 until the app is updated to a release that includes 0.68.0; sudo twcore upgrade --version 0.67.1 --restart switches a server back.
  • The request store's schema goes to 27: the request history, stored payloads and the security log are cleared on the first start. Recorded requests have no traffic figures, and recorded security events have no detail.
  • No configuration key changes and no new message codes.
  • Changed in the protocol:
    • HistoryRow loses bytes and gains sent_bytes, received_bytes and egress. RequestFinished carries sent_bytes and received_bytes instead of bytes; RequestFailed and RequestCancelled carry them as optional (absent when no attempt reached an upstream). RequestRouted and in-flight requests carry egress; each attempt in the routing view carries proxy. ReplayOriginal.bytes becomes received_bytes.
    • Summary, CostBucket and CostBucketGroup gain sent_bytes and received_bytes; CostBucket also gains its token counts and the first-token ttft_p50_ms, ttft_p95_ms and ttft_samples of its bucket. CostDim gains egress (the direct group is named "").
    • New endpoints GET /latency/client (first-token latency by key) and GET /request/{id}/live (live content, an event stream; 404 control.request_not_running once the request has ended).
    • RequestDetail gains heads, upstream_request_body and upstream_response_body. response_body is now what the client received; for a converted request the upstream's own answer is in upstream_response_body.
    • SecurityEventView gains session, sent_model, direction, locations, more_locations, rule_snapshot and outcome_detail. The live SecretsFound items, ContentMatched and ToolCallFlagged carry the same detail as detail.

Traffic. Each request records how much it sent to upstreams and received from them: request and response bodies as they crossed the network, after the gateway's own compression and before any decoding, summed over every attempt of a failover. It also records the proxy it went out through (egress), or none for a direct connection. The totals are in the summary, per time bucket, and grouped by model, upstream, key or proxy.

Live content. GET /request/{id}/live streams a running request's request and response — request and status lines, headers and bodies, the client's side and each upstream attempt's side — first everything so far, then each piece as it passes. Secrets are replaced and credentials masked exactly as in what is written to disk. Finished requests keep the headers, and the upstream's side of the bodies when it differs from the client's (a converted format, or a request with parts removed for the upstream).

First-token latency over time. Every time bucket carries the median and 95th percentile of its first-token times, and first-token latency can be grouped by key.

Security log detail. Every security event now says where each hit was and what exactly was done:

  • Every place the rule matched, in order, up to 50 per event (with a count of the rest): which part of the request or answer (system prompt, a message with its index and role, a tool call or tool result with the tool's name, the answer text), its JSON path in the client's own format, and up to 80 characters of context on either side.
  • The rule as it was when it matched: a built-in rule's id and name with the core version that evaluated it, or a custom rule's pattern and how it matches, so the entry stays readable after the rule is changed or deleted.
  • What was done: the placeholders a replaced credential was sent as; for a cut tool call, the tool, its arguments and the sentence the client received; for a refused request, the refusal; for removed characters, how many segments were removed.
  • The session the request belonged to and the model that was sent.

Contexts and matches are redacted exactly like stored payloads, so no secret is written in clear. The detail is kept in the request log and expires with it. Changes to the guard settings are not written to the security log.

Codex on a ChatGPT account.

  • Codex stops reading as soon as it has response.completed and closes the connection, without waiting for the stream to end; the ChatGPT backend often ends its stream a little later. Core reported the ending only when the upstream stream ended, so a client closing first was recorded as a cancellation, and most Codex turns appeared cancelled. Once the answer's last frame has reached the client, a request whose client then leaves is recorded as finished, with its usage and cost.
  • The ChatGPT backend's usage carries a per-item attribution that grows with the conversation. Usage objects were only read up to 8 KB, so longer sessions recorded no tokens and no cost. They are now read up to 1 MiB.

"error": null in a stream. Some OpenAI-compatible relays and Gemini-format upstreams write "error": null in every streamed chunk. Core took such a chunk for an error reported by the upstream: the request failed at its first chunk and moved to the next upstream, a converted stream ended with an error and lost its text, reply plugins left those chunks untouched, and a Gemini tool result carrying "error": null was marked as failed. A null error is now ignored everywhere. An error that carries no message is recorded with a fixed sentence instead of the chunk it came in, which could hold answer text.

Downloads

Platform Binary Archive for server installation
Linux, x86_64 twcore-x86_64-unknown-linux-gnu twcore-x86_64-unknown-linux-gnu.tar.gz
Linux, aarch64 twcore-aarch64-unknown-linux-gnu twcore-aarch64-unknown-linux-gnu.tar.gz
macOS, Apple silicon twcore-aarch64-apple-darwin —
Windows, x64 twcore-x86_64-pc-windows-msvc.exe —
Windows, ARM64 twcore-aarch64-pc-windows-msvc.exe —

Each file is published with a .sha256 file beside it. A Linux archive contains twcore, the systemd unit twcore.service and LICENSE. ThinkWatch Lite includes its own copy of twcore; the files here are for running core separately, such as on a server.

Server installation

On Linux (x86_64 or aarch64), the install script sets up twcore as a systemd service. This installs 0.68.0:

curl -fsSL https://raw.githubusercontent.com/ThinkWatchProject/ThinkWatch-Core/main/scripts/install.sh | sudo sh -s -- --version 0.68.0

An installation made with the script switches to 0.68.0 with:

sudo twcore upgrade --version 0.68.0 --restart

Configuration, the remote control port and connecting ThinkWatch Lite are described in docs/server.md.

Verifying a download

A .sha256 file holds the SHA-256 of the file followed by its name. With both files in the current directory, on Linux:

sha256sum -c twcore-x86_64-unknown-linux-gnu.tar.gz.sha256

On macOS:

shasum -a 256 -c twcore-aarch64-apple-darwin.sha256

On Windows, in PowerShell, the following prints True when the binary matches:

(Get-FileHash .\twcore-x86_64-pc-windows-msvc.exe).Hash -eq (Get-Content .\twcore-x86_64-pc-windows-msvc.exe.sha256).Split()[0]

The install script and twcore upgrade check the SHA-256 themselves.

What's Changed

  • Codex on a ChatGPT account: finished turns are no longer cancelled, and their usage is read by @fylorn in #310
  • Traffic between the gateway and upstreams, live request content, security log detail by @fylorn in #311
  • v0.68.0, and a null error in a stream chunk is not an error by @fylorn in #312

Full Changelog: v0.67.1...v0.68.0