SQLDesk 0.7.3 — Signal and Noise

Four gaps closed in what 0.7 shipped, three faults found on the way, and one idea running through all of it: say what matters, quiet what does not.

SQLDesk stops appearing in lag monitoring

• Every stream joined a consumer group it never committed under. sqldesk-stream-<id> was a live member with no position at all — exactly what a lag monitor exists to alarm on.
• The default is now no group the broker can see. It cannot literally be none: librdkafka refuses to build a consumer without a group.id. So one is set and never used — the consumer assigns itself every partition and commits nothing, and the broker never registers it.
• Nothing is lost by not joining. A lock already guaranteed one consumer per topic, so the group never balanced anything.
• Existing streams change on upgrade, so the alarms stop without anybody touching a setting.

How a topic is read, chosen per topic

Set on the Analyse panel, where a topic is enabled:

• Consumer group name — optional, for monitoring that ignores consumers by prefix. A name joins that group.
• Save position — off by default, and the page says in words that nothing is saved. On, offsets are committed after each flush, and only after it.
• Start from — latest, earliest, or where it left off. A setting rather than a prompt, because a stream starts when somebody opens a dashboard.

A hole at every hand-over

• Consumption is a chain of jobs, and every one sought to the end when it started — hand-overs included. So each hand-over dropped whatever arrived between two jobs: a small gap in the chart every few minutes.
• Positions are now carried across a hand-over, so the start setting applies only to a cold start. That is also what stops "earliest" replaying the whole topic at every hand-over.

The catalog can be searched, and told what to ignore

• Tables past the first 200 were unreachable. The page is sent the 200 most used and the limit was on the server, so the rest were not merely off the page but impossible to reach from it. A search now runs on the server over every table, by name and description.
• The page says when it is not showing everything — "Showing the 200 most used of 1,432".
• Hide a table. Off the catalog page and out of what MCP offers a model — hundreds of Mimir metrics are noise to both. Still queryable, and a re-harvest keeps it hidden.
• Show hidden, with a count, for when you need to find one.
• Filters: with a description, and missing one.
• An empty description shows the shape of a good one — what it is for, the grain, how it is partitioned, what the key looks like.
• Fixed on the way: the edit set the description from whatever request arrived, so hiding a table would have wiped what somebody had written about it.

Curate the catalog over MCP

• update_table describes or hides a table from whatever MCP client you already use. The first MCP tool that writes.
• A personal API key only, from somebody with the catalog permission. An OAuth token never reaches it — those carry mcp:read — and the tool is listed only to a caller who may use it.
• It will not replace a description a person wrote unless told to with overwrite. What it writes is tagged "via MCP" and left alone by harvests.

Verified dashboards

• A badge beside the title, with a required note on what was checked. Tap it for who, when and what — a tap, because the note has to be readable on a phone.
• It does not outlive what it vouched for. Edit the SQL behind a chart, or add or remove a widget, and it reads "verified, then changed". Moving a widget does not count.
• The same permission as confirming a query in the catalog — a curator's act, so it needs no edit right on the dashboard.

Images

ghcr.io/bot-netizen/sqldesk:0.7.3
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.3

linux/amd64 and linux/arm64.

Upgrading from 0.7.2

• Three migrations, each adding columns with defaults. All reversible.
• Streams change how they look to the broker. The sqldesk-stream-<id> groups stop appearing, which is the point. A topic that should be visible under a name needs one set on its page.
• Nothing else to configure.

Helm — the image tag has no v; leave it unset and the chart uses its own appVersion.

helm upgrade sqldesk ./charts/sqldesk --reuse-values

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.3

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml run --rm server manage db upgrade
docker compose -f compose.prod.yaml up -d

Verified before tagging

• CI green on every job, including the end-to-end suite.
• 2504 backend tests — run with a real Kafka broker, so the streams tests that usually skip all ran — and 717 frontend tests.
• Against a real broker, not a fake: an unnamed consumer never appears in the broker's group list and is never issued a member id; a named one is; a saved position resumes past what was already in the window; earliest reads the backlog; and a hand-over reads what arrived between the jobs.
• The upgrade path itself, on the published image: a database created by 0.7.2 upgraded to 0.7.3 through manage db upgrade; all three migrations then downgraded and upgraded again, cleanly.
• The published image against a real broker: a stream consumer read every message, was never issued a member id and never appeared in the broker's group list, and recorded where it got to for a hand-over.
• The published image through compose.prod.yaml: /ping and /setup at 200 with Content-Encoding: br, 14 periodic jobs registered, refresh_queries and cleanup_query_results clean on the worker, 75 data source types offered.
• Every new rule was mutation-tested — making every consumer join, dropping the hand-over, letting OAuth write, overwriting a person's description, wiping a description on hide, searching only the first 200: each fails its own tests.