SQLDesk 0.7.0 — Kafka Streams

Kafka topics you can query, with the same editor, the same visualizations and the same dashboards as everything else — and nothing new switched on for anybody who does not ask for it.

Streams

• A cluster is a data source; its topics are its tables. Add it the way you add Postgres. Which topics may be queried is chosen separately, because that is the access decision: everybody who can reach the cluster can query whatever is enabled on it.
• A streaming query is a query. Same editor, same parameters, same visualizations, same dashboards. Execute says Start streaming, and no result is ever stored — a window is what a consumer saw while somebody was watching, and writing it down would be making a copy of the one thing whose point is that it is current.
• Analyse a topic before enabling it. It reads the last few hundred messages and says what the columns are, how many could not be parsed, how fast they arrive, how much history that buys, and whether it will be sampled. None of that can be guessed from a topic's name.
• A row budget and an events-per-second ceiling per topic. Over the ceiling, events are kept one in N by a hash of the message key — so per-key counts scale back up honestly — and anything drawn from a sampled stream says so.
• A slot is a topic, not a viewer. Five people watching orders share one consumer and one window and take one slot between them, which makes a team round one dashboard during an incident nearly free. When there is no room, the editor says so beside the button that will not start.
• A topic is consumed only while somebody is watching, and stops about a minute after the last person looks away. The window is kept for ten minutes, so coming back resumes in seconds.

Streaming is a kind of query, not a section of the application

• No broker's name in the navigation. Streaming queries are a half of Queries; streaming dashboards a half of Dashboards. The two halves never overlap, so nothing is filed in two places.
• Which topics may be queried is a tab beside Data Sources, because it configures a connection.
• A dashboard declares its kind when you make it, carries a streaming chip when it is open, and shows a band that turns green while its topics are being consumed. Live keeps its old meaning: refreshing right now, which an ordinary dashboard can also be.
• A folder holds both kinds, on purpose — it is a place, not a type.

Dashboards

• Folders with a stated meaning. A folder says what belongs in it, which a label never did. A locked folder is an administrator's to change and read-only for everyone else, the author included.
• By email, on a schedule. A PNG inline and a one-page PDF, or a link for people who should sign in. Shares to Slack as well.

MCP

• OAuth 2.1: discovery, PKCE, refresh with rotation, revocation and dynamic client registration, so a client can connect without anybody pasting a key. Client ID Metadata Documents are not implemented.

Administration

• Overview, System Status, RQ Status, Storage Status, Running Queries, Streaming Queries, Outdated Queries and MCP. Where the headroom is going, who has been running what, why a quiet stream is quiet, and what to clean up now rather than at its next slot.
• Permissions an administrator hands to a group: streams, the catalog, live dashboards, keeping uploads, MCP and sending dashboards. A permission the install does not offer is offered to nobody, rather than leading somebody to a page that never fills.
• Uploads expire on a clock people can see, and stop, if they may.

Smaller

• Nothing heavy is on by default. Streams, MCP, uploads and the screenshot renderer are each off until a chart value or an environment variable asks for them, so an install pays for what it uses and the parts scale apart.
• Half the initial download: 931 KB gzipped to 525 KB, held there by a CI budget.
• One spinner, drawn in CSS instead of typed from an icon font.

Images

ghcr.io/bot-netizen/sqldesk:0.7.0
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.0

linux/amd64 and linux/arm64. The screenshots image is the optional renderer for pictures in alert emails.

Upgrading from 0.6

One migration, one default that changed, and one thing you have to ask for. Full version in the deployment guide.

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.0

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml run --rm server manage db upgrade
docker compose -f compose.prod.yaml up -d

Helm — the chart migrates itself, in a hook that runs before anything else changes.

helm upgrade sqldesk ./charts/sqldesk --reuse-values \
  --set image.tag=0.7.0

If people upload files, say so on the command line. uploads.enabled defaulted to true in 0.6 and defaults to false in 0.7, so an install that never uploads anything carries no volume. An install that was using the default has nothing for --reuse-values to carry forward, and the upgrade turns uploads off. The files are safe — the claim is annotated helm.sh/resource-policy: keep — and --set uploads.enabled=true brings it back exactly as it was.

The migration adds kind to dashboards and fills it in with the rule 0.6 applied when reading, so no dashboard changes which list it appears in. It is reversible; the downgrade drops the column.

Streams are off until you ask for them — streams.enabled=true in the chart, or SQLDESK_STREAMS_ENABLED=true with a stream worker on Compose. Left off, the streams permissions are offered to nobody, the streaming halves of the Queries and Dashboards menus are not drawn, and neither is the Settings tab for topics.

Nothing else changes an existing dashboard, query or data source. The navigation moved, and the four /streams/… URLs from the old menu redirect to where those pages live now, so links people have already sent each other still work.

Rolling back to 0.6 means putting the old image back and running manage db downgrade once.

Fixed since the release candidate

Both in alert attachments, and both invisible: a failed render is meant to cost the picture and not the alert, which is right, and is why neither had ever been noticed.

• Chromium could not resolve the address the renderer was given. The chart pointed it at the Service's short name, which every other client in the cluster resolves and the browser does not — Python in the renderer's own container gets a 200 from that URL and Chromium fails it with ERR_NAME_NOT_RESOLVED. Every screenshot on Kubernetes timed out.
• A render pass could fetch a query but not the query's result. The result endpoint checks the data source, and a pass knew only about the object it names. Every query attachment ever added to an alert was blank. Dashboards were unaffected — their public handler serves each widget's data itself.

Verified before tagging

• CI green on every job, including the end-to-end suite.
• The published image smoke-tested through compose.prod.yaml: /ping and /setup answering, the scheduler registering its periodic jobs, and refresh_queries and cleanup_query_results both running clean on the worker.
• Alert attachments driven end to end on a cluster, which had never been done: an alert fired with a dashboard and a query attached arrives with both pictures inline, 69,930 and 51,405 bytes.