SQLDesk 0.7.3 — Signal and Noise

Four gaps closed in what 0.7 shipped, three faults found on the way, and one idea running through all of it: say what matters, quiet what does not.

SQLDesk stops appearing in lag monitoring

• Every stream joined a consumer group it never committed under. sqldesk-stream-<id> was a live member with no position at all — exactly what a lag monitor exists to alarm on.
• The default is now no group the broker can see. It cannot literally be none: librdkafka refuses to build a consumer without a group.id. So one is set and never used — the consumer assigns itself every partition and commits nothing, and the broker never registers it.
• Nothing is lost by not joining. A lock already guaranteed one consumer per topic, so the group never balanced anything.
• Existing streams change on upgrade, so the alarms stop without anybody touching a setting.

How a topic is read, chosen per topic

Set on the Analyse panel, where a topic is enabled:

• Consumer group name — optional, for monitoring that ignores consumers by prefix. A name joins that group.
• Save position — off by default, and the page says in words that nothing is saved. On, offsets are committed after each flush, and only after it.
• Start from — latest, earliest, or where it left off. A setting rather than a prompt, because a stream starts when somebody opens a dashboard.

A hole at every hand-over

• Consumption is a chain of jobs, and every one sought to the end when it started — hand-overs included. So each hand-over dropped whatever arrived between two jobs: a small gap in the chart every few minutes.
• Positions are now carried across a hand-over, so the start setting applies only to a cold start. That is also what stops "earliest" replaying the whole topic at every hand-over.

The catalog can be searched, and told what to ignore

• Tables past the first 200 were unreachable. The page is sent the 200 most used and the limit was on the server, so the rest were not merely off the page but impossible to reach from it. A search now runs on the server over every table, by name and description.
• The page says when it is not showing everything — "Showing the 200 most used of 1,432".
• Hide a table. Off the catalog page and out of what MCP offers a model — hundreds of Mimir metrics are noise to both. Still queryable, and a re-harvest keeps it hidden.
• Show hidden, with a count, for when you need to find one.
• Filters: with a description, and missing one.
• An empty description shows the shape of a good one — what it is for, the grain, how it is partitioned, what the key looks like.
• Fixed on the way: the edit set the description from whatever request arrived, so hiding a table would have wiped what somebody had written about it.

Curate the catalog over MCP

• update_table describes or hides a table from whatever MCP client you already use. The first MCP tool that writes.
• A personal API key only, from somebody with the catalog permission. An OAuth token never reaches it — those carry mcp:read — and the tool is listed only to a caller who may use it.
• It will not replace a description a person wrote unless told to with overwrite. What it writes is tagged "via MCP" and left alone by harvests.

Verified dashboards

• A badge beside the title, with a required note on what was checked. Tap it for who, when and what — a tap, because the note has to be readable on a phone.
• It does not outlive what it vouched for. Edit the SQL behind a chart, or add or remove a widget, and it reads "verified, then changed". Moving a widget does not count.
• The same permission as confirming a query in the catalog — a curator's act, so it needs no edit right on the dashboard.

Images

ghcr.io/bot-netizen/sqldesk:0.7.3
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.3

linux/amd64 and linux/arm64.

Upgrading from 0.7.2

• Three migrations, each adding columns with defaults. All reversible.
• Streams change how they look to the broker. The sqldesk-stream-<id> groups stop appearing, which is the point. A topic that should be visible under a name needs one set on its page.
• Nothing else to configure.

Helm — the image tag has no v; leave it unset and the chart uses its own appVersion.

helm upgrade sqldesk ./charts/sqldesk --reuse-values

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.3

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml run --rm server manage db upgrade
docker compose -f compose.prod.yaml up -d

Verified before tagging

• CI green on every job, including the end-to-end suite.
• 2504 backend tests — run with a real Kafka broker, so the streams tests that usually skip all ran — and 717 frontend tests.
• Against a real broker, not a fake: an unnamed consumer never appears in the broker's group list and is never issued a member id; a named one is; a saved position resumes past what was already in the window; earliest reads the backlog; and a hand-over reads what arrived between the jobs.
• The upgrade path itself, on the published image: a database created by 0.7.2 upgraded to 0.7.3 through manage db upgrade; all three migrations then downgraded and upgraded again, cleanly.
• The published image against a real broker: a stream consumer read every message, was never issued a member id and never appeared in the broker's group list, and recorded where it got to for a hand-over.
• The published image through compose.prod.yaml: /ping and /setup at 200 with Content-Encoding: br, 14 periodic jobs registered, refresh_queries and cleanup_query_results clean on the worker, 75 data source types offered.
• Every new rule was mutation-tested — making every consumer join, dropping the hand-over, letting OAuth write, overwriting a person's description, wiping a description on hide, searching only the first 200: each fails its own tests.

SQLDesk 0.7.2 — Over the Wire

Nothing compressed a response, every driver was behind, and a DuckDB upgrade uncovered a fault older than the upgrade.

Nothing in SQLDesk compressed a response

• Not the app, not the chart, not the ingress templates. A /ping asked with Accept-Encoding: gzip came back with no Content-Encoding at all — which is how it survived, because there was nothing to notice.
• A 20,000-row result sent its full 3,715,944 bytes, where 420 KB would do.
• And 6,747,167 bytes of JavaScript and CSS went out byte for byte on every cold page load. They gzip to 1,630,082 — a factor of 4.1.
• The codecs are chosen from measurement, not habit: brotli q4 at 27 ms and gzip 6 at 31 ms, both 8.8×. Level 9 is deliberately absent — 157 ms for ten percent more. For scale, encoding that result as JSON already costs 112 ms.
• The saving that matters is not bandwidth. A sync Gunicorn worker, of which there are four, holds its socket for a fraction as long, and the web tier is the first ceiling an install meets.
• Only text. A PNG from the renderer, an .xlsx (a zip) and a Parquet upload would each get bigger for CPU spent. SQLDESK_COMPRESS_RESPONSES=false turns the whole thing off.

Every number and timestamp from DuckDB was declared a string

• TYPES_MAP has always been keyed by DuckDB's own type names, and up to 1.3 the driver reported DB-API ones instead: NUMBER, STRING, DATETIME, bool.
• Of twelve ordinary column types, five matched. The seven that did not included every numeric type and every timestamp — so every number and every timestamp from a DuckDB source, and therefore from every uploaded CSV and Parquet file, arrived typed as text.
• It was invisible because the browser sniffs the type of any column declared string and recovers most of it. Charts have been quietly getting the right answer by guessing.
• 1.5 reports INTEGER, TIMESTAMP, BOOLEAN and the rest, which the table already knew — so the upgrade fixes it. Eleven of twelve now map with nothing but a str(); the twelfth, DECIMAL(10,2), is parameterised exactly like Trino's and handled the same way.
• A test now asks DuckDB for a column of each of seventeen types, so a rename fails loudly instead of quietly typing everything as text.
• The upgrade also broke the runner outright before that was found: cursor.description's type code became a DuckDBPyType, and .upper() on one forwards to its child-type lookup rather than raising an ordinary attribute error — so the failure read "Tried to get child type by the name of 'upper'", which says nothing about the cause.

Drivers

Eight upgrades, each checked rather than bumped.

	from	to
trino	0.330.0	0.340.0
duckdb	1.3.2	1.5.6
mysqlclient	2.1.1	2.3.0
pymongo	4.6.3	4.18.2
snowflake-connector-python	4.5.0	4.8.0
pyhive	0.6.1	0.7.0
impyla	0.22.0	0.24.0
psycopg2-binary	2.9.11	2.9.13

• DuckDB persists real database files — a stream window is one, and a DuckDB data source points at one — so the storage format was checked in both directions. 1.5.6 reads and writes a file written by 1.3.2, and 1.3.2 reads one written by 1.5.6, so the upgrade and the rollback are both safe.
• MySQL's ssl_mode is what 0.7.1 fixed, so 2.3.0 had to still take it as its own connect argument. It does — and it still ignores an unknown key in the ssl mapping without complaint, which is the behaviour that made the original fault invisible.
• pymongo loses [srv,tls], which turns out never to have been real: neither extra has existed since pymongo 3.x, so the pin has been asking for nothing since the Redash days. SRV connection strings are unaffected — dnspython is a hard dependency.
• pandas, oracledb and pyathena are deliberately held, with the reason written beside each pin. pandas 3 changes copy-on-write and the default string dtype, and both utils/pandas.py and the Python runner depend on today's behaviour. oracledb is an eight-major-version jump. pyathena 3 renames its cursor classes, and its ArrowCursor belongs with the columnar work rather than a patch release.

Trino can use the spooled protocol

• Under the standard protocol every row of every result is paged through the coordinator's heap as JSON, which is Trino's own ceiling on large results.
• Spooling has the workers write segments to object storage for the client to fetch directly, compressed with lz4 or zstd, leaving the coordinator to pass only metadata.
• It is a field on the data source and off unless chosen, because it needs a server that offers it — Trino 466 and later, with spooling configured — and there is no way to tell from here whether one does.
• There is no ADBC driver for Trino, which is what prompted looking: it is not on PyPI, and Trino does not speak Flight SQL natively either, so that driver does not reach it.
• Worth knowing what this is not: the Python client fetches segments sequentially. Its only thread pool backgrounds the segment acknowledgements. Parallel fetch is a Java-client feature.

Images

ghcr.io/bot-netizen/sqldesk:0.7.2
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.2

linux/amd64 and linux/arm64.

Upgrading from 0.7.1

Nothing to do beyond the image. No migration, no configuration change, no default moved.

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.2

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml up -d

Helm — the image tag has no v; the git tag does.

helm upgrade sqldesk ./charts/sqldesk --reuse-values

Leaving image.tag unset is both correct and the safer habit: the chart defaults it to its own appVersion.

One thing to expect. A DuckDB column that rendered as text may now render as a number, a boolean or a date — because it always was one and is finally declared as such. Charts that worked will keep working, since the browser had been guessing the same answer, but a column can change its alignment or its formatting.

If people upload files, --set uploads.enabled=true still applies. That default changed in 0.7.0 and has not moved since.

Verified before tagging

• CI green on every job, including the end-to-end suite.
• 2418 backend tests and 697 frontend tests pass. black 23.1.0 and ruff 0.0.287 — the versions CI pins — clean across 645 files.
• Every new test was mutation-tested. Putting each fault back makes its own test fail: three mutants for the compression config, three for the DuckDB type mapping, one for the Trino encoding.
• Checked against the published image rather than the source, through compose.prod.yaml with the schema created before the services started:
  • /setup answers Content-Encoding: br, and the largest bundle comes down 1,103,381 → 305,161 bytes, a factor of 3.6, through the real server. /ping's five-byte body is correctly left alone.
  • DuckDB types a published-image query as integer, float, datetime, boolean, string and string — the thing that was text until this release.
  • The Trino encoding field offers json, json+lz4 and json+zstd, and both codecs are installed.
  • /ping and /setup at 200, 14 periodic jobs registered, refresh_queries and cleanup_query_results clean on the worker, and 75 data source types still offered.

SQLDesk 0.7.1 — Data Sources

Five faults in the data source layer. Four were found by reading the runners rather than from a report, which is the point: not one of them raises anything an operator would ever see.

A data source that was not there

• Vertica has not been offered since we moved to Python 3.13. vertica-python 1.1.1 imports the standard library's crypt, which PEP 594 removed. import vertica_python raised, enabled() caught it as an ImportError and returned False, and the type vanished from the list — no log line, no failed request, nothing. manage ds list_types on a 0.7.0 install does not mention Vertica at all. Anybody who went looking concluded SQLDesk does not support it.
• The driver is now 1.4.0, which vendors its own crypt. Every numeric type code was checked against it one by one, and a test now fails if an upgrade ever renumbers them.
• Vertica had no TLS option of any kind — not a mode, not a certificate. Every connection, and so every password, crossed the network in the clear with no way to say otherwise. It now offers the five modes the driver accepts.
• read_timeout is gone, because the driver dropped the option and we were passing it into a call that ignores it. A form promising a limit that nothing enforces is the same fault as the one below.

A security control that reported success

• The MySQL SSL Mode dropdown decided nothing. The value was filed into the ssl mapping under the key "preferred" — the value of a lookup table used as its key — and mysqlclient reads only ca, capath, cert, key and cipher from there, ignoring the rest without complaint. An install that chose Verify Identity verified nothing, and the form said it had.
• Worse than a missing control, because nobody goes looking. The mode is now passed as its own connect argument and translated into MySQL's spelling.
• On RDS we shipped Amazon's CA bundle and then asked for no verification against it, so the default applied and an unencrypted connection was accepted if the server offered one. Use SSL now means VERIFY_CA — the chain is proved without also demanding a hostname match, which fails behind a CNAME or a proxy.

Trino

• Columns came back with no type. The map held bare names and cursor.description carries the declared type: varchar(255), decimal(10,2), and since precision became explicit, timestamp(3) — the default. Every parameterised type missed the lookup, so an ordinary table produced untyped columns and nothing downstream could format a date or align a number.
• A decimal was typed as an integer, so every price, rate and ratio arrived with its fractional part treated as noise.
• The error handler was the error. It read .get("failureInfo", {"message", None}) — a set literal where a dict was meant, a comma where a colon belonged. With failureInfo absent the default was a set, .get on a set raises AttributeError, and the code whose only job was to report the failure threw instead.
• The connection leaked. A dbapi connection holds an HTTP session to the coordinator and nothing closed it, so a worker accumulated one per query it had run.

Logos

• A data source without a logo file drew a broken image. A logo is a file named after the type, and all ten places that draw one were a bare <img> with no error handling — so a type without its file rendered the browser's broken-image glyph, silently, because a 404 on an image is reported to nobody. Kafka shipped that way once.
• They go through one component now, and a missing file degrades to a tile with the type's initials in a colour derived from its name. Adding a real logo later is a file drop with no code change.

Added

• Grafana Mimir. The /prometheus prefix is added for you either way you spell the URL, and a Tenant ID is required because Mimir refuses a request without X-Scope-OrgID whenever auth is enabled — which is the default. Its connection test asks the query API rather than the root URL: the inherited check passed on any 200, so a gateway's welcome page counted as success.
• A Prometheus runner that can carry a credential. It could authenticate exactly one way — a TLS client certificate — because it extends BaseQueryRunner rather than BaseHTTPQueryRunner. Anything behind a gateway, a bearer token or basic auth could not be added at all. It now takes a username and password, a bearer token and free-form headers, which also unblocks Thanos, Cortex and VictoriaMetrics.
• MariaDB, under its own name. The MySQL runner could always talk to it, but only for somebody who already knew MariaDB speaks the MySQL protocol.

Images

ghcr.io/bot-netizen/sqldesk:0.7.1
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.1

linux/amd64 and linux/arm64.

Upgrading from 0.7.0

Nothing to do beyond the image. No migration, no configuration change, no default moved. Vertica, Mimir and MariaDB appear on their own once it is running.

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.1

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml up -d

Helm — the image tag has no v; the git tag does.

helm upgrade sqldesk ./charts/sqldesk --reuse-values

The chart defaults image.tag to its own appVersion, so leaving it unset is both correct and the safer habit. --set image.tag=v0.7.1 pulls a tag that was never published.

If people upload files, --set uploads.enabled=true still applies — that default changed in 0.7.0 and has not moved since.

Verified before tagging

• CI green on every job, including frontend-e2e-tests.
• 2401 backend tests and 697 frontend tests pass. black 23.1.0 and ruff 0.0.287 — the versions CI pins — clean across 643 files.
• The new tests were mutation-tested. Putting each fault back makes its own test fail, including the logo fallback, where removing the error handler fails two of eight.
• The published image smoke-tested through compose.prod.yaml on both the schema-first path and the services: /ping and /setup answering, 14 periodic jobs registered by the scheduler, and refresh_queries and cleanup_query_results both clean on the worker.
• Every claim in this release checked against the published image rather than the source: manage ds list_types offers 75 types including vertica, mimir and mariadb; Mimir resolves http://mimir:8080 to http://mimir:8080/prometheus and sends X-Scope-OrgID; MySQL returns VERIFY_IDENTITY and RDS VERIFY_CA; Trino maps decimal(10,2) to float, timestamp(3) to datetime and reports a message with no failureInfo; and vertica-python is 1.4.0 with Vertica.enabled() true.

SQLDesk 0.7.0 — Kafka Streams

Kafka topics you can query, with the same editor, the same visualizations and the same dashboards as everything else — and nothing new switched on for anybody who does not ask for it.

Streams

• A cluster is a data source; its topics are its tables. Add it the way you add Postgres. Which topics may be queried is chosen separately, because that is the access decision: everybody who can reach the cluster can query whatever is enabled on it.
• A streaming query is a query. Same editor, same parameters, same visualizations, same dashboards. Execute says Start streaming, and no result is ever stored — a window is what a consumer saw while somebody was watching, and writing it down would be making a copy of the one thing whose point is that it is current.
• Analyse a topic before enabling it. It reads the last few hundred messages and says what the columns are, how many could not be parsed, how fast they arrive, how much history that buys, and whether it will be sampled. None of that can be guessed from a topic's name.
• A row budget and an events-per-second ceiling per topic. Over the ceiling, events are kept one in N by a hash of the message key — so per-key counts scale back up honestly — and anything drawn from a sampled stream says so.
• A slot is a topic, not a viewer. Five people watching orders share one consumer and one window and take one slot between them, which makes a team round one dashboard during an incident nearly free. When there is no room, the editor says so beside the button that will not start.
• A topic is consumed only while somebody is watching, and stops about a minute after the last person looks away. The window is kept for ten minutes, so coming back resumes in seconds.

Streaming is a kind of query, not a section of the application

• No broker's name in the navigation. Streaming queries are a half of Queries; streaming dashboards a half of Dashboards. The two halves never overlap, so nothing is filed in two places.
• Which topics may be queried is a tab beside Data Sources, because it configures a connection.
• A dashboard declares its kind when you make it, carries a streaming chip when it is open, and shows a band that turns green while its topics are being consumed. Live keeps its old meaning: refreshing right now, which an ordinary dashboard can also be.
• A folder holds both kinds, on purpose — it is a place, not a type.

Dashboards

• Folders with a stated meaning. A folder says what belongs in it, which a label never did. A locked folder is an administrator's to change and read-only for everyone else, the author included.
• By email, on a schedule. A PNG inline and a one-page PDF, or a link for people who should sign in. Shares to Slack as well.

MCP

• OAuth 2.1: discovery, PKCE, refresh with rotation, revocation and dynamic client registration, so a client can connect without anybody pasting a key. Client ID Metadata Documents are not implemented.

Administration

• Overview, System Status, RQ Status, Storage Status, Running Queries, Streaming Queries, Outdated Queries and MCP. Where the headroom is going, who has been running what, why a quiet stream is quiet, and what to clean up now rather than at its next slot.
• Permissions an administrator hands to a group: streams, the catalog, live dashboards, keeping uploads, MCP and sending dashboards. A permission the install does not offer is offered to nobody, rather than leading somebody to a page that never fills.
• Uploads expire on a clock people can see, and stop, if they may.

Smaller

• Nothing heavy is on by default. Streams, MCP, uploads and the screenshot renderer are each off until a chart value or an environment variable asks for them, so an install pays for what it uses and the parts scale apart.
• Half the initial download: 931 KB gzipped to 525 KB, held there by a CI budget.
• One spinner, drawn in CSS instead of typed from an icon font.

Images

ghcr.io/bot-netizen/sqldesk:0.7.0
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.0

linux/amd64 and linux/arm64. The screenshots image is the optional renderer for pictures in alert emails.

Upgrading from 0.6

One migration, one default that changed, and one thing you have to ask for. Full version in the deployment guide.

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.0

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml run --rm server manage db upgrade
docker compose -f compose.prod.yaml up -d

Helm — the chart migrates itself, in a hook that runs before anything else changes.

helm upgrade sqldesk ./charts/sqldesk --reuse-values \
  --set image.tag=0.7.0

If people upload files, say so on the command line. uploads.enabled defaulted to true in 0.6 and defaults to false in 0.7, so an install that never uploads anything carries no volume. An install that was using the default has nothing for --reuse-values to carry forward, and the upgrade turns uploads off. The files are safe — the claim is annotated helm.sh/resource-policy: keep — and --set uploads.enabled=true brings it back exactly as it was.

The migration adds kind to dashboards and fills it in with the rule 0.6 applied when reading, so no dashboard changes which list it appears in. It is reversible; the downgrade drops the column.

Streams are off until you ask for them — streams.enabled=true in the chart, or SQLDESK_STREAMS_ENABLED=true with a stream worker on Compose. Left off, the streams permissions are offered to nobody, the streaming halves of the Queries and Dashboards menus are not drawn, and neither is the Settings tab for topics.

Nothing else changes an existing dashboard, query or data source. The navigation moved, and the four /streams/… URLs from the old menu redirect to where those pages live now, so links people have already sent each other still work.

Rolling back to 0.6 means putting the old image back and running manage db downgrade once.

Fixed since the release candidate

Both in alert attachments, and both invisible: a failed render is meant to cost the picture and not the alert, which is right, and is why neither had ever been noticed.

• Chromium could not resolve the address the renderer was given. The chart pointed it at the Service's short name, which every other client in the cluster resolves and the browser does not — Python in the renderer's own container gets a 200 from that URL and Chromium fails it with ERR_NAME_NOT_RESOLVED. Every screenshot on Kubernetes timed out.
• A render pass could fetch a query but not the query's result. The result endpoint checks the data source, and a pass knew only about the object it names. Every query attachment ever added to an alert was blank. Dashboards were unaffected — their public handler serves each widget's data itself.

Verified before tagging

• CI green on every job, including the end-to-end suite.
• The published image smoke-tested through compose.prod.yaml: /ping and /setup answering, the scheduler registering its periodic jobs, and refresh_queries and cleanup_query_results both running clean on the worker.
• Alert attachments driven end to end on a cluster, which had never been done: an alert fired with a dashboard and a query attached arrives with both pictures inline, 69,930 and 51,405 bytes.

SQLDesk 0.7.0-rc.1 — Kafka Streams

Kafka topics you can query, with the same editor, the same visualizations and the same dashboards as everything else — and nothing new switched on for anybody who does not ask for it.

Streams

• A cluster is a data source; its topics are its tables. Add it the way you add Postgres. Which topics may be queried is chosen separately, because that is the access decision: everybody who can reach the cluster can query whatever is enabled on it.
• A streaming query is a query. Same editor, same parameters, same visualizations, same dashboards. Execute says Start streaming, and no result is ever stored — a window is what a consumer saw while somebody was watching, and writing it down would be making a copy of the one thing whose point is that it is current.
• Analyse a topic before enabling it. It reads the last few hundred messages and says what the columns are, how many could not be parsed, how fast they arrive, how much history that buys, and whether it will be sampled. None of that can be guessed from a topic's name.
• A row budget and an events-per-second ceiling per topic. Over the ceiling, events are kept one in N by a hash of the message key — so per-key counts scale back up honestly — and anything drawn from a sampled stream says so.
• A slot is a topic, not a viewer. Five people watching orders share one consumer and one window and take one slot between them, which makes a team round one dashboard during an incident nearly free. When there is no room, the editor says so beside the button that will not start.
• A topic is consumed only while somebody is watching, and stops about a minute after the last person looks away. The window is kept for ten minutes, so coming back resumes in seconds.

Streaming is a kind of query, not a section of the application

• No broker's name in the navigation. Streaming queries are a half of Queries; streaming dashboards a half of Dashboards. The two halves never overlap, so nothing is filed in two places.
• Which topics may be queried is a tab beside Data Sources, because it configures a connection.
• A dashboard declares its kind when you make it, carries a streaming chip when it is open, and shows a band that turns green while its topics are being consumed. Live keeps its old meaning: refreshing right now, which an ordinary dashboard can also be.
• A folder holds both kinds, on purpose — it is a place, not a type.

Dashboards

• Folders with a stated meaning. A folder says what belongs in it, which a label never did. A locked folder is an administrator's to change and read-only for everyone else, the author included.
• By email, on a schedule. A PNG inline and a one-page PDF, or a link for people who should sign in. Shares to Slack as well.

MCP

• OAuth 2.1: discovery, PKCE, refresh with rotation, revocation and dynamic client registration, so a client can connect without anybody pasting a key. Client ID Metadata Documents are not implemented.

Administration

• Overview, System Status, RQ Status, Storage Status, Running Queries, Streaming Queries, Outdated Queries and MCP. Where the headroom is going, who has been running what, why a quiet stream is quiet, and what to clean up now rather than at its next slot.
• Permissions an administrator hands to a group: streams, the catalog, live dashboards, keeping uploads, MCP and sending dashboards. A permission the install does not offer is offered to nobody, rather than leading somebody to a page that never fills.
• Uploads expire on a clock people can see, and stop, if they may.

Smaller

• Nothing heavy is on by default. Streams, MCP, uploads and the screenshot renderer are each off until a chart value or an environment variable asks for them, so an install pays for what it uses and the parts scale apart.
• Half the initial download: 931 KB gzipped to 525 KB, held there by a CI budget.
• One spinner, drawn in CSS instead of typed from an icon font.

Images

ghcr.io/bot-netizen/sqldesk:0.7.0-rc.1
ghcr.io/bot-netizen/sqldesk-screenshots:0.7.0-rc.1

linux/amd64 and linux/arm64. The screenshots image is the optional renderer for pictures in alert emails.

Upgrading from 0.6

One migration, one default that changed, and one thing you have to ask for. Full version in the deployment guide.

Docker Compose

# .env
SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.7.0-rc.1

docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml run --rm server manage db upgrade
docker compose -f compose.prod.yaml up -d

Helm — the chart migrates itself, in a hook that runs before anything else changes.

helm upgrade sqldesk ./charts/sqldesk --reuse-values \
  --set image.tag=0.7.0-rc.1

If people upload files, say so on the command line. uploads.enabled defaulted to true in 0.6 and defaults to false in 0.7, so an install that never uploads anything carries no volume. An install that was using the default has nothing for --reuse-values to carry forward, and the upgrade turns uploads off. The files are safe — the claim is annotated helm.sh/resource-policy: keep — and --set uploads.enabled=true brings it back exactly as it was.

The migration adds kind to dashboards and fills it in with the rule 0.6 applied when reading, so no dashboard changes which list it appears in. It is reversible; the downgrade drops the column.

Streams are off until you ask for them — streams.enabled=true in the chart, or SQLDESK_STREAMS_ENABLED=true with a stream worker on Compose. Left off, the streams permissions are offered to nobody, the streaming halves of the Queries and Dashboards menus are not drawn, and neither is the Settings tab for topics.

Nothing else changes an existing dashboard, query or data source. The navigation moved, and the four /streams/… URLs from the old menu redirect to where those pages live now, so links people have already sent each other still work.

Rolling back to 0.6 means putting the old image back and running manage db downgrade once.

Verified before tagging

• CI green on every job, including the end-to-end suite.
• The published image smoke-tested through compose.prod.yaml: /ping and /setup answering, the scheduler registering its periodic jobs, and refresh_queries and cleanup_query_results both running clean on the worker.

SQLDesk 0.6.0 — MCP

A tool server over your warehouse, a catalog built from the queries people have already written, and an audit of everything it was asked.

MCP

• One endpoint, JSON-RPC over streamable HTTP, authenticated with the API key on your profile page.
• No service account, and no way to configure one. Every call runs as that user and sees only the data sources that user can read — a tool server with more access than its user is a way to launder permissions.
• Eight tools, in the order they are meant to be used. find_queries and find_dashboards to look for work that already exists; find_context, expand_table and list_data_sources to understand the data; check_sql for the shape and explain_query for the cost; then run_query.
• run_query returns at most 1000 rows — the editor's own ceiling, reused rather than invented — and runs on a worker like any other query, so it appears in Admin under the name of whoever's key it was and can be cancelled there.
• Every request leaves an audit row, including the refused ones. Never the question somebody put to their client, and never the SQL that came back.

MCP gives no more than the editor gives

• run_query and explain_query check what the editor checks, paused sources included.
• Reads only. One statement — SELECT, WITH, SHOW, DESCRIBE — refusing a write anywhere in it, a DELETE inside a WITH and SELECT ... INTO included. That stops a model doing damage by accident; the database account's grants are still what stop it on purpose, so give each data source a read-only user.
• Bounded. One request waits at most ten seconds less than gunicorn's timeout, across all its calls. Batches hold at most ten messages, SQL at most 100,000 characters.

A catalog, built from what you already have

You have no data hub and most installs never will. But the warehouse states its own structure, several engines carry COMMENT ON text nobody was reading, and every dashboard is built on SQL somebody wrote and saved.

• Harvesting reads all three: what exists, what it is called, which tables anyone uses, which columns anyone selects, which joins anyone writes.
• It learns only from queries that have run in the last week — measured by when a query last ran rather than when it was last edited, because a dashboard refreshed every morning and untouched for a year is the most important thing in the warehouse.
• On a schedule, and on demand. Admin → Catalog has a Harvest button per data source, and a Harvest new data sources that fills in what a source added this morning without re-reading every other one.

Measures are proposed, never assumed

• SUM(amount) AS gross_revenue in a saved query is a person naming a metric, and the alias they chose becomes its name.
• Admin → Catalog shows each proposal with how many distinct queries define it that way.
• Agree, or Deny. It reaches a model only once somebody agrees it — and a proposal nobody can reject comes back every night until the list stops being read.
• Aggregates across a join and arithmetic around an aggregate are refused, not guessed. A metric definition that is merely plausible is worse than none.

The semantic layer lives in git

• manage ai export writes it as cube-shaped YAML, one file per table, into a directory you can commit from; manage ai import reads it back on deploy.
• A Download button too, because the person writing the descriptions is often not the person with access to a container.
• SQLDesk never speaks to git itself. Importing never creates anything the warehouse has not stated, nor lets a file redefine what a measure computes.

Core and add-ons

• Core: server, worker, scheduler, Postgres, Redis.
• Add-ons: MCP & Catalog, and Rendering. Off until asked for — compose profiles mcp and rendering, or mcp.enabled and rendering.enabled in the chart.
• The published image is the same size either way. The renderer is its own image and runs as a non-root user.
• A Helm chart for Kubernetes, with NodePort for minikube.

Also in this release

• File uploads are confined to their own folder. The DuckDB behind uploads let SQL read any file the worker could — another organization's uploads, /etc/passwd — write over the application, and install extensions. Each upload source's SQL now reads its own uploads and nothing else, with the setting locked so SQL cannot switch it back.
• Dashboard export is a one-page report. Past twelve widgets or one page tall, Export says so straight away. What fits exports in under a second.
• A panel shows its chart's description, not its query's — as text under the title, rather than an "i" mark nobody on a phone or a wall screen could read.
• A security review of everything reachable from outside: five passes over authentication, the REST API, the query runners, MCP and the browser. Most of what it found needed a signed-in account or a foothold on the cluster network; none of it was reachable anonymously from the internet. Details under 0.6.0-rc.4 in the changelog.
• Twelve guide pages, with the in-app help links pointing into them.

Removed

• The model-provider layer. SQLDesk calls no model; MCP serves tools to the client's. manage ai configure/status/test/disable/forget are gone with it.

Images

ghcr.io/bot-netizen/sqldesk:0.6.0
ghcr.io/bot-netizen/sqldesk-screenshots:0.6.0

linux/amd64 and linux/arm64. The screenshots image is the optional renderer for pictures in alert emails.

Upgrading from 0.5

• Pull, then manage db upgrade.
• Nothing here changes an existing dashboard, query or data source.
• MCP and the catalog are off until you turn them on — mcp.enabled=true in the chart, or the mcp compose profile. With them off, /mcp answers 404 and the Catalog page is not there.

Verified before tagging

• CI green on every job, including the end-to-end suite.
• The published image smoke-tested through compose.prod.yaml: /ping and /setup answering, the scheduler registering its periodic jobs, and refresh_queries and cleanup_query_results both running clean on the worker.

SQLDesk 0.4.0

Installing

# next to compose.prod.yaml and .env
echo 'SQLDESK_IMAGE=ghcr.io/bot-netizen/sqldesk:0.4.0' >> .env
docker compose -f compose.prod.yaml pull
docker compose -f compose.prod.yaml run --rm server manage db upgrade
docker compose -f compose.prod.yaml up -d

latest now points here, so pulling it gets 0.4.0 rather than 0.3.2.

Dashboards you watch rather than read.

A dashboard can be made live: the server refreshes it every 30 seconds to
5 minutes, once for everyone looking, and only while somebody has it open. The
visualizations that go on a wall came with it -- a gauge, a stat with a
sparkline and a target, progress and bullet bars, a status grid, table cells
that colour and carry data bars, reference lines and bands, and a rolling
window so a live chart slides instead of rescaling. Charts are drawn by Apache
ECharts now rather than Plotly, which is what made the animation between one
result and the next possible, and cut the production JavaScript from 9.6 MB to
4.3 MB.

latest and 0.4 point here. Published as
ghcr.io/bot-netizen/sqldesk:0.4.0.

The three release candidates below carry the detail of what changed and why;
this release is rc.3 as rebuilt on 19 September, with no further changes.

Upgrading

From 0.3.2:

• Run manage db upgrade. 0.4 adds a live column to dashboards:

  docker compose -f compose.prod.yaml run --rm server manage db upgrade

• Restart the scheduler as well as the server and worker (up -d does all
  three). Live dashboards are refreshed by a periodic job that a scheduler
  running the old image never starts.

• Dashboard auto-refresh now starts at ten minutes. The 1 and 5 minute
  choices are gone, and a link carrying ?refresh=60 refreshes every ten
  instead. Anything that needs to move faster is what live mode is for.
  SQLDESK_DASHBOARD_REFRESH_INTERVALS still sets the menu; values under ten
  minutes are ignored.

• Nobody can turn live mode on until an admin grants it. Open a group and
  tick Members can make dashboards live. Admins always can.

• Saved charts keep their appearance. A chart saved as custom -- which held
  JavaScript written against the Plotly API -- is drawn as a column chart,
  because there is no longer anything to run that code.

From any 0.4 release candidate: pull and up -d. No migration.

Going back to 0.3.2: manage db downgrade a8e1d0c4b726 while still on the
0.4 image, then switch the image back.

SQLDesk 0.3.2

Fixes from using it: denser tables that match everywhere, list pages whose Filter and Columns controls work, and fewer controls asking the same question.

docker pull ghcr.io/bot-netizen/sqldesk:0.3.2

No migration, nothing to reconfigure — pull and restart. Coming from 0.3.0 or earlier, read the 0.3.1 notes first: 0.3.0 broke file uploads.

Fixed

• The Filter field on the Queries and Dashboards lists closed as you clicked into it, so there was no way to type a search. antd closes a dropdown when its overlay is clicked — right for a menu, wrong for a panel built around a text field.
• Hiding a column on those lists lost it for good. It disappeared from the Columns menu as well as the table, leaving nothing to tick to bring it back. The page filtered its column list and then handed the filtered list to the menu.
• The results table header stood about twice the height of its rows — 63px against 34px. antd moves a sortable column's padding off the header cell and onto a wrapper inside it, so setting the cell's padding added to it rather than replacing it. It is 35px now.

Changed

• One table density everywhere. The compact rows from 0.2.0 applied to the query editor only, so the same table looked one way there and another in a dashboard widget or an alert. It is now a property of the table itself: 13px text, 7px cell padding.
• The Queries and Dashboards lists match it. They were still on the roomier 58px row; they are 42px now, with the same padding and the same 13px text.
• The home page is titled "My Desk", and each counter opens your own list rather than everybody's.
• One Share button on dashboards instead of two side by side. Public link, export as PDF and export as image are one menu.
• Refresh intervals gain 3 hours, 6 hours and daily, and lose the crontab entry. A query already on an expression keeps running on it and still shows it — the server never stopped understanding them — but the menu no longer offers new ones.
• Dashboards lost the schedule control added in 0.2.0. The refresh-rate button beside it already did what people wanted, and two adjacent controls both answering "how often" was the problem. Existing dashboard schedules stop running; the queries behind them keep their own schedules.
• The query editor has the same rounded corners as everything else on the page.

Licence

SQLDesk is Apache 2.0. It is a fork of Redash, copyright © 2013–2020 Arik Fraimovich, licensed under BSD 2-Clause — that licence and its copyright notice are preserved verbatim in LICENSE.redash. See NOTICE for full attribution.

SQLDesk is an independent project. It is not affiliated with, endorsed by, or supported by the Redash project or its contributors.

SQLDesk 0.3.1

Fixes uploading a file, which 0.3.0 made worse rather than better. If you are on 0.3.0, move to this.

docker pull ghcr.io/bot-netizen/sqldesk:0.3.1

No migration, nothing to reconfigure — pull and restart.

What happened

0.3.0 fixed uploads by giving the server and the worker a shared volume at /app/uploads: the server saves the file, but the worker is what runs the query, and DuckDB opens the file by absolute path from there.

The same release also stopped shipping uploads/ inside the image, which was right in itself — a developer's own uploaded files had no business being in a published image.

Together they broke it. Docker takes a volume's ownership from the directory already present in the image. With that directory gone, Docker created the mount point owned by root, and the container does not run as root. Saving an upload then failed outright with a permission error — where before 0.3.0 the upload at least succeeded and only the query against it failed.

The image now creates /app/uploads owned by the user the container runs as, so the volume inherits that.

Why it was not caught

Neither the unit tests nor CI can see this. It is a property of the image and the volume together, and only appears when the documented install is run against the real published image. That is now the second bug of this shape, after the Postgres 18 data directory in 0.1.1.

Licence

SQLDesk is Apache 2.0. It is a fork of Redash, copyright © 2013–2020 Arik Fraimovich, licensed under BSD 2-Clause — that licence and its copyright notice are preserved verbatim in LICENSE.redash. See NOTICE for full attribution.

SQLDesk is an independent project. It is not affiliated with, endorsed by, or supported by the Redash project or its contributors.

SQLDesk 0.3.0

Tealdash is now SQLDesk. The project moved to bot-netizen/sqldesk and the image to ghcr.io/bot-netizen/sqldesk. Nothing else about it changed — this release is the rename, plus one real bug fix that came out of it.

Install

docker pull ghcr.io/bot-netizen/sqldesk:0.3.0

SQLDesk needs Postgres and Redis alongside it. compose.prod.yaml in the repository wires up all three:

cp .env.example .env          # fill in the three secrets it asks for
docker compose -f compose.prod.yaml run --rm server create_db
docker compose -f compose.prod.yaml up -d

It comes up on http://localhost:5000. Images are built for amd64 and arm64.

On macOS, port 5000 belongs to AirPlay Receiver and up fails with "address already in use". Set SQLDESK_PORT and SQLDESK_HOST in .env to another port.

Keep SQLDESK_SECRET_KEY. Data-source passwords are encrypted with it.

Upgrading from Tealdash

• Your existing .env needs no edits. SQLDESK_* is read first, then TEALDASH_*, then REDASH_*. The most recent name wins.
• The image is ghcr.io/bot-netizen/sqldesk now.
• Run docker compose -f compose.prod.yaml run --rm server manage db upgrade. 0.2.0 added a column to dashboards, and this release moves saved chart palettes to the new name — charts render either way, because both former palette names resolve to the default.
• SAML: SQLDeskGroups is read first, then TealdashGroups, then RedashGroups. No identity-provider change needed.

Those three fallbacks exist because breaking them fails quietly rather than loudly — an identity provider sending the old attribute would still let people log in, into the default group, silently without their data sources.

Fixed: uploading a file and then querying it

Uploading a parquet or CSV file appeared to work, and every query against it then failed.

The server saves the upload into its own container. The worker is what actually runs the query, and DuckDB opens the file by absolute path from there — a different container, where it does not exist. compose.prod.yaml gave server, worker and scheduler no volumes at all.

They now share one. That is also what keeps uploads across a compose up that recreates a container; before, they lived in the container's writable layer and went with it.

Development was never affected, because the development compose file bind-mounts the working tree into every service — which is exactly why this reached a release. The published Docker path is the one nothing exercises.

Also

An image built from a working copy no longer carries that copy's .env or its uploaded files. .dockerignore listed only build noise, so COPY . /app took both. Images published from CI were never affected — it builds from a clean checkout, where neither is tracked.

Licence

SQLDesk is Apache 2.0. It is a fork of Redash, copyright © 2013–2020 Arik Fraimovich, licensed under BSD 2-Clause — that licence and its copyright notice are preserved verbatim in LICENSE.redash. See NOTICE for full attribution.

SQLDesk is an independent project. It is not affiliated with, endorsed by, or supported by the Redash project or its contributors.