Repository navigation
v0.5.0
Release v0.5.0
📦 NPM packages
🐳 Docker image
docker pull ms2data/malloy-publisher:0.5.0
docker pull ms2data/malloy-publisher:latest
ℹ️ Details
- Release branch:
release/sdk-0.5.0 - Triggered by: @sagarswamirao
an SSH tunnel with no pinned host key is now refused (ACTION REQUIRED)
proxy.ssh.hostKey pins the bastion's host key. When it was omitted the tunnel
connected to whatever key the far end presented, which means a
machine-in-the-middle on the publisher-to-bastion hop could not be detected. That
was the documented default, so a deployment relying on it is doing what the docs
told it to.
It now fails closed: a connection whose ssh.hostKey is unset is refused when a
query first uses it. If you run an SSH-proxy connection without a pinned host
key, queries through it will start failing after this upgrade.
Do one of the two below before the new image rolls, not after. The refusal
fires on the first query through an unpinned tunnel, so a deployment that waits
to react has already failed those queries.
Two ways forward, and the first is the one to prefer:
- Pin the key. Put the bastion's host key in
ssh.hostKey-- an OpenSSH
known_hostsline or a bare base64 blob, one per line. A load-balanced bastion
presents a different key per backend, so list every backend's key; any listed
key is accepted. - Or opt out for the deployment.
PUBLISHER_ALLOW_UNVERIFIED_SSH_HOST_KEY=true
restores the old behaviour and logs a warning on every unpinned connect.
true,1,yesandonall opt in, case-insensitive; an unrecognised value
fails config load rather than leaving verification silently off.
To find the affected connections before upgrading, look for a connection with a
proxy.ssh block and no ssh.hostKey. After upgrading, you do not have to wait
for a failing query either: the tunnel is dialed lazily, and on config load this
release logs a warning naming each SSH connection that pins no host key while the
opt-in is off, so the list is in the startup log before anyone runs a query.
compile and sqlSource now count against the concurrency cap
PUBLISHER_MAX_CONCURRENT_QUERIES bounds how much work a pod runs at once so a
flood cannot saturate it. It covered query, sqlQuery and sqlTemporaryTable,
but not compile or sqlSource -- and both of those reach the database too:
compile resolves a source's schema against the connection, and sqlSource runs a
live introspection. A burst of either bypassed the cap its sibling routes
enforce. The legacy /projects/... routes and the compile_model MCP tool had
the same gap, so all three surfaces are gated together; leaving one open would
just move the bypass.
What changes for an operator: the cap now has to be sized for authoring traffic
as well as query traffic. An agent loop or a notebook that compiles on every edit
draws on the same pool a query does, so a deployment that sits near its cap may
start seeing 503s on compile and sqlSource that it did not see before. The cap
defaults to 32 and 0 still disables it entirely. The dashboard save
(PUT /environments/:env/packages/:pkg/models/*?) admits here too: it compiles
the submitted text and rewrites the package under its lock.
A deployment that finds the cap too tight once authoring traffic counts against
it can raise PUBLISHER_MAX_CONCURRENT_QUERIES -- 64 or 128 -- rather than
leaving these routes ungated. Raise it knowing what it governs: one pool bounds
aggregate memory for concurrent warehouse work, so a cap sized to absorb
authoring pressure also raises the ceiling on concurrent query memory.
What this does not cover, so the entry is not read as a complete list:
?reload=true answers to the memory governor but not to this cap, and the REST
connection schemas and tables routes and the MCP search_database_schema
tool take no slot.
two server defaults now close instead of open
Two settings that were open by default are closed. Both are silent until
something that relied on the old default stops working, so each needs a
deliberate step if you were depending on it.
The authorize bypass now requires a secret. x-publisher-bypass-authorize
disabled every #(authorize) gate on the presence of the header alone, with no
value to know. It now requires PUBLISHER_BYPASS_AUTHORIZE_SECRET to be set and
the header to carry that value; with the variable unset the bypass is refused
outright rather than allowed. If you relied on the bypass, set the variable and
send it as the header value, or stop relying on it.
MCP binds loopback and no longer allows every origin. The MCP server bound
0.0.0.0 with a bare cors(), so it accepted connections from the network and
cross-origin requests from anywhere. It now binds 127.0.0.1 and reads allowed
origins from MCP_CORS_ORIGINS, defaulting to none. A remote MCP client that
could reach port 4040 can no longer do so: set MCP_HOST=0.0.0.0 to restore the
old bind, and put a gateway in front of it (see docs/security-posture.md).
Know what widening it exposes before you do. MCP tools take environmentName
and packageName as ordinary arguments and the discovery tools treat them as
optional, so a caller who reaches the endpoint can enumerate every loaded
environment and the connections on each. Publisher has no tenant model to scope
that against, so a worker reachable by more than one tenant must not expose MCP.
MCP gets its own host knob rather than reusing PUBLISHER_HOST, because that
variable drove both the REST and MCP listeners -- defaulting it to loopback would
have moved the REST port to localhost too. Precedence is MCP_HOST, then an
explicit PUBLISHER_HOST so --host still moves both together, then
127.0.0.1. The REST default is unchanged.
configEtag, so a writer can tell which Publishers still hold the config it sent
Credentials are never returned on a read, so a system distributing the same connection to several
Publishers could not confirm any of them was still holding the credential it last sent: a read tells
a Publisher holding some password from one holding none, not one holding last month's from one
holding the current one.
Connection.configEtag is a new optional string the writer owns. Publisher stores it with the
connection, returns it on reads, and never derives, validates or interprets it — compute a tag over
the config you are about to send, send the two together, and compare what each Publisher reports
against what you would send now. A different tag, or none, means that Publisher was not given that
configuration. A write that does not carry a tag clears it, so a client that ignores the field is
unaffected and a config changed outside your writer stops hiding behind a tag it no longer matches.
It is deliberately not fingerprint, which identifies the data a connection reaches and excludes
credentials so rotation does not re-address artifacts: two configs differing only by password share
a fingerprint, which is the case this exists to catch. docs/connections.md
has the comparison and the limits.
a colocated persist whose query is built with a given is refused
A given's value is substituted when the compiler compiles. Inside a persisted query the only value available is the declaration default, so it was baked into the relation — and persistence swaps only the source's FROM, leaving nothing to re-apply a filter that lives inside that relation. The table held one caller's slice and was served to everyone, whatever value they supplied. That shape is now refused.
What is still admitted, and is the documented form (row-level-access.md): a given applied when the source is READ — a where: in the source's extend block, or a dimension, measure or join declared there. It never reaches the build, and binds per caller over the materialized rows. Only a given the persisted query is built with is refused.
#@ persist name="refused"
source: refused is raw -> { where: org_id = $ORG_ID; select: * }
#@ persist name="admitted"
source: admitted is raw -> { select: * } extend { where: org_id = $ORG_ID }
On upgrade, such a package keeps loading and its source keeps serving — live, correctly, per caller. What changes is that its materialization run now 422s with the refusal, and an artifact built before the upgrade is unbound on the next reload rather than served. Moving the given out of the persisted query restores materialization; the refusal message names the placement.
The storage= tier still refuses a given reference in any position, so what it accepts is unchanged. One reported value shifts: a #@ preaggregate rollup that also declares storage= runs the colocated check first, so a refusal that read given now reads given_in_persisted_query. Same refusal, different label.
A new reason value. Refusals are reported on the build plan, and this adds given_in_persisted_query to that enum. A consumer generating a strict client from an older copy of the spec can fail to parse a package whose plan carries it — which happens only for a package that actually has the refused shape. Regenerate against this release's api-doc.yaml, or expect the value.
the dashboard editor can now filter a tile whose view is written inline
A dashboard tile's filter control used to refuse to bind on an inline tile — view: x is { aggregate: … } — because the only write path was a + { where: … } refinement
after the view reference, which does not exist to append to when there is no reference.
That excluded the majority of real dashboards: writing a view's body inline, rather than
as a named reference, is the common way people write one, and the bundled
tiled.malloy fixture is entirely inline tiles.
The fix is a second write path, not a workaround: a binding on an inline tile is now a
depth-1 where: statement inside the body's own first stage, which is valid Malloy and
reads back exactly like a reference tile's refinement does. Only that shape is
recognized — a nested where: inside a nest:, a compound predicate such as where: a ~ $A and c = 1, and a source-level where: outside any view are all left exactly as
written, never touched and never reported as a binding. Where a body has more than one
stage, only the first is the tile's own: { … } -> { … } takes its binding in stage one
and nothing is ever written into a later stage, while a body with no single first stage
-- a { … } + { … } compound refinement, or a pipeline starting from a named view --
refuses a filter change with a reason naming the shape.
Recognizing that shape is a question about statements, and every scan here reads a line
at a time, so the two can disagree: a clause list or a predicate carried onto a second
line (where: a ~ $A, then b ~ $B) is only half-visible to a line-oriented reader.
Such a where: is now unmodeled Malloy on both write paths -- read past, written around,
never rewritten -- because rewriting the half that was read would strand the half that
was not. The same isolation rule now governs a + { where: … } refinement, which
previously matched binding clauses anywhere in the refinement with no such check.
Two guards back that up. A given already filtered on by a where: the builder does not
manage cannot also be bound as a managed clause, because the two would filter on the same
control while only one could ever be unbound again; that is refused with a reason naming
the given. And every rewrite is now parsed by Malloy before it is written: a file that
parsed before the edit must still parse after it, or nothing is written. That check sees
the whole file, which the existing read-back gate cannot -- the gate compares tiles, tags
and filters, so text stranded beside a clause it rewrote is invisible to it.
What the live editor shows while you work: adding a filter to an inline tile previews
correctly. Removing or changing one does not take effect in the preview until the file is
saved, because the tile's preview runs the saved view, whose body already holds the saved
where:, and the builder has no way to name that view unbound. A reference tile is exact
either way, because it refines its base view. The saved result is correct in every case;
this is the preview only.
Consequence for an existing file: an author's own where: x = $Y written at depth 1
of an inline view's first stage is now builder-managed the same way a reference tile's
refinement already was. Once that filter's control is touched through the builder and
the file is saved, that where: is regenerated from the control's bindings rather than
preserved verbatim — the same contract a reference tile's refinement already had, now
extended to the more common inline shape.
Comments, in all three spellings Malloy accepts. //, -- and /* … */ are all comments
to Malloy's lexer, and the builder knew only about //. Two consequences are fixed. A
/* … */ was absent from the comment index, so it was invisible to every guard that asks
whether a range about to be deleted holds one: removing a filter across a block comment deleted
it and reported success. And a -- or /* … */ line written between a # tag and the
declaration it annotates stopped the walk that finds a tile's tags, while the parser read
straight past it — so a retag wrote a second # colspan below the comment, the reader read
the lower one back, and the read-back gate was satisfied by a file now carrying two. That
hand-written walk is gone; where a tag block begins now comes from the lexer, which also
means a line inside a /* … */ that happens to begin # is read as the prose it is rather
than as a tag to report or rewrite.
A tile the builder cannot bind is no longer described as somebody else's. A -> pipeline
from a named view, or a chained vx + { … } + { … } where no one block is where a binding
belongs, is declared right there in the dashboard file — but the tile menu said it was declared
on its source, which is untrue and hid the fact that its tags are the builder's to write. Such a
tile now reads as declared here, with the shape named; its label, subtitle, colspan and position
stay editable like any other tile's, only the filter control is off, and the reason a filter
change gives points at the body rather than sending you to the model file.
One rule for the three removal paths. Removing every clause of a where: used to delete a
comment written inside it, or leave one trailing it stranded above the closing brace, while
collapsing a refinement over a comment refused. All three now answer the same question the same
way: a comment goes only with a declaration you asked to delete outright — removing a tile
still takes its own comments with it, and the builder shows that diff before a structural save
— while a filter edit, which rewrites a declaration that stays, refuses rather than destroying
or stranding a comment it was not asked about, and names the comment in the reason.
DashboardEditor takes a resourceUri, and can now open a pinned version
DashboardEditor was the only resource-addressed component in the SDK still taking loose
environmentName / packageName props, under a dashboardName that disagreed with
Dashboard's own dashboard for the same slug, and with no way to pin a versionId the
way every other resource-addressed component can. It now takes resourceUri + dashboard,
matching Dashboard exactly; the old environmentName / packageName / dashboardName
form still works, deprecated rather than removed, so an existing integration is unaffected.
A versionId on the URI pins every read the editor makes — the file, the manifest, the
dashboard list, the catalog behind the filter window's field search, and, through the live
surface it renders, each tile's query and each control's suggest query — the same as it
already does for Dashboard. It never reaches the write: Publisher answers 501 Not Implemented to a versionId on updateModelSource, and a version is a fixed point in
history regardless, so a pin against a package that would otherwise take the editor's
writes now turns Save off instead, with the toolbar caption saying why. A save into a
host's own document store or a browser draft is unaffected, since neither goes through
that endpoint.
What's Changed
- docs: retire the dormant next pre-release channel by @mlennie in #1078
- fix(sdk): a dashboard card is white, and the edit bar has one tick by @kylenesbit in #1188
- chore(release): stamp 0.4.1 on main by @Sha-Bang in #1189
- fix(server): say why a server with no config is serving nothing by @Sha-Bang in #1184
- fix(get_context): report model paths a source actually resolves in by @jswir in #1134
- fix(sdk): a card's edge is a card's edge — dashboard cards, the filter box, and every form field by @kylenesbit in #1197
- fix(server): close the authorize bypass and the MCP bind by default by @sagarswamirao in #1140
- fix(server): refuse a colocated persist whose persisted query references a given by @housejester in #1185
- fix(server): gate compile and sqlSource behind the query-concurrency cap by @sagarswamirao in #1109
- fix(proxy): fail closed on an unverified SSH bastion host key by @sagarswamirao in #1110
- fix(server): key a gate entry with a separator that does not make the file binary by @housejester in #1191
- feat(connections): configEtag, an opaque writer-owned tag echoed on reads by @housejester in #1187
- skills: what an agent reads, and the tests that keep it honest by @jswir in #1195
- fix(sdk): let Malloy's parser locate every dashboard edit, so the builder stops maintaining a second grammar by @Sha-Bang in #1190
- fix(server): redact every connection credential from a failed connection test, not only Postgres by @sagarswamirao in #1196
- docs(security): say who must restrict DuckDB's host reach on a shared worker by @sagarswamirao in #1203
Full Changelog: v0.4.1...v0.5.0