v0.8.0

Release v0.8.0

📦 NPM packages

• @malloy-publisher/sdk@0.8.0
• @malloy-publisher/app@0.8.0
• @malloy-publisher/server@0.8.0

🐳 Docker image

docker pull ms2data/malloy-publisher:0.8.0
docker pull ms2data/malloy-publisher:latest

ℹ️ Details

• Release branch: release/sdk-0.8.0
• Triggered by: @housejester

every document is framable only from its own origin, and the framing policy finally covers all of them (ACTION REQUIRED)

Two changes to Content-Security-Policy: frame-ancestors, shipped together because
either one alone is misleading.

The policy now covers every document. It used to be set inside the route that
serves a package's public/ files, so that was the only place it applied. The
Console catch-all sent no framing header at all, which meant notebooks, dashboards,
models and the Explorer stayed framable from anywhere — including on a deployment
that had set PUBLISHER_FRAME_ANCESTORS and reasonably believed it had configured
this. One middleware now sets the header ahead of every route.

The default is now 'self', not *. A page is framable only from its own
origin unless a deployment says otherwise.

Migration. If nothing embeds Publisher in an iframe from another origin, there
is nothing to do. If something does, set PUBLISHER_FRAME_ANCESTORS to the
embedding origins:

PUBLISHER_FRAME_ANCESTORS="https://app.example.com"
PUBLISHER_FRAME_ANCESTORS="https://app.example.com https://admin.example.com"
PUBLISHER_FRAME_ANCESTORS="*"          # the previous behaviour, chosen deliberately

The value is a CSP source list and is passed through as written. A deployment that
already set this variable keeps working and now gets the coverage it expected; the
break is for one that relied on the * default without setting anything.

The symptom if you miss it: the embedded page renders blank or is blocked, and the
browser console names frame-ancestors. Nothing fails server-side, so there is no
log line to watch for.

---

compiling at the default scope no longer accepts text that declares its own data roots (ACTION REQUIRED)

POST /…/compile and the compile_model MCP tool default to scope: "append",
where the submitted text is a fragment checked against a model that is already
published. Malloy resolves a source's schema at COMPILE time -- duckdb.sql(…)
sends a DESCRIBE before any query runs, and connection.table(…) fetches the
table's schema the same way -- so a compile-only endpoint still reached the
database, the filesystem and the network on the caller's behalf. Against DuckDB's
external access that made unrestricted compile an oracle rather than a check:
read_csv('/etc/…') distinguished an existing file from a missing one by its
error and named the columns of whatever it read, and read_csv('https://…')
issued the request. No rows were returned, so the exposure was disclosure and
SSRF rather than extraction.

Append text is now compiled under the same restricted mode /…/query already
applies. These are refused with a 400:

• import
• connection.table(…) and connection.sql(…)
• given: declarations
• ##! compiler flags
• the raw-SQL function forms: name!type(…), sql_number, sql_string and the
  rest of that family

Migration. Those constructs belong in a model file, so send text declaring
them at scope: "file" (validating an edit to one file) or scope: "package"
(checking every file as saved). Neither is restricted. Two workflows this
changes in practice: validating a not-yet-saved dashboard, which opens with an
import, and validating a new source rooted in a table -- including the
source: line search_database_schema hands back. Both want "file".

Compiling at append also now requires the model named in the URL to load,
because the fragment is judged against that model's published surface and there
is nothing to judge it against otherwise. A model that does not compile
answers 400 carrying the model's own problems, which describe a file the caller
can already read. A model that does not exist answers 400 too, but
deliberately says only that it could not be loaded: naming what was wrong with a
path the caller supplied would answer "does this file exist" for any path.

Append text must also stand alone as top-level Malloy. A fragment that only
parses as a continuation of the model's last statement -- opening with
extend {, for instance -- is refused rather than compiled, because text that
does not parse on its own cannot be checked on its own.

file and package are unchanged and still unrestricted. scope is a
caller-chosen request field with no authorization difference between its values,
so this keeps fragment authoring on the model's published surface rather than
containing a caller who can simply ask for another scope.

---

per-user visibility through a materialized grant table, and a public wrapper served from its fact

A storage-materialized source may join a grant table that is itself scoped by givens. The
visibility idiom — an org-scoped source joining an org-and-user-scoped grant table, with a dimension
that null-checks the join — was refused as dynamic_joined_where. It is now admitted when the joined
source is itself persisted with storage=, joined by name, and admissible on its own. Both artifacts
hold every caller's rows; the serve shape re-applies the grant table's terms per caller and re-emits
the join against that binding, so two users of one org get different answers from the same two
tables. If the grant table's binding is withheld (stale past its window, unbuilt, refused), the
sources joining it serve live and their siblings keep the tier. A revoked grant stays visible until
the grant table rebuilds: give the grant table a freshness window with fallback="live", and do not
refresh it incrementally, since a deleted grant is never re-read by a delta — see
materialization.

The same rule lets a plain extension of a materialized source (source: v is opps extend { join_one: …; where: g.user_id = $USER_ID }) be served from its parent's table, which previously required
#@ -persist and so served live.

A public wrapper over a materialized private fact is served from the fact's table. A source whose
query reads a persisted source (orders is _orders_fact -> { select: * }) has no binding of its own,
so on a storage destination it was an undefined name on the serve shape and every query naming it
served live. The wrapper is now carried onto the shape verbatim when everything it reads is on the
shape; one that reaches a warehouse table or an unmaterialized source still serves live.

New plan field: PersistSourcePlan.joinedTerms names each caller-scoped join, the joined source,
and the terms its binding re-applies. It is optional and additive: the key is absent unless a source
declares such a join, so no existing plan changes shape. A consumer generating a strict client from
api-doc.yaml rejects the field until it regenerates.

---

a refused persist source is skipped, and no longer fails the whole run

Before: a materialization run stopped at the first persist source the eligibility gate refused. It built nothing, including every source the gate admitted, and ended FAILED with that one source's message. A single ineligible source therefore left the rest of its package unrefreshed on every run and every scheduled fire, until someone edited the model.

Now: a run skips a refused source and builds everything else. The run completes (MANIFEST_FILE_READY) and serves the refused source live, as before. Auto-run records each refused source in metadata.refusedSources, keyed by sourceID in the same shape as the build plan's refusedSources, and counts it in metadata.sourcesRefused. A build with caller-supplied buildInstructions reports an instructed source the gate refuses in the manifest's existing failures, under the instruction's sourceEntityId, with the gate's message as its reason and the new SourceFailure.refused: true, since that caller asked for the table; its siblings still build. refused tells a caller that retries failures this one will not clear until the model changes. It is the only schema addition, and it is optional.

A run still fails on a refusal when there is nothing else to build, because every source it targeted was refused, or when sourceNames names a refused source, because that caller asked for exactly the table that cannot be built. When several sources are refused, the error names all of them, not only the first in plan order. A refused #@ preaggregate rollup is recorded but never fails a run, as before.

What to check. Anything that read a FAILED run as the signal that a package holds an ineligible source should read metadata.refusedSources instead; the build plan's refusedSources reports the same refusals before any run. An auto-run whose only shortfall is refusals is metered success, since a refusal is a property of the model rather than of the run. An orchestrated run with an instructed refusal is metered partial, because that refusal is one of its failures: the caller asked for the table and did not get it. publisher_materialization_sources_total gains outcome="refused" and a mode label (auto | orchestrated) on every outcome. With mode="orchestrated" the refused count should stay at zero: a caller that builds from the build plan never instructs a source the plan refused, so a nonzero count means the plan and the build disagreed about a source.

What's Changed

• fix(materialization): a refused persist source is skipped, so it no longer stops the rest of its package building by @housejester in #1218
• fix(skills): eight harness defects from a real-model eval pass by @girishjeswani in #1226
• feat(materialization): per-user visibility through a materialized grant table, and a public wrapper served from its fact by @housejester in #1219
• chore(release): stamp 0.7.0 on main by @jswir in #1225
• fix(release): pin the server the release shipped, and check it from outside the checkout by @jswir in #1224
• fix(server): compile caller fragments under restricted mode at append scope by @sagarswamirao in #1199
• fix(server)!: frame every document with one policy, and default it to 'self' by @sagarswamirao in #1230

Full Changelog: v0.7.0...v0.8.0