Skip to content

v0.9.5

Choose a tag to compare

@github-actions github-actions released this 07 Oct 19:09

Release v0.9.5

📦 NPM packages

🐳 Docker image

docker pull ms2data/malloy-publisher:0.9.5
docker pull ms2data/malloy-publisher:latest

ℹ️ Details

  • Release branch: release/sdk-0.9.5
  • Triggered by: @housejester

A per-package connection call naming a package the server does not hold answers 404, not 400

A call to a per-package duckdb connection route (.../packages/<pkg>/connections/duckdb/...: sqlSource, sqlQuery, sqlTemporaryTable, schemas, tables and the table lookup) that names a package this server does not hold now answers 404, not 400. The message is unchanged: Package "<pkg>" not found in environment "<env>".

The usual cause is a package version that was replaced and unloaded while a caller still held its old name. A router that caches which server holds which package can treat the 404 as "look again", which it could not do with a 400. If you branch on the 400 for this case, branch on 404 instead.

A named package on a server that holds no packages at all also answers this 404. It used to fail with an untyped error. A duckdb call that names no package, on a server holding several, is still a 400.


SDK: the builder's add-tile imports a source from the model that exports it

Adding a tile on a source now writes an import Malloy accepts. The catalog used to credit a source to the first model whose sources listed it, and that list includes names a model only imports, so a tile on order_items could write import { order_items } from "../data_app.malloy" ("Reference to undefined object"), or add a named import to a file that already did import "../storefront.malloy" ("Cannot redefine"). A whole-file import of any model that exports the source now counts as seeing it, and Save accepts a tile on it without a named import.

Public SDK surface: buildCatalog credits a source only to a model whose modelInfo exports it, so a model without modelInfo now contributes no catalog sources, and a host passing a hand-built PackageCatalog should set modelPath to the exporting model. CatalogSource gains optional exporters (every model that exports the source; absent means [modelPath]) and visibleIn (every model that lists it), and NewTile carries exporters. What to do: a host that builds its catalog from models fetched without modelInfo will see an empty tile picker after upgrading, so keep modelInfo on the models it passes to buildCatalog, or pass a hand-built catalog with modelPath set to the exporting model. A document held as text (DashboardEditor textSource) is offered only the sources its run model lists and adds no import.


SDK: KPI strips line up, the range slider looks like its neighbours, and the storefront charts read cleaner

  • KPI strip. A renderer # big_value strip now lays out as a grid with cards of one width, instead of a wrapping row sized to each card's text that left a lone fifth card. A nested (embedded) big_value is unchanged. The column floor is 250px so a wide value never spills onto its neighbour.
  • Range slider. A range_min/range_max control is drawn as an outlined field with the label notched into its edge and min/max captions under the track. Labels, readouts and aria names are unchanged. It is taller than a text field, so a neighbour's helper line sits under its own box.
  • Builder defaults. The builder no longer shows Default: (empty) under a filter control the viewer shows nothing for, and an unwrapped filter default beginning with f (fall) is no longer shown as all.
  • Shape maps. A # shape_map legend now sits below the map, which draws 512px wide instead of 588px, so it fits a narrower tile. A tile narrower than that still clips it.
  • Storefront example. Chart views tag # currency=usd0, so axes read $130,000; tables and KPI cards keep their cents. Category labels on the bar charts are thinned less, not never: the renderer's label-overlap rule is not configurable.

A package stays listed while it reloads, and an install is refused under memory back-pressure

Every listed package now carries a status object with two independent facts: serving, whether a compiled copy answers queries on this server, and loading, whether a load, reinstall or recompile is in progress here, with loadingSince while it is. A package that is reloaded while it serves reports both, because the previous copy keeps answering until the new one is swapped in. Until now the listing, and so /status, left such a package out for the whole compile, which read as the package having left the server; an orchestrator that reads /status to place packages took it that way, unloaded the replica, and the reload in flight then failed for nothing. A package loading for the first time is listed only on request, as its name and status on GET /status?includeLoading=true, so an orchestrator that reads status can tell a dispatched load from an absence; every other listing shows only packages that can be queried, as before, since a listed package has always meant one that can serve.

Three install behaviours change with it:

  • An install, whether a POST or a PATCH with a location, is refused with a 503 while the memory governor reports back-pressure, before the download starts, the way a lazy load and an add already were. A reinstall holds the new compiled copy beside the serving one until the swap, so it was the one allocation the governor could not see.
  • A PATCH whose location matches the one the package was installed from is a metadata update, not a reinstall: the manifest is rebound in place and nothing is downloaded or recompiled beyond what the manifest itself requires. A PATCH that arrives while any install of the package is in progress, downloading or compiling, a first install or a reinstall, waits for it and is decided against the copy that install leaves resident, so it lands on the installed copy, and a PATCH naming a location whose install failed installs it. Only an install records a location; a metadata PATCH never changes it. To make the comparison hold across an in-place reload and a restart, the server records the install location in its own file outside the package directory, <environment>/.install-records/<package>.json, and reads nothing inside the package as one, neither a location an author writes into publisher.json nor a record shipped with the content, since a reload fetches from it. GET …?reload=true and the reload_package tool re-fetch from the recorded location and re-apply the manifest binding; that is the way to fetch the same location again.
  • A POST that carries a manifestLocation binds it as part of the install. The downloaded tree's own publisher.json does not carry it, so until now the package came up serving live and was rebound, with a full reload, by the next drift check.

A metadata PATCH also preserves what it does not mention: description, resource and location were replaced by the body's absent values, and publisher.json lost its description. A field sent as null counts as not mentioned, the rule scope already followed, so a client that serializes unset fields as null does not blank them; an empty string is a value. And the metadata sent with an install is applied inside the install's own lock hold, so a delete queued behind the swap can no longer run between the two and leave the update answering 404 for an install that had completed.

A package installed before this release has no recorded location, so its first PATCH that names its location reinstalls it once and records it; every later one is a metadata update.


a storage= source reads its stored upstreams through the sources between them, and a build reports whether it read them

A storage= source that depends on another stored source is built by reading that source's table in the destination. Until now that worked only when the dependent source named the stored one directly: a non-persisted source in between — checklists_kept is checklists_all extend { … } over a stored site_seasons, or a select: * wrapper the public sources are declared on — was absent from the model the build compiled, so the build failed with Reference to undefined object, and under strictUpstreams (every orchestrated build) the whole run failed with it. The build now carries those intermediate sources into the model, in dependency order and under the ##! flags of the files that declare them, so a dependent source reads its stored upstreams however many non-persisted sources stand between them. The model that build compiles carries only the sources on the dependent's own path — a source elsewhere in the model that cannot be carried no longer fails a chain it is not part of — declares the author model's given:s so a carried source whose text reads one compiles, and sees through an inline parenthesized extension (hits is (daily extend { … }) -> { … }) to the base and joins it reads. It also sees a stored parent reached only through an import — import { weekly } from "orders.malloy" leaves daily, which weekly is built from, out of the importing model's namespace — and stacks on its table rather than recomputing it; before, such a parent read as a warehouse table. A storage= source over a partition= upstream is now built from that upstream's table too: the partition layout is part of the address the publisher files the entry under and no part of the key the compiler substitutes by, so such a downstream was always recomputed from the warehouse — refused outright under strictUpstreams — and now stacks on the table like any other chained source.

An intermediate that reaches the source warehouse — one that joins a table nothing materializes — cannot be carried, since the destination holds no such table. Such a source is now recomputed from the warehouse, with the upstream inlined, and the build succeeds; under strictUpstreams this is the one recompute strict permits, because no build over the stored tables exists for it. Strict still refuses what it exists to refuse — anything whose recompute would rebuild a table the orchestrator pinned: a persisted upstream the build neither materialized nor was handed by reference, including one whose table lives in another destination, and a source that reads only stored upstreams yet could not be built over them (a refinement the build does not carry), where the limit is the build's rather than the source's.

Every manifest entry for a source that reads a persisted upstream now carries upstreamReuse: reused when every upstream was read from its materialized table, recomputed when one was recomputed from its definition, with upstreamRecomputeReason saying which and why. It is reported on both tiers, since a colocated build also inlines an upstream it cannot read. The field is not a location and never names a table. A recomputed source's rows reflect the warehouse at that build's time rather than the upstream's snapshot, so the two can disagree when their refresh cadences differ; this is how to tell. The publisher_storage_chained_build_total counter gains the outcome strict_shape_fallback for the strict recompute. Note what that changes: strictUpstreams no longer means a build never recomputes a stored upstream — for a source that reaches the warehouse, it recomputes and says so, and the warehouse does work the orchestrator did not schedule. strict_shape_fallback is the signal to watch; an orchestrator that must refuse it has no knob yet.

A chained build re-declares on each parent's binding what the parent's source adds to its table — its extend-block where:, dimensions, measures, joins and views — the same way the serve shape does, and thins them through the same tiers when they do not compile: a view, then a join, then a dimension or measure on the parent that reads a join to a source the destination cannot bind drops out; the where: never does. A colocated delta or skip entry carries the same upstreamReuse its rebuild would. A persist source's build SQL is the persisted relation alone, so a stored source declared extend { where: total > 100 } holds every row and is filtered when read; a downstream built over its table without that filter summed the unfiltered rows. Strict's missing-upstream verdict now comes from the compiler's own strict render over the run's whole manifest — the persisted source the SQL actually reads that nothing supplied — so a dependency reached only through a join declared on a stored upstream is refused as it was, and a declared join the query never reads is not.

The storage serve shape now compiles under the ##! flags of the author files whose declarations it carries, and carries an ungrouped-aggregate measure (total.sum() / all(total.sum())) as a measure. Before, a non-persisted wrapper written under access_modifiers (daily_public is daily -> { select: * } include { public: … }), or a source with an all(…) measure and a view that read it, lifted into text the shape could not compile — and one such lift withheld every lifted source in the model, so every query on the public surface ran live against the warehouse while the stored tables sat unread.


BigQuery connections take their project from the service account key again

Since 0.0.196, a BigQuery connection configured with serviceAccountKeyJson and no defaultProjectId has given Malloy no project id. Without one, compiled SQL qualifies every dataset.table path with the BigQuery SDK's placeholder {{projectId}}. Queries the SDK sends itself still work, because the SDK rewrites that token in each request. Anything that sends compiled SQL another way does not, most visibly a build into a storage destination, which runs it through DuckDB's bigquery_query() and fails with Invalid project ID '{{projectId}}'. The connection now takes the key's project_id again, and a blank defaultProjectId falls back to it as well.

The project id is part of the connection digest, so persisted sources on such connections get new BuildIDs and are rebuilt on their next build. Connections that set defaultProjectId are unaffected.

What's Changed

  • fix(server): take a BigQuery connection's project from its service account key by @housejester in #1313
  • docs(readme): add badges like the main Malloy repo by @kylenesbit in #1322
  • fix(sdk): builder add-tile imports from the exporting model, and a visual pass over KPI strips, the range slider, map legends and chart axes by @Sha-Bang in #1321
  • fix: correct wrong facts in the analysis skills, accept givens in the eval ledger, and show compiler messages on curated packages by @jswir in #1305
  • skills: cut the default lists to 8 analysis and 22 modeling skills, and fix wrong facts and rules by @jswir in #1320
  • feat(server): build a chained storage= source through the sources between it and its stored upstreams by @housejester in #1318
  • feat(server): add setupSQL to environment-level DuckDB connections by @housejester in #1328
  • ci(windows): bake and load DuckDB extensions from the D: drive by @sagarswamirao in #1324
  • ci: run the timezone SDK specs on Windows, trim them everywhere, and fail unreachable Postgres without a refused connect by @sagarswamirao in #1325
  • fix(server): answer 404, not 400, when a per-package connection names a package the server does not hold by @jswir in #1315
  • fix(server): report serving and loading per package, keep a reloading package listed, and rebind manifests without reinstalling by @housejester in #1323
  • chore(release): stamp 0.9.2 and 0.9.4 on main by @housejester in #1341

Full Changelog: v0.9.4...v0.9.5