Repository navigation
v0.8.3
Release v0.8.3
📦 NPM packages
🐳 Docker image
docker pull ms2data/malloy-publisher:0.8.3
docker pull ms2data/malloy-publisher:latest
ℹ️ Details
- Release branch:
release/sdk-0.8.3 - Triggered by: @housejester
a partitioned storage build no longer runs out of memory on a wide, many-partition source
A #@ persist partition= build of a wide source with many partition values failed against the
build session's memory_limit — Out of Memory Error: failed to pin block of size 256.0 KiB —
before or shortly after its first rows, where the same source built unpartitioned. The build is
DuckDB's partitioned COPY, which buffers rows per partition inside the buffer manager, charges each
appender one vector per column for every partition it has met, and flushes nothing until it has
appended 524,288 rows. Rows arriving interleaved across partitions brought every partition into
that set at once.
Three changes, on a passthrough-sourced partitioned build only (Postgres, BigQuery, Snowflake).
The insert's SELECT is now ordered by the partition columns at the top of the INSERT statement, so
the build holds one or two partitions at a time; the SELECT it was handed is unchanged, so what the
warehouse runs, its query tag or label, and what the source is addressed by are untouched. That
insert runs on one thread, because a DuckDB-side sort is read in parallel and several appenders each
meet every partition again. And the build session sets partitioned_write_flush_threshold, new
PUBLISHER_PARTITIONED_WRITE_FLUSH_THRESHOLD (rows, default 8192; off turns all three off, for a
source that built fine before). The sort reads the whole result before the first partition file is
written and spills to the build's working directory, so memory is traded for local disk and wall-clock.
An incremental delta into a laid-out table gets the threshold and the single thread through its session.
Measured on 122 columns, 616k rows, 308 partitions at 768MB: from Postgres, interleaved fails at any
threshold, ordered fails at DuckDB's default, ordered on four threads fails, ordered on one thread
with the bound completes in 9.5 s; from BigQuery, interleaved fails at 54 s and the same shape
completes in 36 s. An unpartitioned build, and a chained build, are byte-identical to what they
were.
/status names the server version, and says why it is empty
GET /api/v0/status and the get_status MCP tool now report version, the server's release, and
the MCP handshake reports the same value instead of 0.0.1. A stale copy from the npx cache is
now visible from outside the process.
A server that found no publisher.config.json still reports serving, which is deliberate: some
deployments start empty and create environments over the API. It now also reports emptyReason,
naming the path it checked. It is set for a mistyped --config path too. The field is absent once
any environment exists, and absent when a config was found but lists none.
a reloaded package keeps its warm semantic index, and embeddingIndex.status means what it says
Reloading a package no longer costs you a lexically-ranked answer. A reload
never dropped a package's vectors — they are keyed by package name in
publisher.db — but it did throw away the server's record that they were
current, because that record was tied to the in-memory package object a reload
replaces. So the first get_context after every reload_package, every REST
?reload=true, and every watch-mode save was ranked lexically while the server
re-checked hashes that all still matched. If you author models with watch mode
on, that was one degraded answer per save, including saves that changed nothing
relevant. Now a reload whose files hash the same keeps the warm index and is
ranked semantically on its first call; an edit still re-embeds, and still only
the parts whose text changed.
embeddingIndex.status keeps its name and changes its basis, so read this if
you poll it. On the package resource
(GET /api/v0/environments/{env}/packages/{pkg}), ready used to be derived
from whether cached rows covered the package's current entity names. Vectors
outlive a restart and a reload, so that reported ready immediately — while the
next question was still answered lexically. Anything following the documented
"poll until ready before measuring retrieval quality" could therefore measure a
lexical run and record it as a semantic one, which is a wrong number rather than
a slow start. ready now means one thing: the index is warm, so the next
get_context question about this package is ranked semantically. It is decided
by the same completed sync the search path itself gates on. It describes the
index, not the next response — a question whose own query embedding fails still
falls back, with retrieval_reason: provider-error.
What to do. If you poll for readiness, keep polling status — it is now
accurate, and it is the field to trust. If you instead inferred readiness from
embeddedEntities == totalEntities, stop: those count coverage by entity name,
so they can be equal while status is indexing (an edit that rewrote every
doc without renaming anything leaves each entity holding its stale name vector).
Expect status to read indexing in two places it previously read ready:
just after a server restart, until the first question re-establishes the sync,
and after a doc-only edit. Both clear on the next get_context question.
Because only a question starts the sync, poll in this order: send the package one
get_context question, then poll until ready. That holds after every restart,
not only for a package nothing has queried. A script that polls before asking
anything never sees ready.
If your embedding provider ignores EMBEDDING_DIMENSIONS, the coverage counts
now match reality. The dims column records the length the provider actually
returned, and some providers (Ollama among them) ignore the requested value.
embeddedRows and embeddedEntities were counted against the configured
value instead, so for those providers they read 0 while retrieval was reading
those same vectors happily — and that also pinned status at indexing. Both
now count on the same rule the sync uses to decide a row is current: the current
model, any vector length.
That makes them a count of what is cached, not a prediction of what a search can
read — the scan also matches on vector length, which only a real question knows.
So after a change to EMBEDDING_DIMENSIONS that no question has probed yet, the
old rows are still counted until the next search discards them. status is
already indexing throughout that window, which is why it, and not the counts,
is the field to poll.
Unrelated to the above, and unchanged: --init still drops the vector cache
along with the rest of persisted storage. It resets the server root, and it
remains the reclaim path for rows orphaned by a configuration change.
a given the query reads is no longer silently replaced by its default
Publisher withholds a given the entry model doesn't surface when a gate is the only thing reading
it, so the gate can still evaluate. It also withheld it when the query itself read a given of the
same name, such as a where: on a source from another file that declares its own defaulted
HIDE. That where: then ran at its default: a caller who sent HIDE: us-west got the rows for
'none', with no error. The value is now forwarded, so the request returns a 400 (unknown given 'HIDE'), the same as for any given the entry model doesn't surface. Behavior change: a
request that used to return rows at the default now fails with a 400. To fix the model, import the
given at the entry model.
a gate on a joined field checks what the join reads
The filter-binding check introduced in 0.8.1 now also compares, for a gate on a joined field such
as #(access_filter) child.org_id in $GROUPS, what decides which joined row each row reaches. That
covers the columns the join's ON or with reads, and the joined source's own where:,
parameters and arguments. Behavior change: a query or a model extension that redefines one of
those answers 403 on every query, even when the new definition is equivalent
(rename: raw_id is id; dimension: id is raw_id), as a redefinition of the gated column itself
already did. If an extension of a gated source needs a reshaped join key, give the new dimension a
new name. A caller who re-joins exactly the joined source the author declared, with the same ON,
is now served rather than refused. Fixes a pre-existing gap, present since 0.8.2 or earlier: a
gate reading into a record-literal field named location (dimension: x is { location is … })
was comparable-as-equal regardless of value, because the field-identity check stripped any key
named location by name rather than by shape; a caller could redefine that field and see rows the
gate should have hidden.
What's Changed
- chore(release): stamp 0.8.2 on main by @housejester in #1243
- feat(eval-loop): lead the run report with what to fix, and open each case in a panel by @jswir in #1234
- fix(server): don't replace a given the query reads with its default by @Sha-Bang in #1242
- fix: a true/false dropdown for filter givens, and a 400 that says why a filter value is wrong by @jswir in #1239
- fix(server): keep a reloaded package's warm embedding index by @jswir in #1131
- ci(release): pause the python-client PyPI publish while its upload is refused (work in progress) by @sagarswamirao in #1257
- ci: scan the source tree and Docker image with Trivy, and clear their critical findings by @sagarswamirao in #1251
- ci: attach an SBOM and pin max provenance on the published Docker image by @sagarswamirao in #1252
- docs, test: Parquet --data proven end to end, a stdlib Python example, and the npm name trap by @jswir in #1248
- fix(server): /status names the server version, and says why it is empty by @jswir in #1246
- fix(skills, mcp): stop telling agents to halt, or to make a call that fails, when a tool covers the case by @jswir in #1247
- ci(release): version skills and create-malloy-package at release time, not in PRs by @Sha-Bang in #1256
- feat(security): add a skill for fixing security scan findings by @sagarswamirao in #1254
- fix(server): compare the inputs of a gated path's joins in the filter-binding check by @Sha-Bang in #1250
- feat(skills): move a Power BI model onto Malloy, and prove the numbers match by @Sha-Bang in #1229
- fix(server): insert a partitioned storage build in partition order, on one thread, with a bounded flush threshold by @housejester in #1260
Full Changelog: v0.8.2...v0.8.3
