v0.33.0
v0.33.0
The range starts at d81c107 (#347), the first commit after the v0.32.0
tag, and ends at this file: 12 changes plus their notes. The headline is
OneLake security, which arrives as a whole layer rather than a feature: the
policy model, its enforcement in the engine, and the platform block that makes
a raw read of a secured table fail the way it does in Fabric.
Read the first section if you use OneLake security roles. A secured Spark
session now runs in a process of its own on an engine of its own, holding a
token minted for the caller. That is a behaviour change and a resource change,
and it is on by default.
Read "Two error documents reflected their input" if you expose this emulator
to anything you do not control — two CodeQL-reported XSS holes in the DFS and
Blob error paths.
Read "A task's printed output could appear under another task" if you run
more than one task at a time. The v0.32.0 notes closed with this under Not
fixed here; it is fixed here.
OneLake security, end to end (#352, #354, #357, #358)
Fabric's OneLake security is item-scoped RBAC over a lakehouse: deny by
default, roles carrying row filters as SQL text and column lists, evaluated per
principal. This release implements the model, enforces it in the engine, and
blocks the reads that cannot be filtered.
What a Viewer under a narrowing role now sees:
SELECT count(*) FROM sales -> 2 of 3 rows (RLS)
SELECT * FROM sales -> region_id only (CLS)
SHOW TABLES -> ungranted tables are absent, not merely unreadable
spark.read.load("abfss://…") -> 403 Forbidden (blocked by OneLake)
The owner is untouched: a role narrows the principal it names, and workspace
Admin, Member and Contributor are not restricted by RLS or CLS, which is the
product's rule and not our simplification.
Three pieces are worth knowing about because they change how the stack behaves.
Every qualified name of a secured table is swept from the session. A temp
view shadows an unqualified name only, and the agent registers each table into
its lakehouse schema and into default so unqualified names resolve the way
they do in a lakehouse-attached notebook. That convenience registration was a
way around the filter — measured at 3 rows and both columns where the view gave
2 rows and one — and is now removed for any table a role narrows.
A secured session runs in a user context: its own process, on its own
engine. This is what makes a path read arrive at OneLake as the caller and be
refused. It is also the shape Fabric has — it starts a Spark session per
notebook and shares one only within a single-user boundary — where this
emulator previously served every caller from one engine holding a service
credential. The process holds neither that credential nor the client secret
that mints one.
The privileged half supplies the filtered rows. The user context cannot
read a narrowed table by design, so the system context reads it, applies the
filter and sends the result across as Arrow. Nothing is staged in storage,
which is also what Fabric does: its system context returns rows, not paths.
What this costs
- The agent image grows 1.08 GB → 1.21 GB. It now carries
pysail, so the
agent can start an engine. - A secured session starts an engine per user, measured at ~66 MiB resident
and flat across repeated work. Engines are keyed by principal and shared
across that principal's sessions, so the bill scales with identities rather
than notebooks. - It engages only where there is policy. A statement is treated as secured
only when it names a principal, a workspace and an item, which this emulator
sends only for an item that has data access roles. A stack with no roles
anywhere starts no engines and behaves exactly as before — measured, with the
notebook-drivensuite passing with zero engines started. FABRIC_TWO_CONTEXT=0opts out, and gets the previous, weaker behaviour
knowingly rather than by pinning an old image.
Boundaries, stated
docs/54-onelake-security.md carries the full list. Two worth repeating: the
DataFrame reader is covered through the catalog, not by intercepting
spark.read; and the filtered relation crosses in memory, so it is bounded by
localRelationSizeLimit — a ceiling that must fail loudly rather than
truncate, because a security control that silently returns the first N rows is
worse than one that refuses.
T-SQL security in the Warehouse (#352)
CREATE SECURITY POLICY … FILTER PREDICATE, GRANT/DENY SELECT ON t(col) and
ALTER TABLE … MASKED WITH are enforced by SQL Server itself, because each
caller now connects as its own database principal through the TDS relay. Two
callers, one query, different answers — witnessed by a real go-mssqldb
client, with masking observed as "aXXX@XXXX.com" against "ada@example.test".
A different mechanism from OneLake security's RLS, and not
interchangeable: this one is defined in SQL and applies to a Warehouse or SQL
analytics endpoint, while OneLake security covers Lakehouse-type items across
every engine and does not cover Warehouse at all.
An owner (workspace Admin or Member) gets db_owner, because somebody has to
be able to author a policy; a writer deliberately does not, because CONTROL
implies UNMASK and a writer would see through every mask.
Two error documents reflected their input (#355, #356)
CodeQL alert 71: the DFS error path wrote a caller-supplied path segment into a
JSON document with %q, which escapes for Go string literals and not for
</>. No item matches <seg> was therefore reflected. The DFS document is
now built with encoding/json, which escapes those, and the response refuses
MIME sniffing. The Blob dialect had the same shape — an fmt.Fprintf with two
bare %s into an XML document, fed the same value — and was missed by a search
that stopped at onelake.go.
A task's printed output could appear under another task (#359)
The v0.32.0 notes recorded this under Not fixed here. redirect_stdout
assigns sys.stdout, one attribute on one module per interpreter, and the
agent serves overlapping statements — so of three concurrent tasks that each
printed, one response carried another's output and two carried nothing. The
writes were correct; only the attribution was wrong.
A proxy in the sys module dict now routes each write to the running statement's
buffer through a ContextVar, so nothing is saved or restored and no statement
can restore over another. Worth recording for anyone attempting the same fix:
a property on the module's type — the mechanism that works for sys.argv —
does not work here, because print reads stdout from the module dict and
never consults the type.
/statements says what it drops (#360)
The route accepted env and spark_conf and applied neither, returning
{"status":"ok"} with the field discarded. Both are now named in the agent log
with the reason, along with any field the route does not recognise.
Named rather than refused, deliberately: every released databricks-emulator
still sends both, so refusing would break the callers that exist today. Neither
will be implemented — Fabric's Livy statement payload is {"code", "kind"},
and a statement-level spark_conf is not statement-scoped: it outlives the
statement on the session's Spark session, and on a shared session it leaks into
every other one. A task's environment belongs in the code it runs, where no
agent can drop it.
Release dispatch reached one platform of three (#347)
The dispatch target had been renamed, and every release since succeeded on
GitHub's redirect — so the release workflow reported success while two of the
three platforms were never told at all. Their acceptance runs simply did not
happen, and nothing said so. Platforms are now dispatched by their real names,
and a name that does not resolve fails the release instead of redirecting
quietly.
OpenMetadata comes from the family's registry (#350)
docker.getcollate.io failed two platform nightlies inside an hour. The images
are mirrored into GHCR by the hub with index and digest intact, so the
governance profile now pulls from there — a vendor registry having a bad
morning should not decide whether the family's nightlies run.
The sidecars are referenced by their current names (#348)
emulator-spark-agent and emulator-sail have been the names since v0.26.0.
The fabric-emulator-* aliases stay published, so nothing breaks; this only
stops new references asking for them. Historical accounts that name the old
image keep it.
A spike, kept (#353)
e2e/sail-session-isolation answers one question by measurement — can two
Spark Connect sessions against Sail be isolated — and it decided the
architecture above: newSession() is JVM-only and silently degraded on Sail,
so nothing was ever isolated there, while builder.create() does isolate. It
ships as a harness rather than a parity witness, and exits 0 whatever it finds,
because a spike reports and does not gate.
Consumers should bump their pinned digests
emulator-sail and emulator-spark-agent are tagged for the dependency they
carry, not their content: both also ship first-party code that changes
independently, so this release republishes the same tags over different bytes.
A consumer pinned by digest — the recommended form — stays on the previous
image until it bumps.
The platform repositories now pin every image they pull by digest, with the
version beside it for readability, and move the two together. If you maintain a
consumer that does not, repo:tag@sha256:… is the form to adopt: docker
ignores the tag and fetches the digest, so a version bumped without its digest
runs the old image under the new name.