v0.34.0

The notebookutils release. The shim a Fabric notebook imports went from
partial and partly-wrong to the whole documented surface — 44 members across 8
namespaces, each exercised end to end — and closing it turned up ten defects
that unit tests could not see.

⚠️ Upgrade first for this

notebookutils.fs.rm(path) on a directory deleted the entire subtree.
OneLake's DELETE ignored ?recursive= completely, so a bare rm took the tree
with it. ADLS Gen2 answers 409 DirectoryNotEmpty; the emulator was more
destructive than the thing it emulates, on the one filesystem operation with no
undo. Fixed: a non-empty directory is now refused unless recurse=True.

Breaking changes

All are corrections toward Fabric's documented contract — a framework
introspects these signatures and refuses to start when a parameter name differs
— but code written against v0.33.0 may need edits.

v0.33.0	v0.34.0	Note
fs.put(path, content, overwrite=True)	put(file, content, overwrite=False)	default changed; an overwrite must now be asked for
fs.append(path, content)	append(file, content, createFileIfNotExists=False)
fs.head(path, maxBytes=)	head(file, max_bytes=)
fs.cp(src, dst) / mv	cp(src, dest) / mv
fs.rm(dir) removed subtrees	rm(path, recurse=False) refuses a non-empty directory	see above
lakehouse.get(lakehouseId)	get(name)	addressed by NAME, as documented; an id still resolves

Positional callers are mostly unaffected; callers passing these by keyword are
not.

Fixed

• fs.rm destroyed directory trees (above).
• session.stop() and session.restartPython() never worked inside a
  session. Both read environment variables that nothing in the tree ever set,
  so they raised "this is running outside a notebook session" — inside one.
  They are now bound per statement, because one agent serves many concurrent
  sessions and a process variable would let stop() in one notebook end
  another's.
• restartPython() left the session without display, displayHTML and
  %run. Those are kernel builtins on Fabric; a Python restart cannot remove
  them.
• A referenced notebook resolved builtin/ to its own folder. nbResPath
  means the ROOT notebook's resources, and nothing ever sent a root — so a
  notebook read different files depending on how it was started.
• A cell's language was classified and then ignored. The run loop sent
  everything that was not sql to the Python executor, so correct Scala failed
  with a Python SyntaxError pointing at the user's own code, and a
  %%configure block of JSON failed the same way.
• The long-running-operation Location header always said https://, so a
  client following it — the documented route to a result — could not reach an
  emulator started with -disable-tls.
• display() and displayHTML() were absent entirely, raising NameError
  on one of the most common lines in any Fabric notebook.

Added

• The whole documented notebookutils surface: 44 members over fs,
  notebook, credentials, lakehouse, runtime, session, udf and
  variableLibrary, with the documented parameter names in the documented
  order. Every member is exercised end to end, not merely present.
• help() on every module, plus getHelpString — the discovery mechanism
  Fabric's own fs page opens by documenting. Derived by introspection, so it
  cannot drift from the code.
• Notebook item management (create, get, list, update, delete,
  getDefinition, updateDefinition), lakehouse definition round-trips,
  udf.run, runtime.getCurrentWorkspaceId, fs.refreshMounts.
• display() publishes rich output under a kernel — text/html with a
  text/plain alternative — and prints without one. The shipped JupyterLab
  binds Fabric's display, not IPython's.
• Microsoft's own stubs are vendored (third_party/notebookutils-stubs/)
  and held beside the documentation, so where the two Microsoft sources
  disagree the divergence is computed rather than assumed.

Honest limits

display() renders a correct HTML table, not Fabric's interactive widget
with its chart views and inspect panel; no local front end can prove
equivalence with that. The Files mount is a point-in-time copy, not blobfuse.
%%configure is accepted and ignored, out loud. Scala, R and C# cells are
refused by name rather than mis-executed.

And the standing one: everything here is verified against Microsoft's published
contract, not against a tenant. It means conforms to the documentation — never
matches Fabric.

v0.33.0

v0.33.0

The range starts at d81c107 (#347), the first commit after the v0.32.0
tag, and ends at this file: 12 changes plus their notes. The headline is
OneLake security, which arrives as a whole layer rather than a feature: the
policy model, its enforcement in the engine, and the platform block that makes
a raw read of a secured table fail the way it does in Fabric.

Read the first section if you use OneLake security roles. A secured Spark
session now runs in a process of its own on an engine of its own, holding a
token minted for the caller. That is a behaviour change and a resource change,
and it is on by default.

Read "Two error documents reflected their input" if you expose this emulator
to anything you do not control — two CodeQL-reported XSS holes in the DFS and
Blob error paths.

Read "A task's printed output could appear under another task" if you run
more than one task at a time. The v0.32.0 notes closed with this under Not
fixed here; it is fixed here.

---

OneLake security, end to end (#352, #354, #357, #358)

Fabric's OneLake security is item-scoped RBAC over a lakehouse: deny by
default, roles carrying row filters as SQL text and column lists, evaluated per
principal. This release implements the model, enforces it in the engine, and
blocks the reads that cannot be filtered.

What a Viewer under a narrowing role now sees:

SELECT count(*) FROM sales        -> 2 of 3 rows        (RLS)
SELECT * FROM sales               -> region_id only     (CLS)
SHOW TABLES                       -> ungranted tables are absent, not merely unreadable
spark.read.load("abfss://…")      -> 403 Forbidden      (blocked by OneLake)

The owner is untouched: a role narrows the principal it names, and workspace
Admin, Member and Contributor are not restricted by RLS or CLS, which is the
product's rule and not our simplification.

Three pieces are worth knowing about because they change how the stack behaves.

Every qualified name of a secured table is swept from the session. A temp
view shadows an unqualified name only, and the agent registers each table into
its lakehouse schema and into default so unqualified names resolve the way
they do in a lakehouse-attached notebook. That convenience registration was a
way around the filter — measured at 3 rows and both columns where the view gave
2 rows and one — and is now removed for any table a role narrows.

A secured session runs in a user context: its own process, on its own
engine. This is what makes a path read arrive at OneLake as the caller and be
refused. It is also the shape Fabric has — it starts a Spark session per
notebook and shares one only within a single-user boundary — where this
emulator previously served every caller from one engine holding a service
credential. The process holds neither that credential nor the client secret
that mints one.

The privileged half supplies the filtered rows. The user context cannot
read a narrowed table by design, so the system context reads it, applies the
filter and sends the result across as Arrow. Nothing is staged in storage,
which is also what Fabric does: its system context returns rows, not paths.

What this costs

• The agent image grows 1.08 GB → 1.21 GB. It now carries pysail, so the
  agent can start an engine.
• A secured session starts an engine per user, measured at ~66 MiB resident
  and flat across repeated work. Engines are keyed by principal and shared
  across that principal's sessions, so the bill scales with identities rather
  than notebooks.
• It engages only where there is policy. A statement is treated as secured
  only when it names a principal, a workspace and an item, which this emulator
  sends only for an item that has data access roles. A stack with no roles
  anywhere starts no engines and behaves exactly as before — measured, with the
  notebook-driven suite passing with zero engines started.
• FABRIC_TWO_CONTEXT=0 opts out, and gets the previous, weaker behaviour
  knowingly rather than by pinning an old image.

Boundaries, stated

docs/54-onelake-security.md carries the full list. Two worth repeating: the
DataFrame reader is covered through the catalog, not by intercepting
spark.read; and the filtered relation crosses in memory, so it is bounded by
localRelationSizeLimit — a ceiling that must fail loudly rather than
truncate, because a security control that silently returns the first N rows is
worse than one that refuses.

T-SQL security in the Warehouse (#352)

CREATE SECURITY POLICY … FILTER PREDICATE, GRANT/DENY SELECT ON t(col) and
ALTER TABLE … MASKED WITH are enforced by SQL Server itself, because each
caller now connects as its own database principal through the TDS relay. Two
callers, one query, different answers — witnessed by a real go-mssqldb
client, with masking observed as "aXXX@XXXX.com" against "ada@example.test".

A different mechanism from OneLake security's RLS, and not
interchangeable: this one is defined in SQL and applies to a Warehouse or SQL
analytics endpoint, while OneLake security covers Lakehouse-type items across
every engine and does not cover Warehouse at all.

An owner (workspace Admin or Member) gets db_owner, because somebody has to
be able to author a policy; a writer deliberately does not, because CONTROL
implies UNMASK and a writer would see through every mask.

Two error documents reflected their input (#355, #356)

CodeQL alert 71: the DFS error path wrote a caller-supplied path segment into a
JSON document with %q, which escapes for Go string literals and not for
</>. No item matches <seg> was therefore reflected. The DFS document is
now built with encoding/json, which escapes those, and the response refuses
MIME sniffing. The Blob dialect had the same shape — an fmt.Fprintf with two
bare %s into an XML document, fed the same value — and was missed by a search
that stopped at onelake.go.

A task's printed output could appear under another task (#359)

The v0.32.0 notes recorded this under Not fixed here. redirect_stdout
assigns sys.stdout, one attribute on one module per interpreter, and the
agent serves overlapping statements — so of three concurrent tasks that each
printed, one response carried another's output and two carried nothing. The
writes were correct; only the attribution was wrong.

A proxy in the sys module dict now routes each write to the running statement's
buffer through a ContextVar, so nothing is saved or restored and no statement
can restore over another. Worth recording for anyone attempting the same fix:
a property on the module's type — the mechanism that works for sys.argv —
does not work here, because print reads stdout from the module dict and
never consults the type.

/statements says what it drops (#360)

The route accepted env and spark_conf and applied neither, returning
{"status":"ok"} with the field discarded. Both are now named in the agent log
with the reason, along with any field the route does not recognise.

Named rather than refused, deliberately: every released databricks-emulator
still sends both, so refusing would break the callers that exist today. Neither
will be implemented — Fabric's Livy statement payload is {"code", "kind"},
and a statement-level spark_conf is not statement-scoped: it outlives the
statement on the session's Spark session, and on a shared session it leaks into
every other one. A task's environment belongs in the code it runs, where no
agent can drop it.

Release dispatch reached one platform of three (#347)

The dispatch target had been renamed, and every release since succeeded on
GitHub's redirect — so the release workflow reported success while two of the
three platforms were never told at all. Their acceptance runs simply did not
happen, and nothing said so. Platforms are now dispatched by their real names,
and a name that does not resolve fails the release instead of redirecting
quietly.

OpenMetadata comes from the family's registry (#350)

docker.getcollate.io failed two platform nightlies inside an hour. The images
are mirrored into GHCR by the hub with index and digest intact, so the
governance profile now pulls from there — a vendor registry having a bad
morning should not decide whether the family's nightlies run.

The sidecars are referenced by their current names (#348)

emulator-spark-agent and emulator-sail have been the names since v0.26.0.
The fabric-emulator-* aliases stay published, so nothing breaks; this only
stops new references asking for them. Historical accounts that name the old
image keep it.

A spike, kept (#353)

e2e/sail-session-isolation answers one question by measurement — can two
Spark Connect sessions against Sail be isolated — and it decided the
architecture above: newSession() is JVM-only and silently degraded on Sail,
so nothing was ever isolated there, while builder.create() does isolate. It
ships as a harness rather than a parity witness, and exits 0 whatever it finds,
because a spike reports and does not gate.

Consumers should bump their pinned digests

emulator-sail and emulator-spark-agent are tagged for the dependency they
carry, not their content: both also ship first-party code that changes
independently, so this release republishes the same tags over different bytes.
A consumer pinned by digest — the recommended form — stays on the previous
image until it bumps.

The platform repositories now pin every image they pull by digest, with the
version beside it for readability, and move the two together. If you maintain a
consumer that does not, repo:tag@sha256:… is the form to adopt: docker
ignores the tag and fetches the digest, so a version bumped without its digest
runs the old image under the new name.

v0.32.0

v0.32.0

The range starts at 9a8f484 (#332), the first commit after the v0.31.0
tag, and ends at this file: 10 changes plus their notes. One is a data-loss fix
that consumers of this image will want promptly, one is a T-SQL reflection
repair, and the rest are a witness, docs gates and dependency bumps.

Read the first section if you run more than one task at a time on the Spark
agent — two concurrent tasks could read each other's parameters and each
other's resolved secrets, both reporting success. It is the reason
databricks-emulator currently documents for_each_task concurrency > 1 as
a known limitation.

Read the second if you reflect Warehouse tables: a UUID or BSON column came
back as varchar, so a client round-tripping the type wrote text into a binary
column.

---

Two concurrent tasks could read each other's parameters (#338)

The statement agent isolates a session's user globals — a name bound in one
session is not visible in another — but it did not isolate module state, and
sys is one object per interpreter. Task parameters are delivered by assigning
sys.argv, and resolved secrets by updating os.environ, so two tasks
dispatched in the same wave overwrote each other and each read whichever
assignment landed last.

Both tasks report SUCCESS. That is the shape that matters: no error, no warning,
and a task that quietly processed another task's inputs. A loop that writes one
output per input silently produces fewer outputs than it has inputs, and the run
is green.

Measured against the published 0.31.0 agent, three concurrent statements in
three distinct sessions, each told to write a table named by its own parameter:

alpha  ok     asked alpha wrote alpha
beta   ok     asked beta  wrote alpha     ← read another session's argv
gamma  error  CommitFailedError: Protocol changed since last commit
tables on disk: ['alpha']

One correct, one silently wrong, one killed by the Delta commit conflict that
follows from every writer targeting the same table. With this release:

tables: ['alpha', 'beta', 'gamma']   — each holding its own value

task_scope.py gives each session its own sys.argv and os.environ around
execution, and python/tests/test_spark_agent_task_scope.py holds it.

Consumers should bump their pinned agent digest. The tags do not move on
their own: emulator-spark-agent:4.2.0 and :latest are rebuilt by this
release, so a consumer pinned by digest — which is the recommended form, and
what the platform repos use — stays on the pre-fix image until it bumps.
databricks-emulator's e2e/composite-tasks is the known case, tracked as
databricks-emulator#64.

Not fixed here: captured stdout still crosses sessions, so a task's
printed output can be attributed to another concurrently running task. The data
a task writes is correct; the log line saying it did may appear under a
sibling. Scoped separately from this fix, which covers sys.argv and
os.environ.

A UUID or BSON column reflected as varchar (#340)

Warehouse reflection reported uniqueidentifier and BSON columns as varchar.
A client that reflects a table to learn its shape, then writes back what it
read, put text into a column the engine treats as binary. The repair returns
varbinary, which is what the type is.

Class B strict mode, witnessed by a real TDS client (#341)

-tsql-strict was graded on this repository's own assertions. It is now driven
by an unmodified TDS client, so the row rests on something that is not us. Same
conversion the family has been applying to every go:-only claim.

Docs gates and dependency bumps

• #342 — the docs gate checked sidebar reachability but not the links
  themselves, so a link to a page that had moved passed. Both are checked now.
• #343 — entra-emulator 0.9.0.
• #334 — great-expectations 1.20.0, and the numpy 2 move it required.
• #333 — mcp 2.0.0.
• #332 — kafka-python 3.0.11.
• #336 — Go 1.26.6, plus secret and vulnerability scanning in CI.
• #337 — dbt_expectations measured on both dbt adapters rather than
  assumed to behave the same on each.

---

Upgrading

Nothing in this release changes a documented API shape. Two things are worth
doing rather than assuming:

1. Bump the pinned emulator-spark-agent digest, or the concurrency fix
   above does not reach you. docker buildx imagetools inspect ghcr.io/calvinchengx/emulator-spark-agent:4.2.0 reports the new one.
2. Re-check any suite that pinned concurrency: 1 to work around #338.
   The workaround is no longer needed, and leaving it in place hides whether
   the fix reached you.

v0.31.0

v0.31.0

The range starts at 322d20d (#325), the first commit after the v0.30.0
tag, and ends at this file: 6 changes plus their notes. One is the headline and
the rest are repairs — two of them dependency bumps that broke something on the
way in, which is the more useful half of a dependency bump.

Read this if you trigger runs on an ApacheAirflowJob after publishing DAG
files — which is every consumer of that item type. A published DAG could run
as the previous version of itself and report success.

Read the second section if you run notebooks against the Spark agent: a
dropped engine session used to poison every later statement until the container
was restarted.

---

A changed DAG could run as its previous version, and pass (#328)

Publishing DAG files and starting a Run raced this emulator's own scheduler.
TriggerAndWait waited for the DAG to load, which is right for a brand-new
file: until it parses there is nothing to unpause, so the wait blocks. A
changed file has the opposite shape. The DAG is already registered, every
check passes instantly, and the run is created from whatever structure is
serialised at that moment — the version published moments earlier.

The failure cannot announce itself. It is a green run whose task instances
belong to code that has been replaced, and it surfaces downstream as a task the
trigger rule references having no instance at all, or a newly added task
returning in state removed while its downstream fails. Both read as DAG bugs.
A consumer diagnosed them that way before finding this, then worked around it
by sleeping 45 seconds before every run.

That workaround could never have worked, and the reason is here rather than
there. PUT .../files only stores bytes on the item; the write into the
scheduler's DAG folder happens in the Run handler, immediately before the
trigger. A consumer sleeping before starting a run is sleeping before the files
exist on disk at all. The race is entirely inside this process, between our own
SyncDAGs and our own trigger, and no amount of consumer patience closes it.

Reproduced on v0.29.0 on demand:

published:  branch_0, branch_1, branch_2, branch_3, join   (5 tasks)
run got:    branch_0, branch_1, branch_2, join             -> failed

Two plausible signals were measured and rejected

Both look correct and are not, which is why they are recorded here:

signal	why it fails
last_parsed_time vs the file's mtime	came back 20ms ahead of a write whose content that parse had not read — the cycle began before the write landed. It reports that a parse finished, never that this file was ingested.
/dagSources/{file_token} vs the bytes on disk	DagCode matched disk a full 13 seconds before the task structure changed.

The gap is Airflow's own min_serialized_dag_update_interval, 30s by default:
the processor may read a file and skip rewriting the serialised DAG. Task
instances come from that serialisation, so it is the only thing worth waiting
for.

It also explains the consumer's history. A 15-second sleep let four stale runs
through and 45 seconds appeared to fix it; both were guesses either side of a
threshold nobody had identified.

What it does now

The Run handler reads the DAG's task set before syncing, and the trigger
waits for it to change. Taken after the sync it would be worthless — the stale
answer and the current one are indistinguishable without a baseline.

SyncDAGs now reports whether any file actually changed, and the wait is
skipped entirely when nothing did. Only a changed file can leave a
serialisation stale, and re-running an unmodified DAG is the ordinary case; it
would otherwise pay the full timeout every time. A removed file counts as a
change, so the file sets are compared and not only the bytes that arrived. An
unchanged file also keeps its timestamp rather than being restamped by the
wipe-and-rewrite.

Two deliberate non-failures. No baseline means a DAG this emulator has never
served, which cannot be stale and is already covered by the load wait. And a
change that alters no task — a callable's body, a default argument — produces
no observable difference, so the wait expires and proceeds rather than failing
every such run.

A dropped Spark session poisoned every later statement (#327)

The agent built its session once at import and never again. When the engine
dropped that session, every subsequent statement failed until the container
restarted — an open notebook simply stopped working, with nothing in its own
history to explain why.

It now detects the drop, rebuilds, rebinds every namespace, and reports what
the rebuild cost rather than hiding it.

The detection deliberately requires session beside is not running. The
latter alone also matches a stopped container or daemon, and each false
positive costs an open notebook its temporary views — a rebuild is not free,
so it must not fire on a symptom it does not own. Closes #312.

Warehouse reads parquet-go 0.32's LogicalType (#326)

LogicalType became a thrift union carrying one Value instead of a struct of
optional pointers, so the annotation is now read by type. Same three questions
in the same order; TimeUnit moved the same way. Arrived with the dependency
bump that required it, rather than after it.

Also in the range: svelte 5.56.9 with portal/dist rebuilt (#325),
kafka-go 0.4.51, and pyarrow 25.0.1 (#331).

CI bounds the package installs that could consume an entire job (#329). Nothing
in the shipped emulator changes, but it is the reason a run of this release's
own branch was cancelled rather than completed — a job spent its whole budget
installing packages, and three unrelated jobs were still in flight when the run
stopped.

Upgrading

Nothing to do. AirflowRuntime is an internal interface and its two signature
changes do not reach the HTTP API.

Consumers sleeping between publish and run can delete that sleep. It was never
buying what it appeared to buy.

v0.30.0

v0.30.0

The range starts at 8ff0b72 (#320), the first commit after the v0.29.0
tag: 4 commits. Two of them are the same lesson from opposite directions, a
witness that could not see the thing it was witnessing, and an error that
discarded the only sentence naming its cause.

Read this if you run the built-in Airflow sidecar, or if you consume the
statement agent: the executor changed, and the agent image now carries dbt.

---

The Airflow sidecar runs CeleryExecutor, because Fabric gives a DAG no choice

The sidecar ran SequentialExecutor on SQLite, argued in its own comment as a
local fidelity target rather than a throughput one. That reading was wrong.
Microsoft documents Fabric's default as CeleryExecutor and lists
AIRFLOW__CORE__EXECUTOR among the configurations a user cannot override,
so on real Fabric it is always Celery and no DAG can opt out.

Sequential runs one task at a time, so a DAG with parallel branches serialises,
passes, and demonstrates behaviour no Fabric user can have.

The witness could not see it, which is how this survived a green e2e: the
DAG had a single task, and one task cannot distinguish the two executors. It
now fans out to three branches that each record their own window, and the
consumer asserts the wall span is well under the sum of the work:

PASS: ... ran 3 branches CONCURRENTLY (3.0s wall for 9.0s of work)

A DAG-sync failure is not a DAG that failed

Both finalised as AirflowRunFailed, whose message is the bare "The job
failed." An operator whose emulator could not write the DAG files saw a
failed run beside an empty dags folder and no reason at all, because the real
error was discarded one line after it was returned.

Found the hard way: a consumer platform's first end-to-end run failed exactly
like this. The cause was a DAG volume shared with the Airflow sidecar, coming
up 0775 airflow:root while the emulator runs distroless as uid 65532, neither
owner nor group, so no write. Finding that took a permissions audit rather than
a read of the error.

AirflowDAGSyncFailed now carries its own code and a message naming what to
check. Tests pin both halves: that sync and run failures stay distinguishable,
and that the new code has a message of its own, since a distinct code still
answering "The job failed." would move the problem rather than fix it.

The statement agent carries dbt

emulator-spark-agent now ships dbt-databricks, because
databricks-emulator
terminates dbt_task and hands this image a dbt project and a generated
profile. dbt is an ordinary warehouse client, so running it as a job changes
who invokes it, not what it connects to. Nothing in Fabric's own surfaces calls
it; the image is the family's rather than Fabric's.

It lives in a spark-agent-dbt group rather than in sail-delta, because
sail-delta also builds the engine matrix's "Sail + delta-rs" probe and that
column exists to measure the shipped runtime. An adapter the probe never calls
would make it a lookalike of the agent rather than the agent.

dbt-databricks pins databricks-sdk<0.118 where this repository's own
databricks-sdk group wants >=0.130. Both are correct and they are never
installed together, so the pair is declared in conflicts rather than resolved
by loosening one, which would silently move the version a witness runs against.

The image is smaller than it was, with dbt in it

Adding dbt first measured 894 MB → 1789 MB, which reads as doubling the image.
It was not:

	v0.29.0	v0.30.0
venv	452 MB	671 MB
/root/.cache	538 MB	gone
image	894 MB	1019 MB

Every agent image up to v0.29.0 shipped uv's download cache, wheels nothing
ever unpacks again: build input baked into a runtime image. --no-cache on the
same line removes it. The honest cost of dbt is the venv, +219 MB, and the
image grows by 125 MB net rather than 895.

The parity ledger separates two kinds of own-only row

family_parity.py fabric reports 106 of 113 green claims independently
witnessed. Seven rows were own-only, and they are not one kind of gap: three
can never have a third-party witness at all and are now marked boundary: in
the manifest, three wait on the real-Fabric leg, and one is convertible today.
So the honest denominator is 110, not 113, and 106/110 is a floor
rather than a score with three permanently unreachable rows counted against
it.

Upgrading

• The Airflow sidecar now needs a metadata DB and a broker for Celery. The
  shipped compose brings them up; a hand-assembled stack that sets
  AIRFLOW__CORE__EXECUTOR itself should stop, since real Fabric does not let
  a user set it either.
• AirflowDAGSyncFailed is a new error code. Anything matching on
  AirflowRunFailed to mean "the DAG failed" now sees the sync failure under
  its own code, which is the point.
• The agent image is 1019 MB, up 125 MB, and carries dbt.

v0.29.0

Changelog

• 6a12003 Claim the CTAS shapes dbt actually emits: comments, CTEs, and a real REPLACE (#309)
• 8856544 Encode a SQL result's typed values instead of dropping the socket. (#303)
• 06e8a98 Fail a run whose Environment was not applied, instead of reporting it honoured. (#304)
• a419e8b chore(deps): take sqlparse 0.6.0 over dbt-core's upper bound (#311)
• abf9dec e2e: witness typed columns, and stop the suite agent duplicating the fix (#306)
• 853e6e0 test(cmd): give serve() the headroom poll() already has (#305)

v0.28.0

v0.28.0

The range starts at 74bfd0a (#282), the first commit after the v0.27.0
tag: 17 commits. Two are bugs a consumer would have hit, one is a behaviour
change worth reading before you upgrade, and most of the rest are witnesses for
rows that were already graded.

One behaviour change. A notebook that binds an Environment now gets its
packages installed. If you have been working around that, the workaround is no
longer needed. Nothing else changes a contract, and there is no upgrade step.

---

A notebook's Environment is applied, not just resolved

An Environment can be bound two ways: a Livy session names one by id, and a
notebook names one in its own # META dependencies. Only the first worked.

resolveComputeBinding is shared by notebooks and Spark Job Definitions, and
both stored the resolved Environment on their run. Only the job-definition
driver and the Livy session ever handed it to the agent. So a notebook naming
an Environment ran with none of its packages and died on the first import:

Cell 0 failed: Error: ModuleNotFoundError: No module named 'contoso_product'

while the run detail cheerfully reported the environment it had ignored.
Resolved, reported, ignored is worse than unimplemented, because the
metadata says the dependency was honoured.

The notebook driver now applies it before the first cell. It takes the Livy
behaviour rather than the Spark Job Definition one, and that is deliberate: a
notebook run is a session. A submitted job is one shot, so sparkjobdrive
refuses a JAR-bearing Environment outright; a session lets JARs be reported as
skipped and carries on, which is what an interactive notebook on a Connect
engine should do.

The docs were the reason this survived so long, and they were not wrong.
The Environments row said an Environment is "applied to the session"; the
notebook row said the emulator "resolves attached lakehouse/Environment
metadata". Both true. Read together, a consumer concludes that binding an
Environment to a notebook installs the packages, which is the natural reading
and was the wrong one. Both rows now name the path.

The witness lives in its own suite, e2e/environment-notebook, and the reason
is worth knowing if you ever extend it: the spark agent is one long-lived
process, so a package any suite installs is importable by every session after
it. Written inside e2e/environment first, the notebook negative half passed on
the Livy half's install and proved nothing. Binding a different Environment in
the same agent is refused by design, so there is no in-stack fix. Both halves
now assert the reason rather than the outcome, because a notebook submitted
before the agent finishes starting also "fails", and the first version of the
suite went green on a stack that had executed nothing.

Two fixes a consumer would have hit

The MLV refresh decoded its query twice, so it never ran on a real engine.
json.Marshal emits a quoted JSON string and Python already unquotes it as a
literal, so the json.loads wrapper received bare SQL and raised
JSONDecodeError on every refresh. The Go tests record statements rather than
executing them, which is why they never saw it; e2e/sail now runs the real
agent and reads the materialised rows back.

The eventstream drain emitted a Kusto schema real Kusto refuses.
kustoIngestTable checked column names against a character class alone, so a
KQL keyword passed. kind is an ordinary field name in event data and cannot
stand as a bare column name in a schema declaration: real Kusto answers with
400 KustoBadRequestException / SYN0002. Every name is now quoted, ['kind'],
rather than the keywords this emulator happens to know. A keyword list in the
emitter would have been a list of the names we had thought of, which is exactly
the fault that let kind through.

The destination table name in those same two commands had the identical
fault and is quoted too. That half is witnessed rather than argued: quoting a
table name touches every drain, not just one that names a keyword, so
e2e/rti asks Microsoft's own engine directly. It records the engine refusing
the bare form, accepts the quoted one in both commands, and then merges a fully
quoted ['Readings'] into a table the suite created bare and filled with four
rows, ending with three columns and four rows. Acceptance alone would not have
ruled out a quoted name resolving somewhere else and stranding a drain's rows in
a table its owner cannot see.

The agent is gated on its other consumer now

emulator-spark-agent is published here and consumed by databricks-emulator.
A break in the path only that repo exercises shipped in four releases with
this repo's suite green throughout, and the reason was structural rather than
bad luck:

e2e/livy addresses Delta BY PATH      OPTIMIZE delta.`uri`   -> never calls resolve()
databricks-emulator addresses BY NAME OPTIMIZE events        -> nothing else

resolve() was where the break was, so no witness here could reach it, and the
consumer found out on upgrade. e2e/agent-contract now stands up Sail plus the
agent built from the published Dockerfile and drives the by-name shapes.

The release dispatch no longer announces images that do not exist

dispatch waited on fixtures alone, so it told contoso-fabric-platform a
release was ready while the images that release publishes were still building.
Measured on v0.27.0 rather than inferred: the dispatch fired at 14:36:17 and the
spark-agent image finished pushing at 14:44:03, so for 7m46s a downstream
repo had been told to verify a fabric-emulator-spark-agent:0.27.0 that was not
there.

New witnesses for rows that were already graded

Pipeline activities, each driven by the real client rather than a shaped
request: AzureFunctionActivity including key and URL, the
AzureDataExplorerCommand activity through the Kusto SDK, and Custom,
HDInsightSpark and SparkJobDefinition on a real agent. The Reflex event
trigger is witnessed with Microsoft's own Blob SDK, and fabric-cicd's own poll
loop now witnesses the forced 202 outcome.

Also: the ADX command tests use KQL real Kusto accepts, the concurrent
Delta-overwrite claim cites the sail race probe, real-fabric authenticates
via OIDC and gates on every credential it uses, and v0.26.0 and v0.27.0 got the
release notes they had been missing.

Upgrading

docker pull ghcr.io/calvinchengx/fabric-emulator:0.28.0

If you bind an Environment to a notebook, it now takes effect. Consumers that
bind-mounted /opt/wheels to compensate can drop the mount and declare only the
Environment.

v0.27.0

v0.27.0

The range starts at 53a09f6 (#279), the first commit after the v0.26.0
tag: 3 commits. Small, and all three are about the same thing from different
directions: code paths that were documented, graded, or shared, and never
actually run.

Nothing here changes a contract. There is no upgrade step.

---

The spark-agent stops assuming one consumer's habits

databricks-emulator pulls this repo's spark-agent image, and its delta
witness died on a statement it never wrote:

MERGE INTO events AS t ...
IllegalArgumentException: invalid argument: found DETAIL at 9:15

Three defects, each hidden behind the one before it, and the first two are the
same mistake: the agent is shared by two emulators, and each defect took one
consumer's habits for the world.

1. _CREATE_DELTA_LOCATION required USING immediately after the table
   name. That is dbt-fabricspark's shape, so every witness in this repo was
   green. The ordinary CREATE TABLE events (id INT, name STRING) USING delta LOCATION '…' did not match, so the location was never recorded, and a MERGE
   two statements later fell through to a DESCRIBE DETAIL that Sail has no
   grammar for. A missed record has no symptom at the point it happens. The
   pattern now takes an optional column list, paren-aware so DECIMAL(10,2)
   survives, and clauses between USING and LOCATION. Writing the shapes
   down found a second, latent one in this repo: PARTITIONED BY before
   LOCATION was unrecorded too.

2. resolve() let the engine's parse error escape. The module's own
   comments say Sail cannot parse DESCRIBE DETAIL, and then it issued one
   anyway and let found DETAIL at 9:15 surface, pointing at column 9 of the
   user's MERGE. It now raises DeltaOpError naming the table, keeping the
   engine's words as detail rather than as the headline.

3. localRelationSizeLimit is now asked for, not assumed. With the
   fall-through fixed, MERGE reached delta-rs and failed while returning its
   result: delta_ops answers via spark.createDataFrame, and pyspark calls
   int() on the served conf. Sail 0.7.0 serves '3221225472'; the older Sail
   that databricks-emulator pins serves '3GB'.

   v0.26.0 removed the preset on 0.7.0 evidence, which was correct for this
   repo, and that measurement was taken while defect 1 masked the case that
   needed it. Neither constant is right for a shared image, so connectconf.py
   now reads the value and rewrites only what will not parse, to the same size.
   No engine's limit is reduced, and no client is handed a string it chokes on.

The guard is the interesting part. python/tests/test_agent_consumer_contract.py
records each statement shape a named consumer actually sends, cited to the file
it comes from, with the answer the agent owes it. No engine, milliseconds, runs
on every PR. It fails in the repo that would ship the regression rather
than the repo that would suffer it, which is the property a shared artifact
needs. Four of its rows fail against the code before this change.

Verified on both engines rather than only the one this repo pins:
databricks-emulator's e2e/delta on Sail 0.22.0 advances the log to v4,
confirmed by delta-rs; this repo's e2e/sail on Sail 0.7.0 passes.

FABRIC_DATABRICKS_URL is finally exercised

It was documented in the README, in docs/04-configuration.md, in the v0.25.0
notes and in internal/api/databricksremote.go, and it appeared in no
workflow anywhere.

That matters because the parity row grades the Databricks activities 🟢 Real
(notebook + python, local or FABRIC_DATABRICKS_URL). The local half ran on
every push. The remote half had never executed once. So this is primarily an
unexercised code path getting exercised; the third-party witness is a side
effect.

The chain is entra → fabric → databricks-emulator → spark-agent → sail, plus
the client. Sail and the agent build from this checkout, so the chain
breaks when this tree breaks it. A pinned image would have hidden exactly that.

Five more pipeline activities witnessed

Including the WebHook park, which is the one that holds a run open waiting for
a callback rather than completing on its own.

Upgrading

Nothing to do.

If you consume this repo's spark-agent image from another emulator, you get the
CREATE TABLE … LOCATION and localRelationSizeLimit fixes above, and a
DeltaOpError that names the table instead of an engine parse error pointing
at the wrong column.

v0.26.0

v0.26.0

The range starts at fb1e6aa (#254), the first commit after the v0.25.0
tag: 26 commits. Most of them are one thing done repeatedly. DAX scalars
were pinned against Power BI Desktop captures, function family by function
family, and the exercise found a parity defect that no local test could have
reported. Alongside that the Spark engine moved to Sail 0.7.0, arm-emulator
became a default service in this repo's compose, and the compute images gained
names that say what is inside them.

One of these changes what docker compose up starts, and one changes what a
DAX expression returns. Read the first section before you pull.

---

Behaviour changes, read this one first

• arm-emulator now starts with docker compose up in this repo. Fabric
  capacities are ARM resources, and without arm there was nowhere for
  Microsoft.Fabric/capacities to go: GET /v1/capacities served only the
  seeded local one, which is a visibly smaller Fabric than the documentation
  describes. arm is now wired exactly as the family BOM wires it, same issuer,
  tenant and subscription, so the two stacks cannot disagree.

  FABRIC_ARM_URL uses - rather than :-, matching KV_ARM_URL in the BOM,
  so an explicitly empty value opts back out to the seeded capacity and
  standalone fabric-cicd still works. The binary default is unchanged:
  still empty, still standalone. Only this compose file changed, and
  docs/04-configuration.md and docs/07-control-plane-api.md now distinguish
  the two rather than calling the feature opt-in.

• DIVIDE's third argument is now honoured. DIVIDE(numerator, denominator [, alternateResult]) returns alternateResult when the
  denominator is 0. The arity guard was len(args) < 2, so a third argument
  was accepted and then never read: DIVIDE(x, 0, 0) answered BLANK where
  Fabric answers 0.

  This is the failure direction that ships. The query parses, evaluates and
  returns a plausible number, so there is no local symptom at all and nothing
  to investigate until someone compares against a tenant. IF, twelve lines
  below, has the same < 2 guard and does read args[2], which is what makes
  this an oversight rather than a decision. The guard is now < 2 || > 3, so a
  fourth argument is rejected rather than silently dropped.

  Provenance, stated because it differs from the goldens next door: the
  alternateResult semantics come from Microsoft's DAX reference, not from a
  Desktop capture. Nobody ran the msmdsrv oracle for this, so the assertions
  live in dax_arity_test.go and deliberately not in desktop_goldens.json,
  whose every entry means "Desktop answered this".

• The Spark engine is Sail 0.7.0, and the Connect client moves with it to
  pyspark-client 4.2.0. The UDF runs inside the Sail server's embedded
  CPython, so server and client must agree; pysail 0.7.0's own test extra
  pins 4.2.0 where 0.6.6 pinned 4.1.1. The client is pinned in five groups,
  not one (sail, sail-delta, jupyter, spark-connect, data-science-loop),
  and all five move together. The JVM overlay is unaffected: it never uses
  pyspark-client at all, going through spark-submit and Livy with the image's
  own pyspark.

DAX pinned against Desktop, not against our reading

The bulk of the range. Each of these is a family of scalars whose answers now
come from a Power BI Desktop capture rather than from our interpretation of the
reference:

SIGN, ASIN, ATAN, PI, SIN, COS, TAN, DEGREES, RADIANS,
DATE, YEAR, MONTH, DAY, TIME, HOUR, MINUTE, SECOND, INT,
MIN, AVERAGE, COUNT, POWER, the remaining Phase 3 scalars, and ROW.

Three additions are about the edges rather than the happy path: SWITCH with
no else branch, the 1.0 == 1 comparison, and eight BLANK and error edges that
the numeric goldens cannot express. The DIVIDE fix above also salvaged 16
Desktop probes that had been captured but were unusable while the third
argument was being dropped.

Engine claims corrected

Two notes that were true of an engine this repo no longer pins:

• The localRelationSizeLimit preset is dropped. On 0.7.0 evidence it capped
  3 GiB to 64 MiB, so it was making things worse rather than better. (v0.27.0
  revisits this: the preset was measured while a separate defect masked the
  case that needed it, and the fix is to ask the engine rather than assume
  either constant.)
• The VALUES behaviour was credited to an engine that is no longer pinned.
  The credit is removed rather than reassigned, because nobody re-measured it.

Compute images say what is inside them

Both sidecars now also publish under names that describe their contents rather
than the repo that builds them, tagged with the version of the dependency each
is pinned for:

ghcr.io/calvinchengx/emulator-sail:0.7.0            # the pysail pin
ghcr.io/calvinchengx/emulator-spark-agent:4.2.0     # the pyspark-client pin

The old naming conflated two things. fabric-emulator-sail carries upstream
Sail, and databricks-emulator consumes it with no Fabric in the picture at
all; and the tag was fabric's release number while the Sail inside was 0.6.6,
so 0.22.0 and 0.25.0 contained the same engine with nothing in either tag
saying so.

Tags are read from pyproject.toml by scripts/image_tags.py rather than
typed into the workflow, because a version written in release.yml is a second
copy of a number: the release that bumps pysail without editing the workflow
publishes a tag naming the version it no longer contains, and nothing fails.

This is step one, and it moves no consumer. Both names publish, and the old
names keep their release-version tags, so fabric-emulator-sail:0.26.0 and
fabric-emulator-spark-agent:0.26.0 exist as before.

Witnesses

• Microsoft's az CLI drives git integration, CopyJob, and four pipeline
  activities.
• ty can now see pyspark, so the statement agent is actually type-checked
  rather than silently skipped.

Family alignment

• entra-emulator 0.8.1: the Go library, the pin, and the three composes the
  pins gate cannot see.
• The consumer repository is now contoso-fabric-platform, named in
  release.yml and in the acceptance dispatch.

Upgrading

docker compose up in this repo now starts arm-emulator as well. If you
want the previous shape, set FABRIC_ARM_URL= explicitly empty and capacities
fall back to the seeded local one. Running the binary is unchanged.

If you pin pyspark-client yourself, move it to 4.2.0 alongside Sail
0.7.0. Server and client share the embedded CPython and must agree. The JVM
overlay needs nothing.

If a DAX expression relies on DIVIDE ignoring its third argument, it now
returns that argument on a zero denominator. That is the direction of the
truth, and any query written against Fabric was already expecting it.

Image names are additive. Nothing that pins fabric-emulator-sail or
fabric-emulator-spark-agent by release version needs to change.

v0.25.0

v0.25.0

The range starts at 707688c (#218), the first commit after the v0.24.0
tag: 26 commits. The short version is that three surfaces stopped being
item management and started executing. Eventstream gained destinations
and operators, the Azure Batch Custom activity runs by default, and
Databricks activities can submit to a real workspace. Alongside those,
DAX gained an optional real oracle and its first Desktop-pinned scalars,
and the conformance kit landed with write landing asserted out of band on
all three backends.

Two of those change behaviour on upgrade. Read the first section before
you pull.

---

Behaviour changes, read this one first

• The Azure Batch Custom activity now runs by default. It used to be
  refused unless opted in, which made every pipeline that actually uses
  Batch a false pass: the activity reported nothing useful and the run went
  green. A Custom activity's command is a shell process on the Spark
  agent, which is the same machine a notebook cell can already
  subprocess.run on, behind the same bearer and the same workspace RBAC,
  so refusing it by default was protecting nothing.

  The command executes in the agent's container rather than the
  emulator's process, so the blast radius is the engine and not the API.
  extendedProperties become environment variables, as Batch documents,
  and the command's own exit code decides the activity: a non-zero exit
  fails it with that code and the command's stderr, and a report that
  cannot be read fails too, because an unknown exit status is not success.

  If you have a pipeline with a Custom activity, its command now
  executes. FABRIC_CUSTOM_ACTIVITY=off restores the refusal so nothing
  reaches the agent; the old opt-in spelling shell still means on.
  Batch-node features (resourceLinkedService / folderPath,
  autoUserSpecification, referenceObjects) stay refused by name even
  when the activity is on.

• input_file_name() in a SQL string now resolves, or errors. The
  agent's shim tags each file's rows at read time and points
  F.input_file_name at the tag, but spark.sql("SELECT input_file_name() …") never touched the patched function and failed on the engine as if
  the shim were not there. spark.sql is now wrapped beside the PySpark
  patch: a lexer finds the call in code and leaves strings and comments
  alone, a view built from a tagged frame registers a clean name plus a
  shadow that still carries the tag, and the query is rewritten onto that
  column.

  A UDF could not have fixed this, which is the part worth recording. The
  function takes no arguments, so a registered UDF cannot see which row it
  is evaluating and could only return a constant, which is the
  silently-wrong-lineage failure the module exists to refuse. A relation
  that was never a file read now raises InputFileNameError rather than
  resolving to "".

• The sc facade refuses toDF() on scalars, as real PySpark does.
  Seq(1,2,3,4).toDF() is Scala, and the facade had been accepting the
  Python transliteration of it. The contract file now pins refusals as
  well as answers, so over-permissiveness is checkable rather than only
  under-permissiveness.

Eventstream executes end to end

v0.23.0 and v0.24.0 left Eventstream as item management plus a raw Kafka
topic. This range closes the topology.

• Kafka on both engines (#226). Fabric notebooks resolve
  eventstream.itemid / eventstream.datasourceid against a real Apache
  Kafka KRaft broker. Sail consumes the records and builds the Kafka schema
  into the engine, so CAST(value …) runs on Sail rather than the source
  being mapped onto rate. The JVM overlay keeps the native
  spark-sql-kafka source. subscribePattern, assign, JSON offsets,
  includeHeaders, SASL PLAIN, PEM SSL, GSSAPI and JKS/P12 stores are all
  honoured, with the Java-shaped ones converted in the wrap because
  kafka-python wants PEM and a ticket rather than Java stores. OSS
  format("kafka") reads and writes work on Sail too. Checkpointed
  streaming stays on the JVM overlay. The broker is apache/kafka (ASF,
  Apache-2.0, multi-arch) rather than Redpanda, whose license moved to BSL.

• Three destinations and three operators (#231, #237). Custom HTTP
  produce now feeds a bound Lakehouse Delta table, a Reflex
  EventReceived trigger fired as a real item job, or an Eventhouse
  KQL database through direct ingest (.create-merge plus .ingest inline, which is the path kustainer actually supports, not Fabric's
  streaming-ingest protocol). Filter, GroupBy and tumbling Window run
  on the produce batch between source and destination. Kafka stays the raw
  source; destinations see operator output.

  Binding is an emulator-native surface (POST …/eventstreams/{id}/ destinations and …/operators) because Fabric's topology has no public
  REST, the same situation Reflex triggers are already in. Join, Union,
  Expand and hopping or sliding windows are refused by name: they need more
  than one stream or cross-batch state.

Databricks activities can reach a real workspace

Set FABRIC_DATABRICKS_URL (plus FABRIC_DATABRICKS_TOKEN, and
FABRIC_DATABRICKS_TLS_INSECURE for a self-signed emulator cert) and
DatabricksNotebook / DatabricksSparkPython import the workspace file
and submit through Jobs 2.2. dbfs: and /Workspace paths become legal
only then, because without a workspace to submit to, reinterpreting a
Databricks path as a lakehouse path would invent a mapping nobody wrote.

Unset keeps the existing behaviour: the submission contract terminates
locally and the Spark agent executes the code. DatabricksSparkJar and
libraries stay refused on both paths.

A real DAX oracle, and the first Desktop-pinned scalars

The bounded Go evaluator stays the default on every OS, and empty
FABRIC_DAX_URL is what GitHub-hosted ubuntu and mac CI tests. What is new
is that a machine which can run Windows may now attach real VertiPaq:
FABRIC_DAX_URL points executeQueries at a pump in front of msmdsrv
(POST /v1/deploy, then POST /v1/dax). Set-but-unreachable is a 502 and
not a silent fallback to the internal evaluator, because those two answers
are different facts. This is not a compose default and not a
macos-latest / ubuntu-latest sidecar; docs/52
says what each host actually needs.

The point of an oracle is what it settles. Five functions are now pinned
against goldens captured from Power BI Desktop, and every one of them
caught a difference from the obvious implementation:

• ACOS (#236), captured alongside the discovery that Desktop rejects a
  DATATABLE with a missing sourceColumn and an empty catalog name. The
  pump maps both to 409 and retries without Initial Catalog=, so a
  still-open .pbix stays queryable, which is what makes DATATABLE
  deployable at all.
• ABS and ROUND (#242). ABS(BLANK()) stays BLANK. ROUND is half away
  from zero for negatives as well as positives, and a negative digit place
  rounds to tens or hundreds, so ROUND(1234, -2) is 1200. Multiplying by
  a power of ten and calling math.Round is not enough to match: 2.15
  shifted one place is IEEE 21.4999…, which floors to 2.1 where Desktop
  says 2.2, so the shift goes through a decimal exponent string instead.
• LOG and LOG10 (#253). Desktop's LOG defaults to base 10, and
  BLANK, non-positive arguments, and base 1 all error, which is not what
  EXP(BLANK()) = 1 would have led you to guess.

Every-push CI replays the captured goldens against the Go evaluator with
FABRIC_DAX_URL empty, so the oracle's answers keep gating the default
path even where no Windows host exists. Headless msmdsrv was measured
too (docs/33, Phase 0c): it listens, but ROW
needs a table, so Desktop remains the oracle.

Framework conformance: write landing is asserted out of band

The conformance kit landed (#235) and then got live backends (#243).
Contract 4, write landing, is now ✅ on sail, jvm and warehouse, and
the assertion is deliberately not the writer's own catalog: a notebook run
plus an out-of-band OneLake DFS listing, or a TDS INSERT plus a fresh
SELECT on a new connection. A writer that would confirm its own write
is refused before any I/O runs. That was the false-green shape the kit
exists to remove.

Contracts 1 to 3 and 5 to 7 stay ❌ with a pointer to the prose that closes
each one, rather than a silent skip. Wiring the JVM Livy session for that
row also fixed a real defect: spark-submit had been building a session
with neither the Delta catalog nor the Entra token provider, so
saveAsTable failed with
DELTA_CONFIGURE_SPARK_SESSION_WITH_EXTENSION_AND_CATALOG.

The Sail column closes four more rows

• MERGE, change data feed, and JSON multiLine (#220). CREATE TABLE … LOCATION is recorded so DESCRIBE is real, subquery MERGE and
  INSERT * are intercepted, and the CDF and multiLine notebook APIs are
  wrapped, announced and materialised.
• Durable streaming sinks (#223). writeStream.start for delta,
  parquet and memory runs as one announced micro-batch. foreachBatch,
  kafka and Eventstream still fall through to the overlay, and
  checkpointed streaming stays there. A sail-delta spike established that
  Sail rejects writeStream.foreachBatch at start, so wrapping durable
  sinks as foreachBatch plus a batch write is out of scope rather than
  pending.
• The JVM overlay is pinned to Spark 3.5.5 on Java 11, because Fabric
  Runtime 1.3 is Spark 3.5.5 with Delta 3.2 on Java 11 and the short
  apache/spark:3.5.5 tag is Java 17.

Fabric Core MCP, catalog search, and folders

POST /v1/mcp/core serves the published Fabric Core tools over Streamable
HTTP, wrapping the existing Core REST handlers so RBAC and LRO stay one
code path rather than two that can drift. Catalog sea...