v0.28.0
v0.28.0
The range starts at 74bfd0a (#282), the first commit after the v0.27.0
tag: 17 commits. Two are bugs a consumer would have hit, one is a behaviour
change worth reading before you upgrade, and most of the rest are witnesses for
rows that were already graded.
One behaviour change. A notebook that binds an Environment now gets its
packages installed. If you have been working around that, the workaround is no
longer needed. Nothing else changes a contract, and there is no upgrade step.
A notebook's Environment is applied, not just resolved
An Environment can be bound two ways: a Livy session names one by id, and a
notebook names one in its own # META dependencies. Only the first worked.
resolveComputeBinding is shared by notebooks and Spark Job Definitions, and
both stored the resolved Environment on their run. Only the job-definition
driver and the Livy session ever handed it to the agent. So a notebook naming
an Environment ran with none of its packages and died on the first import:
Cell 0 failed: Error: ModuleNotFoundError: No module named 'contoso_product'
while the run detail cheerfully reported the environment it had ignored.
Resolved, reported, ignored is worse than unimplemented, because the
metadata says the dependency was honoured.
The notebook driver now applies it before the first cell. It takes the Livy
behaviour rather than the Spark Job Definition one, and that is deliberate: a
notebook run is a session. A submitted job is one shot, so sparkjobdrive
refuses a JAR-bearing Environment outright; a session lets JARs be reported as
skipped and carries on, which is what an interactive notebook on a Connect
engine should do.
The docs were the reason this survived so long, and they were not wrong.
The Environments row said an Environment is "applied to the session"; the
notebook row said the emulator "resolves attached lakehouse/Environment
metadata". Both true. Read together, a consumer concludes that binding an
Environment to a notebook installs the packages, which is the natural reading
and was the wrong one. Both rows now name the path.
The witness lives in its own suite, e2e/environment-notebook, and the reason
is worth knowing if you ever extend it: the spark agent is one long-lived
process, so a package any suite installs is importable by every session after
it. Written inside e2e/environment first, the notebook negative half passed on
the Livy half's install and proved nothing. Binding a different Environment in
the same agent is refused by design, so there is no in-stack fix. Both halves
now assert the reason rather than the outcome, because a notebook submitted
before the agent finishes starting also "fails", and the first version of the
suite went green on a stack that had executed nothing.
Two fixes a consumer would have hit
The MLV refresh decoded its query twice, so it never ran on a real engine.
json.Marshal emits a quoted JSON string and Python already unquotes it as a
literal, so the json.loads wrapper received bare SQL and raised
JSONDecodeError on every refresh. The Go tests record statements rather than
executing them, which is why they never saw it; e2e/sail now runs the real
agent and reads the materialised rows back.
The eventstream drain emitted a Kusto schema real Kusto refuses.
kustoIngestTable checked column names against a character class alone, so a
KQL keyword passed. kind is an ordinary field name in event data and cannot
stand as a bare column name in a schema declaration: real Kusto answers with
400 KustoBadRequestException / SYN0002. Every name is now quoted, ['kind'],
rather than the keywords this emulator happens to know. A keyword list in the
emitter would have been a list of the names we had thought of, which is exactly
the fault that let kind through.
The destination table name in those same two commands had the identical
fault and is quoted too. That half is witnessed rather than argued: quoting a
table name touches every drain, not just one that names a keyword, so
e2e/rti asks Microsoft's own engine directly. It records the engine refusing
the bare form, accepts the quoted one in both commands, and then merges a fully
quoted ['Readings'] into a table the suite created bare and filled with four
rows, ending with three columns and four rows. Acceptance alone would not have
ruled out a quoted name resolving somewhere else and stranding a drain's rows in
a table its owner cannot see.
The agent is gated on its other consumer now
emulator-spark-agent is published here and consumed by databricks-emulator.
A break in the path only that repo exercises shipped in four releases with
this repo's suite green throughout, and the reason was structural rather than
bad luck:
e2e/livy addresses Delta BY PATH OPTIMIZE delta.`uri` -> never calls resolve()
databricks-emulator addresses BY NAME OPTIMIZE events -> nothing else
resolve() was where the break was, so no witness here could reach it, and the
consumer found out on upgrade. e2e/agent-contract now stands up Sail plus the
agent built from the published Dockerfile and drives the by-name shapes.
The release dispatch no longer announces images that do not exist
dispatch waited on fixtures alone, so it told contoso-fabric-platform a
release was ready while the images that release publishes were still building.
Measured on v0.27.0 rather than inferred: the dispatch fired at 14:36:17 and the
spark-agent image finished pushing at 14:44:03, so for 7m46s a downstream
repo had been told to verify a fabric-emulator-spark-agent:0.27.0 that was not
there.
New witnesses for rows that were already graded
Pipeline activities, each driven by the real client rather than a shaped
request: AzureFunctionActivity including key and URL, the
AzureDataExplorerCommand activity through the Kusto SDK, and Custom,
HDInsightSpark and SparkJobDefinition on a real agent. The Reflex event
trigger is witnessed with Microsoft's own Blob SDK, and fabric-cicd's own poll
loop now witnesses the forced 202 outcome.
Also: the ADX command tests use KQL real Kusto accepts, the concurrent
Delta-overwrite claim cites the sail race probe, real-fabric authenticates
via OIDC and gates on every credential it uses, and v0.26.0 and v0.27.0 got the
release notes they had been missing.
Upgrading
docker pull ghcr.io/calvinchengx/fabric-emulator:0.28.0
If you bind an Environment to a notebook, it now takes effect. Consumers that
bind-mounted /opt/wheels to compensate can drop the mount and declare only the
Environment.