v0.25.0

v0.25.0

The range starts at 707688c (#218), the first commit after the v0.24.0
tag: 26 commits. The short version is that three surfaces stopped being
item management and started executing. Eventstream gained destinations
and operators, the Azure Batch Custom activity runs by default, and
Databricks activities can submit to a real workspace. Alongside those,
DAX gained an optional real oracle and its first Desktop-pinned scalars,
and the conformance kit landed with write landing asserted out of band on
all three backends.

Two of those change behaviour on upgrade. Read the first section before
you pull.

---

Behaviour changes, read this one first

• The Azure Batch Custom activity now runs by default. It used to be
  refused unless opted in, which made every pipeline that actually uses
  Batch a false pass: the activity reported nothing useful and the run went
  green. A Custom activity's command is a shell process on the Spark
  agent, which is the same machine a notebook cell can already
  subprocess.run on, behind the same bearer and the same workspace RBAC,
  so refusing it by default was protecting nothing.

  The command executes in the agent's container rather than the
  emulator's process, so the blast radius is the engine and not the API.
  extendedProperties become environment variables, as Batch documents,
  and the command's own exit code decides the activity: a non-zero exit
  fails it with that code and the command's stderr, and a report that
  cannot be read fails too, because an unknown exit status is not success.

  If you have a pipeline with a Custom activity, its command now
  executes. FABRIC_CUSTOM_ACTIVITY=off restores the refusal so nothing
  reaches the agent; the old opt-in spelling shell still means on.
  Batch-node features (resourceLinkedService / folderPath,
  autoUserSpecification, referenceObjects) stay refused by name even
  when the activity is on.

• input_file_name() in a SQL string now resolves, or errors. The
  agent's shim tags each file's rows at read time and points
  F.input_file_name at the tag, but spark.sql("SELECT input_file_name() …") never touched the patched function and failed on the engine as if
  the shim were not there. spark.sql is now wrapped beside the PySpark
  patch: a lexer finds the call in code and leaves strings and comments
  alone, a view built from a tagged frame registers a clean name plus a
  shadow that still carries the tag, and the query is rewritten onto that
  column.

  A UDF could not have fixed this, which is the part worth recording. The
  function takes no arguments, so a registered UDF cannot see which row it
  is evaluating and could only return a constant, which is the
  silently-wrong-lineage failure the module exists to refuse. A relation
  that was never a file read now raises InputFileNameError rather than
  resolving to "".

• The sc facade refuses toDF() on scalars, as real PySpark does.
  Seq(1,2,3,4).toDF() is Scala, and the facade had been accepting the
  Python transliteration of it. The contract file now pins refusals as
  well as answers, so over-permissiveness is checkable rather than only
  under-permissiveness.

Eventstream executes end to end

v0.23.0 and v0.24.0 left Eventstream as item management plus a raw Kafka
topic. This range closes the topology.

• Kafka on both engines (#226). Fabric notebooks resolve
  eventstream.itemid / eventstream.datasourceid against a real Apache
  Kafka KRaft broker. Sail consumes the records and builds the Kafka schema
  into the engine, so CAST(value …) runs on Sail rather than the source
  being mapped onto rate. The JVM overlay keeps the native
  spark-sql-kafka source. subscribePattern, assign, JSON offsets,
  includeHeaders, SASL PLAIN, PEM SSL, GSSAPI and JKS/P12 stores are all
  honoured, with the Java-shaped ones converted in the wrap because
  kafka-python wants PEM and a ticket rather than Java stores. OSS
  format("kafka") reads and writes work on Sail too. Checkpointed
  streaming stays on the JVM overlay. The broker is apache/kafka (ASF,
  Apache-2.0, multi-arch) rather than Redpanda, whose license moved to BSL.

• Three destinations and three operators (#231, #237). Custom HTTP
  produce now feeds a bound Lakehouse Delta table, a Reflex
  EventReceived trigger fired as a real item job, or an Eventhouse
  KQL database through direct ingest (.create-merge plus .ingest inline, which is the path kustainer actually supports, not Fabric's
  streaming-ingest protocol). Filter, GroupBy and tumbling Window run
  on the produce batch between source and destination. Kafka stays the raw
  source; destinations see operator output.

  Binding is an emulator-native surface (POST …/eventstreams/{id}/ destinations and …/operators) because Fabric's topology has no public
  REST, the same situation Reflex triggers are already in. Join, Union,
  Expand and hopping or sliding windows are refused by name: they need more
  than one stream or cross-batch state.

Databricks activities can reach a real workspace

Set FABRIC_DATABRICKS_URL (plus FABRIC_DATABRICKS_TOKEN, and
FABRIC_DATABRICKS_TLS_INSECURE for a self-signed emulator cert) and
DatabricksNotebook / DatabricksSparkPython import the workspace file
and submit through Jobs 2.2. dbfs: and /Workspace paths become legal
only then, because without a workspace to submit to, reinterpreting a
Databricks path as a lakehouse path would invent a mapping nobody wrote.

Unset keeps the existing behaviour: the submission contract terminates
locally and the Spark agent executes the code. DatabricksSparkJar and
libraries stay refused on both paths.

A real DAX oracle, and the first Desktop-pinned scalars

The bounded Go evaluator stays the default on every OS, and empty
FABRIC_DAX_URL is what GitHub-hosted ubuntu and mac CI tests. What is new
is that a machine which can run Windows may now attach real VertiPaq:
FABRIC_DAX_URL points executeQueries at a pump in front of msmdsrv
(POST /v1/deploy, then POST /v1/dax). Set-but-unreachable is a 502 and
not a silent fallback to the internal evaluator, because those two answers
are different facts. This is not a compose default and not a
macos-latest / ubuntu-latest sidecar; docs/52
says what each host actually needs.

The point of an oracle is what it settles. Five functions are now pinned
against goldens captured from Power BI Desktop, and every one of them
caught a difference from the obvious implementation:

• ACOS (#236), captured alongside the discovery that Desktop rejects a
  DATATABLE with a missing sourceColumn and an empty catalog name. The
  pump maps both to 409 and retries without Initial Catalog=, so a
  still-open .pbix stays queryable, which is what makes DATATABLE
  deployable at all.
• ABS and ROUND (#242). ABS(BLANK()) stays BLANK. ROUND is half away
  from zero for negatives as well as positives, and a negative digit place
  rounds to tens or hundreds, so ROUND(1234, -2) is 1200. Multiplying by
  a power of ten and calling math.Round is not enough to match: 2.15
  shifted one place is IEEE 21.4999…, which floors to 2.1 where Desktop
  says 2.2, so the shift goes through a decimal exponent string instead.
• LOG and LOG10 (#253). Desktop's LOG defaults to base 10, and
  BLANK, non-positive arguments, and base 1 all error, which is not what
  EXP(BLANK()) = 1 would have led you to guess.

Every-push CI replays the captured goldens against the Go evaluator with
FABRIC_DAX_URL empty, so the oracle's answers keep gating the default
path even where no Windows host exists. Headless msmdsrv was measured
too (docs/33, Phase 0c): it listens, but ROW
needs a table, so Desktop remains the oracle.

Framework conformance: write landing is asserted out of band

The conformance kit landed (#235) and then got live backends (#243).
Contract 4, write landing, is now ✅ on sail, jvm and warehouse, and
the assertion is deliberately not the writer's own catalog: a notebook run
plus an out-of-band OneLake DFS listing, or a TDS INSERT plus a fresh
SELECT on a new connection. A writer that would confirm its own write
is refused before any I/O runs. That was the false-green shape the kit
exists to remove.

Contracts 1 to 3 and 5 to 7 stay ❌ with a pointer to the prose that closes
each one, rather than a silent skip. Wiring the JVM Livy session for that
row also fixed a real defect: spark-submit had been building a session
with neither the Delta catalog nor the Entra token provider, so
saveAsTable failed with
DELTA_CONFIGURE_SPARK_SESSION_WITH_EXTENSION_AND_CATALOG.

The Sail column closes four more rows

• MERGE, change data feed, and JSON multiLine (#220). CREATE TABLE … LOCATION is recorded so DESCRIBE is real, subquery MERGE and
  INSERT * are intercepted, and the CDF and multiLine notebook APIs are
  wrapped, announced and materialised.
• Durable streaming sinks (#223). writeStream.start for delta,
  parquet and memory runs as one announced micro-batch. foreachBatch,
  kafka and Eventstream still fall through to the overlay, and
  checkpointed streaming stays there. A sail-delta spike established that
  Sail rejects writeStream.foreachBatch at start, so wrapping durable
  sinks as foreachBatch plus a batch write is out of scope rather than
  pending.
• The JVM overlay is pinned to Spark 3.5.5 on Java 11, because Fabric
  Runtime 1.3 is Spark 3.5.5 with Delta 3.2 on Java 11 and the short
  apache/spark:3.5.5 tag is Java 17.

Fabric Core MCP, catalog search, and folders

POST /v1/mcp/core serves the published Fabric Core tools over Streamable
HTTP, wrapping the existing Core REST handlers so RBAC and LRO stay one
code path rather than two that can drift. Catalog search and the remaining
folder REST landed with it. The witness is the unmodified Python mcp
SDK in CI, and an az rest driver is a second stranger on the
control-plane rows those tools sit on.

Purview lineage is derived, not stored

A Process that reads A and writes B is the edge, so there is no lineage
table (#238). The inputs / outputs attributes Process lacked are now
seeded, and the Atlas lineage endpoint walks them with depth, width and
direction bounds. Tests came with the second commit and immediately earned
their place: soft delete had been writing ACTIVE back onto the status
column, so a deleted Process still appeared in the graph.

Honesty fixes in the project's own CI

Five gates that reported green while doing nothing:

• The sc-facade oracle never ran (#222). job.py computed a host
  path for rdd_contract that does not exist in the container, so the JVM
  check raised ModuleNotFoundError and was never reached. Mounting the
  contract beside job.py made it run, and its first working run found the
  toDF defect above. An oracle that has never executed is not evidence,
  which is the same standard applied below.
• Three make check gates CI never ran (#221):
  check_example_portability, check_capture_redaction and
  check_entra_install were enforced only on a laptop. A test now asserts
  the two lists agree, so the next one cannot drift silently.
• A schedule-only workflow is not proved by the next commit (#224).
  Three such fixes landed in one week unexecuted and one was broken, so CI
  now asserts every cron-only workflow has run since it last changed. The
  checker is stdlib-only (its own jobs lack pyyaml), reads last-changed
  from the API pinned to HEAD because actions/checkout fetches depth 1,
  and asserts only where the API is reachable, with FRESHNESS_STRICT=1 on
  the witnesses job so a permissions regression there still fails loudly.
• ci:real-fabric is no longer cited as a witness (#239). It is gated
  and has never executed, so it is not evidence. The claims that named it
  now cite examples/medallion-pyspark, whose bronze pipeline really does
  move CSV into Tables/ and assert the bytes; git-integration, folders,
  CopyJob and RBAC stay on Go tests, because those CI jobs do not call
  those APIs.
• entra bound to localhost in e2e (#227), so every token mint from
  the sempy container, which dials host.docker.internal, was refused.
  Only the labs leg was affected, because sempy uses a pre-minted token,
  which is why the suite failed selectively.

The e2e fleet also gained a witness with unmodified microsoft/fabric
Terraform (#228). The provider is a packaged Microsoft client in the same
sense az rest is, and driving it forced the seeded capacity region and
the WorkspaceInfo / lakehouse GET payloads onto the tenant shape the Go
SDK actually reads.

Packaging, security, and docs

• cryptography is bumped to 49.0.0 for the three open wheel and
  path-building advisories (#230).
• spark_agent must run on Python 3.8, and now says so (#229). The JVM
  image ships 3.8; eventstream_kafka imported datetime.UTC (3.11) under
  ruff's UP017 with a py312 target and broke the JVM suite on main. There
  is a test for the floor now.
• entra is pinned to 0.7.0 (#219), so the family BOM can move.
• The README and the docs site match what the emulator runs (#234). The
  homepage still said nothing computes and the README still counted 89
  greens; both now describe Eventstream destinations, Custom on by default,
  Core MCP and the optional DAX oracle. Witnessed parity claims went from
  107 to 118 across this range.

Upgrading

Pull 0.25.0. Existing workspaces, items and OneLake contents are
unchanged, and nothing new is contacted unless you set a variable for it:
FABRIC_DAX_URL and FABRIC_DATABRICKS_URL are both empty by default,
and Kafka stays behind --profile eventstream.

The upgrade you may have to act on is the Custom activity. A pipeline that
was passing because the activity was refused will now run its command in
the agent container and fail on a non-zero exit; set
FABRIC_CUSTOM_ACTIVITY=off if you want the old refusal back. Notebooks
that call input_file_name() in a SQL string get a real answer instead of
an engine error, and get InputFileNameError where the frame was never a
file read.