v0.30.0

v0.30.0

The range starts at 8ff0b72 (#320), the first commit after the v0.29.0
tag: 4 commits. Two of them are the same lesson from opposite directions, a
witness that could not see the thing it was witnessing, and an error that
discarded the only sentence naming its cause.

Read this if you run the built-in Airflow sidecar, or if you consume the
statement agent: the executor changed, and the agent image now carries dbt.

---

The Airflow sidecar runs CeleryExecutor, because Fabric gives a DAG no choice

The sidecar ran SequentialExecutor on SQLite, argued in its own comment as a
local fidelity target rather than a throughput one. That reading was wrong.
Microsoft documents Fabric's default as CeleryExecutor and lists
AIRFLOW__CORE__EXECUTOR among the configurations a user cannot override,
so on real Fabric it is always Celery and no DAG can opt out.

Sequential runs one task at a time, so a DAG with parallel branches serialises,
passes, and demonstrates behaviour no Fabric user can have.

The witness could not see it, which is how this survived a green e2e: the
DAG had a single task, and one task cannot distinguish the two executors. It
now fans out to three branches that each record their own window, and the
consumer asserts the wall span is well under the sum of the work:

PASS: ... ran 3 branches CONCURRENTLY (3.0s wall for 9.0s of work)

A DAG-sync failure is not a DAG that failed

Both finalised as AirflowRunFailed, whose message is the bare "The job
failed." An operator whose emulator could not write the DAG files saw a
failed run beside an empty dags folder and no reason at all, because the real
error was discarded one line after it was returned.

Found the hard way: a consumer platform's first end-to-end run failed exactly
like this. The cause was a DAG volume shared with the Airflow sidecar, coming
up 0775 airflow:root while the emulator runs distroless as uid 65532, neither
owner nor group, so no write. Finding that took a permissions audit rather than
a read of the error.

AirflowDAGSyncFailed now carries its own code and a message naming what to
check. Tests pin both halves: that sync and run failures stay distinguishable,
and that the new code has a message of its own, since a distinct code still
answering "The job failed." would move the problem rather than fix it.

The statement agent carries dbt

emulator-spark-agent now ships dbt-databricks, because
databricks-emulator
terminates dbt_task and hands this image a dbt project and a generated
profile. dbt is an ordinary warehouse client, so running it as a job changes
who invokes it, not what it connects to. Nothing in Fabric's own surfaces calls
it; the image is the family's rather than Fabric's.

It lives in a spark-agent-dbt group rather than in sail-delta, because
sail-delta also builds the engine matrix's "Sail + delta-rs" probe and that
column exists to measure the shipped runtime. An adapter the probe never calls
would make it a lookalike of the agent rather than the agent.

dbt-databricks pins databricks-sdk<0.118 where this repository's own
databricks-sdk group wants >=0.130. Both are correct and they are never
installed together, so the pair is declared in conflicts rather than resolved
by loosening one, which would silently move the version a witness runs against.

The image is smaller than it was, with dbt in it

Adding dbt first measured 894 MB → 1789 MB, which reads as doubling the image.
It was not:

	v0.29.0	v0.30.0
venv	452 MB	671 MB
/root/.cache	538 MB	gone
image	894 MB	1019 MB

Every agent image up to v0.29.0 shipped uv's download cache, wheels nothing
ever unpacks again: build input baked into a runtime image. --no-cache on the
same line removes it. The honest cost of dbt is the venv, +219 MB, and the
image grows by 125 MB net rather than 895.

The parity ledger separates two kinds of own-only row

family_parity.py fabric reports 106 of 113 green claims independently
witnessed. Seven rows were own-only, and they are not one kind of gap: three
can never have a third-party witness at all and are now marked boundary: in
the manifest, three wait on the real-Fabric leg, and one is convertible today.
So the honest denominator is 110, not 113, and 106/110 is a floor
rather than a score with three permanently unreachable rows counted against
it.

Upgrading

• The Airflow sidecar now needs a metadata DB and a broker for Celery. The
  shipped compose brings them up; a hand-assembled stack that sets
  AIRFLOW__CORE__EXECUTOR itself should stop, since real Fabric does not let
  a user set it either.
• AirflowDAGSyncFailed is a new error code. Anything matching on
  AirflowRunFailed to mean "the DAG failed" now sees the sync failure under
  its own code, which is the point.
• The agent image is 1019 MB, up 125 MB, and carries dbt.