v0.2.9
dbt_task works on the statement agent the platforms actually run. Before
this, every dbt_task in a compose deployment failed, and the failure said
only that the transport had broken.
docker pull ghcr.io/calvinchengx/databricks-emulator:0.2.9
Three defects, one path (#71)
Found while moving contoso-data-product's gold step from a host dbt
subprocess onto Jobs. All three were invisible to ci:e2e-dbt-task, which
runs against emulator-spark-agent while every platform runs
fabric-emulator-spark-agent (#72).
The generated profile carried the client's address
The profile was built from -public-url, the origin advertised to clients.
In compose that is a host-published address like http://127.0.0.1:18470, and
the agent is another container, where that resolves to the agent itself. dbt
never reached the warehouse.
This is the mistake dbtProjectFiles already documents for the project files,
one field over: a path meaningful in this process is meaningless in the agent.
New -agent-url / DATABRICKS_AGENT_URL names the origin the agent uses,
defaulting to -public-url, so a deployment where one name reaches this
process from everywhere is unchanged.
A dbt failure left as an exception, and took the evidence with it
v0.2.7 printed run_results.json before re-raising, so a failing dbt test
still returned its artefacts. That ordering assumes every agent turns an
exception into a reply. One does not:
raise SystemExit("boom") -> connection closed, no response at all
raise ValueError("nope") -> {"status":"error","evalue":"ValueError: nope"}
So the caller saw Post /statements: EOF — a transport error — on a run where
dbt had simply reported failing tests, and the artefacts, already printed and
correct, died with the response that would have carried them.
Ordering cannot fix that. Not raising can. The failure is now a field in the
same envelope as the artefacts, {"artifacts": {...}, "failure": "..."}, and
nothing in the generated code raises for it. An agent cannot drop what it does
not have to interpret, and the verdict can no longer be separated from its
evidence. A genuine Python fault still raises and is still reported the
ordinary way.
The missing-dbt-databricks SystemExit became a RuntimeError for the same
reason: otherwise that explanation also reached the operator as EOF.
spark_env_vars never reached dbt
The agent accepts an env field in the request and silently ignores it —
status ok, value None. A project reads its sources through env_var(), so
gold failed with "Env var required but not provided: 'LAKEHOUSE_ID'" for a
variable the task had plainly set.
The task's spark_env_vars are now set inside the generated code, before
dbt parses, which is the only version of this that is true on every agent. The
general case — every task kind, and the {{secrets/...}} references that ride
on this same field — is #73.
Upgrading
Nothing is required. If your emulator and your agent reach each other by a
different name than your clients use, set DATABRICKS_AGENT_URL; in compose
that is the service name, e.g. http://databricks:8447.