v0.2.7
v0.2.7
A dbt_task now hands back what dbt wrote. A failing test used to report that
something failed and never which one.
docker pull ghcr.io/calvinchengx/databricks-emulator:0.2.7
run_results.json comes back from a dbt_task (#61)
v0.2.6 ran dbt inside the agent and discarded everything dbt produced. The
caller got an exit code and stdout, so a failing dbt test said only that
something failed — not which test, on how many rows, or with what message.
That is the difference between a run that failed and a run that recorded what
it measured, and it is what stopped a consumer moving its gold step onto
Jobs: its snapshot carries a contract_failures list built from
target/run_results.json, and it asserts args.which == "test" before
believing the artefact, so a dbt run result cannot be mistaken for a test
verdict. Through a dbt_task that list went dark — the right execution shape
bought at the cost of the evidence the execution exists to produce.
The ordering is the fix. The agent collects run_results.json and prints
it BEFORE re-raising the failure. A failing dbt test is exactly when the
artefact is worth having; emitting it after the raise would surface it on every
run except the ones that matter. A test asserts that ordering in the generated
source and fails if the two are swapped.
run_results.json alone — manifest.json is large, changes on every parse,
and no caller has asked for it.
Read it from runs/get-output:
dbt_output: { artifacts: { "run_results.json": "<the file, verbatim>" } }
A deviation, named here rather than left to be discovered. Real Databricks
returns artifacts_link and artifacts_headers — a URL the caller then
fetches. This returns the file inline. A link would need an artefact store,
an expiry and a second round trip to emulate handing back one JSON file, so
code written against the real API will not find artifacts_link here.
dbt_output is absent rather than empty when a run produced no artefacts,
so "produced none" and "was not a dbt run" stay distinguishable.
The payload rides through stdout, which is the agent's only channel back, and
is stripped before the logs are stored so logs remains what dbt printed. A
payload that is absent, unterminated or unparseable yields no artefacts and
unchanged stdout: losing an artefact must not also lose the log that would
explain why.
Docs and CI (#59, #62)
The site build exited 0 on a dangling intra-doc link and could not see a page
missing from the sidebar; both are now gated. The security jobs have a timeout,
and the history scan names what it finds instead of reporting a bare failure.
Upgrading
Nothing to do. dbt_output is additive, and a consumer that ignores it sees
the same logs and error as before.
A consumer that reads contract_failures from a host-run dbt can now get the
same artefact from a dbt_task — which is the point of this release.