v0.31.0
v0.31.0
The range starts at 322d20d (#325), the first commit after the v0.30.0
tag, and ends at this file: 6 changes plus their notes. One is the headline and
the rest are repairs — two of them dependency bumps that broke something on the
way in, which is the more useful half of a dependency bump.
Read this if you trigger runs on an ApacheAirflowJob after publishing DAG
files — which is every consumer of that item type. A published DAG could run
as the previous version of itself and report success.
Read the second section if you run notebooks against the Spark agent: a
dropped engine session used to poison every later statement until the container
was restarted.
A changed DAG could run as its previous version, and pass (#328)
Publishing DAG files and starting a Run raced this emulator's own scheduler.
TriggerAndWait waited for the DAG to load, which is right for a brand-new
file: until it parses there is nothing to unpause, so the wait blocks. A
changed file has the opposite shape. The DAG is already registered, every
check passes instantly, and the run is created from whatever structure is
serialised at that moment — the version published moments earlier.
The failure cannot announce itself. It is a green run whose task instances
belong to code that has been replaced, and it surfaces downstream as a task the
trigger rule references having no instance at all, or a newly added task
returning in state removed while its downstream fails. Both read as DAG bugs.
A consumer diagnosed them that way before finding this, then worked around it
by sleeping 45 seconds before every run.
That workaround could never have worked, and the reason is here rather than
there. PUT .../files only stores bytes on the item; the write into the
scheduler's DAG folder happens in the Run handler, immediately before the
trigger. A consumer sleeping before starting a run is sleeping before the files
exist on disk at all. The race is entirely inside this process, between our own
SyncDAGs and our own trigger, and no amount of consumer patience closes it.
Reproduced on v0.29.0 on demand:
published: branch_0, branch_1, branch_2, branch_3, join (5 tasks)
run got: branch_0, branch_1, branch_2, join -> failed
Two plausible signals were measured and rejected
Both look correct and are not, which is why they are recorded here:
| signal | why it fails |
|---|---|
last_parsed_time vs the file's mtime |
came back 20ms ahead of a write whose content that parse had not read — the cycle began before the write landed. It reports that a parse finished, never that this file was ingested. |
/dagSources/{file_token} vs the bytes on disk |
DagCode matched disk a full 13 seconds before the task structure changed. |
The gap is Airflow's own min_serialized_dag_update_interval, 30s by default:
the processor may read a file and skip rewriting the serialised DAG. Task
instances come from that serialisation, so it is the only thing worth waiting
for.
It also explains the consumer's history. A 15-second sleep let four stale runs
through and 45 seconds appeared to fix it; both were guesses either side of a
threshold nobody had identified.
What it does now
The Run handler reads the DAG's task set before syncing, and the trigger
waits for it to change. Taken after the sync it would be worthless — the stale
answer and the current one are indistinguishable without a baseline.
SyncDAGs now reports whether any file actually changed, and the wait is
skipped entirely when nothing did. Only a changed file can leave a
serialisation stale, and re-running an unmodified DAG is the ordinary case; it
would otherwise pay the full timeout every time. A removed file counts as a
change, so the file sets are compared and not only the bytes that arrived. An
unchanged file also keeps its timestamp rather than being restamped by the
wipe-and-rewrite.
Two deliberate non-failures. No baseline means a DAG this emulator has never
served, which cannot be stale and is already covered by the load wait. And a
change that alters no task — a callable's body, a default argument — produces
no observable difference, so the wait expires and proceeds rather than failing
every such run.
A dropped Spark session poisoned every later statement (#327)
The agent built its session once at import and never again. When the engine
dropped that session, every subsequent statement failed until the container
restarted — an open notebook simply stopped working, with nothing in its own
history to explain why.
It now detects the drop, rebuilds, rebinds every namespace, and reports what
the rebuild cost rather than hiding it.
The detection deliberately requires session beside is not running. The
latter alone also matches a stopped container or daemon, and each false
positive costs an open notebook its temporary views — a rebuild is not free,
so it must not fire on a symptom it does not own. Closes #312.
Warehouse reads parquet-go 0.32's LogicalType (#326)
LogicalType became a thrift union carrying one Value instead of a struct of
optional pointers, so the annotation is now read by type. Same three questions
in the same order; TimeUnit moved the same way. Arrived with the dependency
bump that required it, rather than after it.
Also in the range: svelte 5.56.9 with portal/dist rebuilt (#325),
kafka-go 0.4.51, and pyarrow 25.0.1 (#331).
CI bounds the package installs that could consume an entire job (#329). Nothing
in the shipped emulator changes, but it is the reason a run of this release's
own branch was cancelled rather than completed — a job spent its whole budget
installing packages, and three unrelated jobs were still in flight when the run
stopped.
Upgrading
Nothing to do. AirflowRuntime is an internal interface and its two signature
changes do not reach the HTTP API.
Consumers sleeping between publish and run can delete that sleep. It was never
buying what it appeared to buy.