Skip to content

v0.32.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Aug 05:49
· 49 commits to main since this release
v0.32.0
ea7dfda
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.32.0

The range starts at 9a8f484 (#332), the first commit after the v0.31.0
tag, and ends at this file: 10 changes plus their notes. One is a data-loss fix
that consumers of this image will want promptly, one is a T-SQL reflection
repair, and the rest are a witness, docs gates and dependency bumps.

Read the first section if you run more than one task at a time on the Spark
agent — two concurrent tasks could read each other's parameters and each
other's resolved secrets, both reporting success. It is the reason
databricks-emulator currently documents for_each_task concurrency > 1 as
a known limitation.

Read the second if you reflect Warehouse tables: a UUID or BSON column came
back as varchar, so a client round-tripping the type wrote text into a binary
column.

Two concurrent tasks could read each other's parameters (#338)

The statement agent isolates a session's user globals — a name bound in one
session is not visible in another — but it did not isolate module state, and
sys is one object per interpreter. Task parameters are delivered by assigning
sys.argv, and resolved secrets by updating os.environ, so two tasks
dispatched in the same wave overwrote each other and each read whichever
assignment landed last.

Both tasks report SUCCESS. That is the shape that matters: no error, no warning,
and a task that quietly processed another task's inputs. A loop that writes one
output per input silently produces fewer outputs than it has inputs, and the run
is green.

Measured against the published 0.31.0 agent, three concurrent statements in
three distinct sessions, each told to write a table named by its own parameter:

alpha  ok     asked alpha wrote alpha
beta   ok     asked beta  wrote alpha     ← read another session's argv
gamma  error  CommitFailedError: Protocol changed since last commit
tables on disk: ['alpha']

One correct, one silently wrong, one killed by the Delta commit conflict that
follows from every writer targeting the same table. With this release:

tables: ['alpha', 'beta', 'gamma']   — each holding its own value

task_scope.py gives each session its own sys.argv and os.environ around
execution, and python/tests/test_spark_agent_task_scope.py holds it.

Consumers should bump their pinned agent digest. The tags do not move on
their own: emulator-spark-agent:4.2.0 and :latest are rebuilt by this
release, so a consumer pinned by digest — which is the recommended form, and
what the platform repos use — stays on the pre-fix image until it bumps.
databricks-emulator's e2e/composite-tasks is the known case, tracked as
databricks-emulator#64.

Not fixed here: captured stdout still crosses sessions, so a task's
printed output can be attributed to another concurrently running task. The data
a task writes is correct; the log line saying it did may appear under a
sibling. Scoped separately from this fix, which covers sys.argv and
os.environ.

A UUID or BSON column reflected as varchar (#340)

Warehouse reflection reported uniqueidentifier and BSON columns as varchar.
A client that reflects a table to learn its shape, then writes back what it
read, put text into a column the engine treats as binary. The repair returns
varbinary, which is what the type is.

Class B strict mode, witnessed by a real TDS client (#341)

-tsql-strict was graded on this repository's own assertions. It is now driven
by an unmodified TDS client, so the row rests on something that is not us. Same
conversion the family has been applying to every go:-only claim.

Docs gates and dependency bumps

  • #342 — the docs gate checked sidebar reachability but not the links
    themselves, so a link to a page that had moved passed. Both are checked now.
  • #343 — entra-emulator 0.9.0.
  • #334 — great-expectations 1.20.0, and the numpy 2 move it required.
  • #333 — mcp 2.0.0.
  • #332 — kafka-python 3.0.11.
  • #336 — Go 1.26.6, plus secret and vulnerability scanning in CI.
  • #337 — dbt_expectations measured on both dbt adapters rather than
    assumed to behave the same on each.

Upgrading

Nothing in this release changes a documented API shape. Two things are worth
doing rather than assuming:

  1. Bump the pinned emulator-spark-agent digest, or the concurrency fix
    above does not reach you. docker buildx imagetools inspect ghcr.io/calvinchengx/emulator-spark-agent:4.2.0 reports the new one.
  2. Re-check any suite that pinned concurrency: 1 to work around #338.
    The workaround is no longer needed, and leaving it in place hides whether
    the fix reached you.
Assets 13
Loading