v0.27.0

v0.27.0

The range starts at 53a09f6 (#279), the first commit after the v0.26.0
tag: 3 commits. Small, and all three are about the same thing from different
directions: code paths that were documented, graded, or shared, and never
actually run.

Nothing here changes a contract. There is no upgrade step.

---

The spark-agent stops assuming one consumer's habits

databricks-emulator pulls this repo's spark-agent image, and its delta
witness died on a statement it never wrote:

MERGE INTO events AS t ...
IllegalArgumentException: invalid argument: found DETAIL at 9:15

Three defects, each hidden behind the one before it, and the first two are the
same mistake: the agent is shared by two emulators, and each defect took one
consumer's habits for the world.

1. _CREATE_DELTA_LOCATION required USING immediately after the table
   name. That is dbt-fabricspark's shape, so every witness in this repo was
   green. The ordinary CREATE TABLE events (id INT, name STRING) USING delta LOCATION '…' did not match, so the location was never recorded, and a MERGE
   two statements later fell through to a DESCRIBE DETAIL that Sail has no
   grammar for. A missed record has no symptom at the point it happens. The
   pattern now takes an optional column list, paren-aware so DECIMAL(10,2)
   survives, and clauses between USING and LOCATION. Writing the shapes
   down found a second, latent one in this repo: PARTITIONED BY before
   LOCATION was unrecorded too.

2. resolve() let the engine's parse error escape. The module's own
   comments say Sail cannot parse DESCRIBE DETAIL, and then it issued one
   anyway and let found DETAIL at 9:15 surface, pointing at column 9 of the
   user's MERGE. It now raises DeltaOpError naming the table, keeping the
   engine's words as detail rather than as the headline.

3. localRelationSizeLimit is now asked for, not assumed. With the
   fall-through fixed, MERGE reached delta-rs and failed while returning its
   result: delta_ops answers via spark.createDataFrame, and pyspark calls
   int() on the served conf. Sail 0.7.0 serves '3221225472'; the older Sail
   that databricks-emulator pins serves '3GB'.

   v0.26.0 removed the preset on 0.7.0 evidence, which was correct for this
   repo, and that measurement was taken while defect 1 masked the case that
   needed it. Neither constant is right for a shared image, so connectconf.py
   now reads the value and rewrites only what will not parse, to the same size.
   No engine's limit is reduced, and no client is handed a string it chokes on.

The guard is the interesting part. python/tests/test_agent_consumer_contract.py
records each statement shape a named consumer actually sends, cited to the file
it comes from, with the answer the agent owes it. No engine, milliseconds, runs
on every PR. It fails in the repo that would ship the regression rather
than the repo that would suffer it, which is the property a shared artifact
needs. Four of its rows fail against the code before this change.

Verified on both engines rather than only the one this repo pins:
databricks-emulator's e2e/delta on Sail 0.22.0 advances the log to v4,
confirmed by delta-rs; this repo's e2e/sail on Sail 0.7.0 passes.

FABRIC_DATABRICKS_URL is finally exercised

It was documented in the README, in docs/04-configuration.md, in the v0.25.0
notes and in internal/api/databricksremote.go, and it appeared in no
workflow anywhere.

That matters because the parity row grades the Databricks activities 🟢 Real
(notebook + python, local or FABRIC_DATABRICKS_URL). The local half ran on
every push. The remote half had never executed once. So this is primarily an
unexercised code path getting exercised; the third-party witness is a side
effect.

The chain is entra → fabric → databricks-emulator → spark-agent → sail, plus
the client. Sail and the agent build from this checkout, so the chain
breaks when this tree breaks it. A pinned image would have hidden exactly that.

Five more pipeline activities witnessed

Including the WebHook park, which is the one that holds a run open waiting for
a callback rather than completing on its own.

Upgrading

Nothing to do.

If you consume this repo's spark-agent image from another emulator, you get the
CREATE TABLE … LOCATION and localRelationSizeLimit fixes above, and a
DeltaOpError that names the table instead of an engine parse error pointing
at the wrong column.