v0.2.0

v0.2.0

SQL is the theme. A warehouse handle that answered REST now speaks
HiveServer2 over Thrift, and two unmodified Microsoft-ecosystem clients drive
it: databricks-sql-connector and dbt-databricks. Parity goes from 16 to
28 green rows.

Also here: the consumer toggle that lets a caller name a warehouse instead of
an id, MLflow, command execution, the Databricks CLI as a third opinion on the
REST surface, and one security fix worth reading.

docker pull ghcr.io/calvinchengx/databricks-emulator:0.2.0

Warehouse SQL over Thrift

POST /sql/1.0/endpoints/{id} — and /sql/protocolv1/o/{org}/{id} when {id}
is a warehouse — is now TBinary HiveServer2 against the same warehouse handle
and Sail attachment the REST path uses.

• databricks-sql-connector==4.4.0, unmodified, runs SELECT 1 and gets
  one typed cell back through inline COLUMN_BASED_SET.
• dbt-databricks==1.12.4, unmodified, runs dbt debug and a dbt run of
  two models where the second is a ref() of the first. GetSchemas and
  GetTables forward SHOW SCHEMAS / SHOW TABLES and remap the JDBC column
  names dbt expects.
• Cloud Fetch, Arrow+LZ4 and GetCatalogs stay refused by name rather than
  half-answered. A missing engine fails naming DATABRICKS_SPARK_CONNECT_URL.

databricks-target: resolve by name, not by id

A consumer should not have to know a warehouse id to run against both the
emulator and a real workspace. databricks-target takes the name and returns
the id and HTTP path for whichever target is selected, so the only difference
between an emulator run and a real one is configuration.

This is what contoso-databricks-platform consumes.

Security: one place may skip certificate checking, and it is named

entra-emulator, keyvault-emulator and the UC OSS sidecar each serve a
self-signed certificate, and the previous answer was InsecureSkipVerify in
four separate clients. That makes every sibling hop a man-in-the-middle
target, and scatters the exemption where no reviewer can count it.

internal/tlsclient is now the single place allowed to disable checking. Point
DATABRICKS_SIBLING_CA_FILE (or -sibling-ca-file) at a sibling's PEM and the
hop verifies properly.

Also in this release

• MLflow — experiments, run params and metrics, and registered model
  versions persist under data/mlflow/. Artifact list and log-model stay 501.
• Command execution — Python and SQL run through /api/1.2/contexts and
  /commands on a RUNNING cluster handle, on the attached Sail agent. Scala
  and R stay 501.
• Sail, UC OSS and witnessed Delta writes — jobs, SQL, secrets and
  cluster-create run against a real engine, and Connect proxies to
  DATABRICKS_SPARK_CONNECT_GRPC_URL so an HTTP agent cannot be mistaken for
  Spark Connect.
• The Databricks CLI as a witness. The SDK and the Terraform provider
  already drive this REST; the CLI is a third opinion rather than a duplicate,
  because it resolves auth itself, shapes its own requests, and has its own
  view of a well-formed response.
• A Starlight docs site, with witness keys aligned to the family ledger.

Fixed: the real-workspace leg had never run

real-databricks.yml gated on secrets in a job-level if:, which GitHub
does not evaluate there. The workflow failed to parse, ran zero jobs, and
showed red against every push — so a leg that had never executed once looked
like a measured failure.

The gate is a step now, and when the secrets are absent it prints a warning
saying in as many words that the run proves nothing about real Databricks. The
gate also requires DATABRICKS_WAREHOUSE, which the steps actually consume: a
gate that lets a job start without a value it needs fails obscurely instead of
skipping honestly.

Upgrading

Pull 0.2.0. Nothing on the existing REST surface changed shape. If you
previously relied on sibling hops skipping certificate verification, set
DATABRICKS_SIBLING_CA_FILE.