Shape 0.9.3

0.9.2 was published to TestPyPI only and was never released to PyPI; its changes are part of 0.9.3.

Added

• Synapse dedicated SQL pool live test. The Fabric plugin ships a live test that writes
  to a real Synapse dedicated SQL pool, driven by SHAPE_TEST_SYNAPSE_URI,
  SHAPE_TEST_SYNAPSE_STAGING and the service-principal settings
  SHAPE_TEST_SYNAPSE_TENANT_ID, SHAPE_TEST_SYNAPSE_CLIENT_ID and
  SHAPE_TEST_SYNAPSE_CLIENT_SECRET. The fabric-live workflow resumes the pool before the
  tests and pauses it afterwards.
• --source-config KEY=REF (global option). Passes an Azure read credential to a command
  by reference. KEY is token, account_key, sas_token or connection_string; REF is an
  env://, file:// or kv:// reference. A literal secret is refused, and the value lives only
  for that command.
• --lakehouse-schema NAME on shape generate --scale-mode fabric_spark and
  shape demo init. Writes the tables into a schema of a schema-enabled Lakehouse.
• --table-prefix now works with the Delta sink and with the database and Fabric
  destinations (PostgreSQL, MySQL, Snowflake, Databricks, SQL Server, Warehouse, Synapse, SQL
  database, Lakehouse, Eventhouse). Where a destination cannot honour a prefix, the command stops
  before writing and says so, instead of ignoring the option.
• Two timeout settings for Fabric SQL writes. SHAPE_SQL_QUERY_TIMEOUT (whole seconds,
  default 900) limits each SQL statement of the Warehouse, SQL database and Synapse writers, so
  a stalled load fails instead of waiting forever. SHAPE_WAREHOUSE_INSERT_SECONDS (whole
  seconds, default 600) limits how long the slower row-by-row Warehouse fallback may take per
  table. A value that is not a whole number is ignored and the default applies.

Verified

• Kerberos sign-in. Shape 0.9.1 --auth kerberos was verified on 2026-10-12 against a
  lab Active Directory domain with SQL Server 2022 (Windows authentication only), from an
  Ubuntu 24.04 client: 21,750 rows in 9 tables, exit 0, and the server recorded
  auth_scheme = KERBEROS. A keytab with mode 644 and a wrong principal both exit 2. No code
  changed for this.

Fixed

• Nested columns (issue #1). shape profile now profiles Parquet, Delta and JSON Lines
  tables that hold list, struct or map columns, where it stopped with unhashable type: nested Arrow column. The profile records the column's exact Arrow type, its null rate and, for a list
  or a map, the distribution of its lengths; the other columns are unchanged. diff, check,
  inspect and generate --from work on such a profile, and generate writes the column with
  the same Arrow type. generate stops with an error that names the column for a union,
  extension, run-end encoded or view type inside a nested column, and shape fidelity refuses a
  nested column by name. An error whose message holds a backtrace is now one shape: error: line.
• Local Delta tables no longer hang at exit. shape profile (and every read of a local Delta
  table) could hang, or abort with terminate called without an active exception, as the process
  ended after a failed read (for example a damaged data file or a nested column). A local table is
  now read through Arrow's own file system.
• shape profile-model / semantic-model://. A table the semantic model cannot read stops
  with one error naming the model and table and suggesting a refresh, instead of the raw
  sempy/ADOMD error.
• Warehouse and Synapse primary keys. Fabric Warehouse rejected a primary key written
  inside CREATE TABLE (error 24584). Shape now creates the table first and then adds the key
  with ALTER TABLE ... PRIMARY KEY NONCLUSTERED ... NOT ENFORCED. SQL Server and Azure SQL keep
  their enforced keys.
• Time ranges stay text. A column of values such as 18:00-18:59 is no longer read as a
  UTC offset when profiling; it stays text.
• Profiling a database with zoned text dates. shape profile-db no longer stops at a
  sampled text column that holds dates with a time zone. The column keeps its SQL type and
  statistics, is left out of relationship inference, and carries a profiling_note saying so.
• Schema-enabled Lakehouses. Tables are written under Tables/<schema>/<table>, where
  the SQL endpoint and semantic models can see them. Shape asks Fabric which kind of Lakehouse
  the target is, uses dbo unless you give a schema, refuses a schema for a classic Lakehouse,
  and, when it cannot tell, stops and tells you to set schema_enabled=true|false. Before, the
  Delta sink assumed a classic Lakehouse and wrote the tables where schema-enabled Lakehouses
  could not see them. Writing uses one extra read call to Fabric. This needs a credential; with
  only a raw storage token, give the schema in the path (.../Tables/dbo).
• Missing Delta tables are reported as no Delta table at <path>, with credentials
  removed from the path.
• The abfss:// sink writes into a Fabric Lakehouse. It used to fail with
  OperationNotAllowedOnThePath, because it checked for the Lakehouse with a request OneLake
  refuses on an item folder. It now finds the Lakehouse in the workspace listing. A Lakehouse
  given by GUID is no longer checked up front; a missing one is reported by the first write.
• MySQL without TLS. The caching_sha2_password sign-in crash is replaced by a message
  that says to use verified TLS (ssl_ca=PATH&ssl-mode=VERIFY_CA) or ask the administrator for
  a supported authentication method. Shape does not fetch an untrusted key.
• Azure storage credentials. Storage environment keys are now used in a fixed order:
  explicit options, then AZURE_STORAGE_ACCOUNT_KEY, AZURE_STORAGE_SAS_TOKEN,
  AZURE_STORAGE_CONNECTION_STRING, then the Fabric or Synapse notebook identity, then
  DefaultAzureCredential. An ambient account key no longer overrides the identity you selected
  for OneLake. A sign-in error that cannot succeed on retry (a tenant mismatch or a disabled
  account, for example) now fails once with one clear message; timeouts and throttling still
  retry.
• Warehouse loads when COPY INTO fails. When OneLake cannot supply the storage token
  (SQL error 13840), the load falls back to parameterized INSERT in batches of 5,000 rows,
  keeping keys, write modes, rollback and the loaded-row check. The fallback is slower and is no
  longer silent: it prints a WARNING on stderr, records the note in WriteResult.notes and
  summary(), and fails with a rollback past SHAPE_WAREHOUSE_INSERT_SECONDS. Batched
  inserts now size text columns from the whole batch, not the first row, so they no longer
  fall back to one round trip per row.
  Emptied OneLake staging folders are removed after a load.
• Fabric notebook deploy. Fabric sometimes answers with a long-running operation address
  on a regional *.analysis.windows.net host. Shape now polls the Fabric API for that
  operation instead of refusing it, and never sends your token to the other host.
• Fabric errors say what happened. Errors show the HTTP status and Fabric's code and
  message, with secrets removed. A failed Spark job keeps Fabric's error code and details, or
  says Fabric gave no reason.
• Fabric portal connection strings. The ADO.NET-style string from the Fabric portal
  (Data Source=...;Initial Catalog=...) is accepted by the SQL writers and shape profile-db.
• Semantic models for service principals. If Fabric's direct lookup fails for a service
  principal, Shape lists models over REST and warns that it did.
• Synapse. The dedicated-pool writer sends its DDL in autocommit, stages timestamps in
  Parquet as INT96, and writes NaN and infinite values as NULL.
• Fabric Git sync test. The nightly job and fabric-live use FABRIC_GIT_WORKSPACE_ID
  (the Git-connected workspace) in place of FABRIC_WORKSPACE_ID; tenant, client and remote
  are variables, the client secret and Git token stay secrets. This affects people who run the
  live tests, not users of the package.
• shape profile validate --safe and wide or multi-table profiles. Multi-table
  --dataset profiles and single wide tables now validate clean. A profile records which
  columns an internal sample covered as positions, sampling.internal[].column_indexes, in
  place of names (columns). The profile format version is unchanged, and profiles written by
  0.9.1 still load and report the same findings until you profile again. Profile content ids
  change for the same data.
• Privacy. A scenario pack's stream topic emits only its payload_fields columns, and
  pack validate rejects a field that is not a column. A pack run creates its manifest
  exclusively, so two runs in the same second keep their own manifests. shape chaos --input
  and shape incremental --input skip the ground-truth logs of an earlier chaos run and name a
  file that cannot be read as a table. shape registry ROOT commit runs its leak scan on any
  JSON document with tables, so removing redaction_manifest no longer lets a leaking
  profile through. shape.yml gates.NAME.mode also accepts the four gates only
  shape verify runs (reconciliation, timeseries_quality, memorization, utility).
  shape fabric setup names shape fabric notebook in its next steps.
• VS Code "Compare with Git HEAD". The comparison document now opens; the custom
  editor had claimed it and no diff appeared.

Changed

• Default Lakehouse name for shape fabric setup is shape_lakehouse (was
  shape-lakehouse). Fabric rejects the hyphen with HTTP 400. Names must begin with a letter and
  use only letters, digits and underscores (at most 123 characters); Shape now checks this before
  any request.
• Replaced the 49 bundled AMA CPT procedure values and descriptions with synthetic
  SP-01 through SP-49 and Synthetic procedure NN. The letters and hyphen distin...