feat: support schema evolution DDLs (ADD/DROP/RENAME/ALTER COLUMN)

[puchengy]

Closes #62.

Adds support for the schema-evolution DDLs requested in #62 by extending BaseLanceNamespaceSparkCatalog.alterTable to handle Spark TableChange.ColumnChange requests, delegating to the underlying Lance dataset schema-evolution API via a new LanceSchemaEvolution helper.

Supported operations

DDL	Lance API
ALTER TABLE ... ADD COLUMN[S]	Dataset.addColumns
ALTER TABLE ... DROP COLUMN [IF EXISTS]	Dataset.dropColumns
ALTER TABLE ... RENAME COLUMN	Dataset.alterColumns (rename)
ALTER TABLE ... ALTER COLUMN ... DROP/SET NOT NULL	Dataset.alterColumns (nullability)

Atomic, all-or-nothing semantics

To honor Spark's alterTable contract, an accepted request is validated in full against the current schema and then committed through exactly one Lance core operation (addColumns / dropColumns / alterColumns — each commits its whole batch atomically):

• Same-kind changes are batched into that single call (e.g. ADD COLUMNS (a, b); rename + nullability edits to the same column are merged into one ColumnAlteration).
• Requests that would require more than one core mutation are rejected before the first write: mixing column additions, drops, and alterations in one statement, or combining any column change with TBLPROPERTIES changes (which commit separately). This keeps a rejected ALTER TABLE from leaving the table partially mutated. Heterogeneous batching in one commit is not yet supported by the core.

Deliberately unsupported (rejected before any mutation)

• ALTER COLUMN ... TYPE — throws UnsupportedOperationException. The lance-core:10.0.0-rc.3 JNI create_column_alteration reads the cast target from the Java ArrowType.toString() (e.g. "Int(64, true)") and parses it with Rust DataType::from_str, swallowing the parse failure with .ok() — so the cast target is dropped and the type change becomes a silent no-op (the commit lands but the stored type is unchanged; verified for both integer- and float-widening). This is fixed upstream in fix(java): carry alterColumns cast type across FFI via C Data Interface lance#8417 (carries the cast type across FFI via the Arrow C Data Interface), but that fix is not yet in a published org.lance:lance-core artifact — the newest published version (11.0.0-beta.3) predates it, and Lance does not publish nightly/snapshot Java builds. Once a lance release containing #8417 ships, this can be enabled by bumping <lance.version> and replacing the rejection with the castTo path.
• ADD COLUMN with a DEFAULT value — Lance's all-null add cannot backfill a default, so it is rejected rather than silently filling NULL.
• ADD COLUMN on a legacy-format (file_format_version='LEGACY') table — the core all-null add path rejects legacy datasets; rejected up front with a clear message instead of a raw core error.
• Positional (FIRST/AFTER) adds, nested-column adds, and column-comment updates.

Tests

• Java unit tests in SparkLanceNamespaceTestBase (run against both directory and REST namespaces): ADD/DROP/RENAME COLUMN and ALTER COLUMN DROP NOT NULL happy paths; batched multi-column ADD; and rejection cases for mixed-kind requests, column-change-plus-TBLPROPERTIES, DROP COLUMN IF EXISTS on a missing column, DEFAULT values, legacy format, and unsupported type changes.
• PySpark integration tests in TestDDLAlterTableColumns.
• TestSparkDirectoryNamespace full suite passes locally (66/66); make lint clean.

Docs updated in docs/src/operations/ddl/alter-table.md.

🤖 Generated with Claude Code

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The schema-evolution direction is useful, but this revision does not yet preserve Spark's alterTable contract or the connector's supported table formats. The four independent findings share that boundary: requested changes and options must either take effect exactly or fail before any mutation.

A viable revision should validate the complete ordered request against the current schema, commit it through one atomic core operation (or reject multi-change requests until that exists), and explicitly handle unsupported defaults, options, and legacy-format additions before writing.

[lance-gatekeeper]

apply commits each change during iteration, so a later rejection leaves earlier mutations durable. Spark's TableCatalog.alterTable contract says that when any change is rejected, none should be applied.

Reproducer run on this head

assertThrows(
    UnsupportedOperationException.class,
    () ->
        catalog.alterTable(
            ident,
            TableChange.addColumn(new String[] {"added"}, DataTypes.IntegerType),
            TableChange.updateColumnType(new String[] {"id"}, DataTypes.LongType)));
assertFalse(
    Arrays.asList(catalog.loadTable(ident).schema().fieldNames()).contains("added"));

The exception assertion passed, but the final assertion failed because added was still present after reload.

Validate the full ordered request first, then commit it atomically. If the core cannot batch these changes, reject a multi-change request before the first write.

[lance-gatekeeper]

AddColumn.defaultValue() is never read, so this reports success while existing rows receive NULL instead of the requested backfill.

Reproducer run on this head

ColumnDefaultValue defaultValue =
    new ColumnDefaultValue(
        "7", new LiteralValue<>(7, DataTypes.IntegerType));
catalog.alterTable(
    ident,
    TableChange.addColumn(
        new String[] {"with_default"},
        DataTypes.IntegerType,
        true,
        null,
        null,
        defaultValue));
Row row = spark.table(fullName).select("with_default").head();
assertFalse(row.isNullAt(0));
assertEquals(7, row.getInt(0));

The first assertion failed: the observed value was NULL.

Honor the default through an appropriate Lance operation, or reject non-null defaults before any mutation instead of silently discarding them.

[lance-gatekeeper]

This ignores DeleteColumn.ifExists(), so DROP COLUMN IF EXISTS still fails for a missing column.

Reproducer run on this head

assertDoesNotThrow(
    () ->
        catalog.alterTable(
            ident,
            TableChange.deleteColumn(new String[] {"missing"}, true)));

The assertion failed because dropColumns threw for missing.

Resolve the current schema before mutation and skip a missing field when ifExists is true; retain the error when it is false.

[lance-gatekeeper]

This all-null addColumns path rejects datasets created with the connector's supported file_format_version='LEGACY', so the newly documented DDL does not work across supported table formats.

Reproducer run on this head

spark.sql(
    "CREATE TABLE " + fullName +
    " (id INT) TBLPROPERTIES ('file_format_version'='LEGACY')");
assertDoesNotThrow(
    () -> spark.sql("ALTER TABLE " + fullName + " ADD COLUMN added INT"));

The assertion failed because Dataset.addColumns rejected the legacy dataset.

Use a legacy-capable schema-evolution path, or explicitly scope and document the operation to stable-format tables and reject legacy tables before calling the unsupported core API.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new prevalidation resolves the earlier default, IF EXISTS, and legacy-format findings, but all-or-none behavior is still not preserved because validated changes are committed as separate Lance mutations.

A viable revision must compile each request into one proven-atomic core mutation. Until heterogeneous batching exists, reject multi-mutation requests—including schema-plus-property requests—before the first write.

[lance-gatekeeper]

This loop still commits each accepted change through a separate native mutation, so prevalidation cannot prevent partial durable state when a later change fails for a data-dependent reason.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName + " (id BIGINT, name STRING)");
spark.sql("INSERT INTO " + fullName + " VALUES (NULL, 'Alice')");

Identifier ident = Identifier.of(new String[] {"default"}, tableName);
assertThrows(
    Exception.class,
    () ->
        catalog.alterTable(
            ident,
            TableChange.addColumn(new String[] {"added"}, DataTypes.IntegerType),
            TableChange.updateColumnNullability(new String[] {"id"}, false)));
assertFalse(
    Arrays.asList(catalog.loadTable(ident).schema().fieldNames()).contains("added"));

The nullability change was rejected because id contains NULL, but the final assertion failed (added remained present).

Apply the ordered request through one atomic core transaction. If the core cannot do that yet, reject any invocation requiring multiple native mutations—and mixed column/property requests in the catalog layer—before the first write.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The single-core-mutation direction fixes partial commits, but the current compiler still accepts ordered and nested requests that it does not translate faithfully. These independent failures share one contract: every accepted TableChange sequence must have the same ordered result Spark specifies.

A viable revision should canonicalize dependent renames onto the original core target, preserve or prevalidate intermediate changes that a core builder would collapse, and reject nested paths before mutation unless full-path semantics are implemented.

[lance-gatekeeper]

Keying builders by each change's literal path breaks dependent changes after a rename: validation treats names as evolving, but the compiled core batch targets the initial schema. The request is accepted and then fails instead of producing the requested schema.

Canonicalize post-rename paths back to the original source builder, or reject rename-dependent sequences during prevalidation when the core batch cannot represent them.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName
    + " (id BIGINT NOT NULL, name STRING NOT NULL)");
catalog.alterTable(
    ident,
    TableChange.renameColumn(new String[] {"name"}, "full_name"),
    TableChange.updateColumnNullability(
        new String[] {"full_name"}, true));

assertArrayEquals(
    new String[] {"id", "full_name"},
    catalog.loadTable(ident).schema().fieldNames());

The call threw IllegalArgumentException: Column "full_name" does not exist from Dataset.alterColumns.

[lance-gatekeeper]

Repeated nullability edits on one path are reduced to the last builder value, so required intermediate validation is silently skipped. Spark's TableCatalog.alterTable contract requires changes to be applied in order.

Reject non-equivalent repeated transitions before mutation, or perform every data-dependent intermediate validation before coalescing them into one core operation.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName
    + " (id BIGINT NOT NULL, name STRING)");
spark.sql("INSERT INTO " + fullName + " VALUES (1, NULL)");

assertThrows(
    RuntimeException.class,
    () -> catalog.alterTable(
        ident,
        TableChange.updateColumnNullability(
            new String[] {"name"}, false)));

assertThrows(
    RuntimeException.class,
    () -> catalog.alterTable(
        ident,
        TableChange.updateColumnNullability(
            new String[] {"name"}, false),
        TableChange.updateColumnNullability(
            new String[] {"name"}, true)));

The single SET NOT NULL assertion passed because existing data contains NULL; the ordered two-change assertion failed because no exception was thrown and the final true overwrote the required first transition.

[lance-gatekeeper]

This existence check truncates every field path to its top-level name. For a nested IF EXISTS request, s.missing is treated as present whenever s exists, then sent to the core and rejected. That violates the DeleteColumn.ifExists no-error contract and the documented top-level-only validation boundary.

Reject every multi-part DROP/RENAME/nullability path before mutation, or implement full-path existence tracking rather than using only element zero.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName
    + " (id BIGINT NOT NULL, s STRUCT<x: INT>)");

catalog.alterTable(
    ident,
    TableChange.deleteColumn(
        new String[] {"s", "missing"}, true));

assertArrayEquals(
    new String[] {"id", "s"},
    catalog.loadTable(ident).schema().fieldNames());

The no-op request threw IllegalArgumentException: Column s.missing does not exist from Dataset.dropColumns.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

This revision resolves the prior dependent-target, repeated-target, and nested-path findings, but the compiler still does not preserve exact column identities and ordered rename semantics. Both remaining failures share the same boundary: accepted changes must be equivalent to Spark's ordered request while Lance source paths must identify top-level fields unambiguously.

A viable revision should separately simulate rename destination occupancy in request order while retaining the current-schema representability checks, and restore canonical Lance paths for every DROP/RENAME/nullability source.

[lance-gatekeeper]

This validates each rename source against the current schema but never checks destination occupancy in request order. Because the core batch has no entry ordering, an invalid first rename can be accepted as a simultaneous rename, violating Spark's ordered alterTable contract.

Keep the current-schema checks needed by the core, but also simulate source removal and destination insertion in request order; reject a rename when its destination is occupied at that step.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName + " (a INT, b INT)");

assertThrows(
    RuntimeException.class,
    () -> catalog.alterTable(
        ident,
        TableChange.renameColumn(new String[] {"a"}, "b"),
        TableChange.renameColumn(new String[] {"b"}, "c")));

assertArrayEquals(
    new String[] {"a", "b"},
    catalog.loadTable(ident).schema().fieldNames());

The assertThrows assertion failed because the call returned successfully. In ordered execution, a -> b must fail before b -> c because b already exists.

[lance-gatekeeper]

This passes a column name as an unescaped Lance field path. A legal top-level name containing a backtick validates successfully, but the core path parser cannot resolve the raw name; the same issue applies to the raw source paths created at lines 216 and 220.

Use FieldPathUtils.canonicalPath for DROP and ColumnAlteration source paths so special characters are quoted and embedded backticks are escaped.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName + " (`a``b` INT, keep INT)");

catalog.alterTable(
    ident,
    TableChange.deleteColumn(new String[] {"a`b"}, false));

assertArrayEquals(
    new String[] {"keep"},
    catalog.loadTable(ident).schema().fieldNames());

The dataset was created successfully, but the drop threw IllegalArgumentException: Column ab does not exist`.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The ordered-rename fix resolves that part of the prior review, but this revision still does not preserve exact top-level column identities when a name contains Lance path syntax.

A viable revision should canonicalize every DROP, RENAME, and nullability source path before passing it to Lance.

[lance-gatekeeper]

Passing name verbatim to a Lance field-path API breaks legal top-level columns whose names contain path syntax. The request passes Spark-side validation but an accepted DROP fails; the same raw-source defect remains in the ColumnAlteration builders at lines 238 and 242.

Use FieldPathUtils.canonicalPath for the DROP and all ColumnAlteration source paths so special characters are quoted and embedded backticks are escaped.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName + " (`a``b` INT, keep INT)");

catalog.alterTable(
    ident,
    TableChange.deleteColumn(new String[] {"a`b"}, false));

assertArrayEquals(
    new String[] {"keep"},
    catalog.loadTable(ident).schema().fieldNames());

Expected the table schema to contain only keep; the DROP instead threw IllegalArgumentException: Column ab does not exist`.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new canonicalization fixes RENAME and nullability source paths, but DROP still does not preserve exact top-level column identities because the core drop API rejects embedded backticks under both canonical and raw representations.

A viable revision should use a DROP representation proven against embedded backticks, or reject those requests before mutation until the core supports them.

[lance-gatekeeper]

Canonicalizing the DROP name still leaves a legal Spark column undroppable. For a field named a`b, this passes the canonical string shown below, but Dataset.dropColumns treats it as an absent literal field rather than resolving the actual name. RENAME and nullability passed with the same special name, so this is specific to the DROP API boundary.

field name:     a`b
DROP argument:  `a``b`

If this core version has no exact representation for such names (the raw name fails too), detect and reject these DROP requests before dropColumns; otherwise use a proven DROP-specific representation. Keep canonical paths for ColumnAlteration.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName + " (`a``b` INT, keep INT)");

catalog.alterTable(
    ident,
    TableChange.deleteColumn(new String[] {"a`b"}, false));

assertArrayEquals(
    new String[] {"keep"},
    catalog.loadTable(ident).schema().fieldNames());

Expected the schema to contain only keep; instead dropColumns threw RuntimeException because the canonical argument was not found while the available field was a`b.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new core-capability guard fixes the existing-field failure, but it changes DROP COLUMN IF EXISTS into an error when a missing requested name contains a backtick.

A viable revision should resolve absence first and apply the unsupported-name rejection only to an existing field that would reach dropColumns.

[lance-gatekeeper]

This guard runs before the missing/ifExists branch, so a missing top-level name containing a backtick throws instead of being ignored. IF EXISTS is an existence contract and must not depend on whether an absent spelling would be representable by the core.

Check absence first: allow a missing ifExists change to reach the existing skip path, preserve the missing-column error when it is false, and reject the backtick only when the field actually exists.

Reproducer run on this head

spark.sql("CREATE TABLE " + fullName + " (id INT)");

catalog.alterTable(
    ident,
    TableChange.deleteColumn(new String[] {"missing`name"}, true));

assertArrayEquals(
    new String[] {"id"},
    catalog.loadTable(ident).schema().fieldNames());

Expected a no-op; the call instead threw UnsupportedOperationException: Dropping a column whose name contains a backtick is not supported by the current Lance version: missingname`.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The missing-name ordering fix now preserves DROP COLUMN IF EXISTS while rejecting only existing backtick-named fields that the current core cannot drop. The complete change keeps accepted schema evolution atomic and faithful to Spark’s ordered request semantics.