Apache Gluten write support in lance-spark

[malinjawi]

Summary

This discussion proposes one small public API in lance-spark to support Apache Gluten's columnar write path: a factory that wraps executor-produced Lance FragmentMetadata in the commit-message type expected by LanceBatchWrite.

The intended write flow was discussed in Apache Gluten issue #12263 and narrowed in the maintainer write-path analysis:

1. A Gluten executor exports its columnar output as an Arrow C stream.
2. The existing Lance Java API writes that stream as one or more uncommitted Lance fragments.
3. The executor returns the fragment metadata to Spark as a WriterCommitMessage.
4. The existing LanceBatchWrite.commit(...) performs the driver-side Lance transaction.

Steps 1 and 2 do not require a lance-spark change. Step 4 already exists. The proposed factory supplies the missing bridge in step 3.

I am opening a Discussion before a PR because this adds a public API, following the lance-spark contribution guidelines.

Current limitation

LanceDataWriter currently creates the expected commit message internally:

return new LanceBatchWrite.TaskCommit(fragments);

LanceBatchWrite.commit(...) later consumes those messages and commits their fragments. However, the TaskCommit constructor is package-private, so a columnar writer outside org.lance.spark.write cannot create a compatible message from its List<FragmentMetadata>.

The driver-side commit should remain in LanceBatchWrite. Reusing it avoids duplicating append and overwrite operations, transaction construction, optimistic-concurrency handling, namespace integration, storage options, stable-row-ID handling, and staged-commit behavior in Gluten.

Proposed API

Add a public static factory on LanceBatchWrite:

public static WriterCommitMessage taskCommit(
    List<FragmentMetadata> fragments);

Example use from an external columnar writer:

List<FragmentMetadata> fragments =
    Fragment.write()
        .datasetUri(datasetUri)
        .data(arrowArrayStream)
        .writeParams(writeParams)
        .execute();

WriterCommitMessage message =
    LanceBatchWrite.taskCommit(fragments);

Returning WriterCommitMessage keeps TaskCommit and its fragment accessor as implementation details. The caller only needs to create the message and return it through Spark's existing writer-commit protocol; LanceBatchWrite remains responsible for reading and committing it.

If maintainers prefer callers to depend on the concrete type, the factory could instead return TaskCommit. We do not currently see a need to make the constructor or getFragments() public.

Scope

This proposal changes only lance-spark. It does not change fragment-writing behavior or introduce another writer implementation.

The initial Gluten integration is expected to enable the columnar path only for existing-table batch writes where it can preserve the behavior of the current Lance writer. Its first scope is:

• append and schema-preserving overwrite;
• unsharded tables; and
• tables that do not require blob-reference transformations.

Gluten will fall back to the existing Lance Spark row writer when those conditions are not met. These restrictions belong to the Gluten integration and do not need to be encoded in the commit-message factory.

This proposal does not include:

• changes to the Lance Java API or file format;
• serializing or exposing LanceSchema;
• making SparkWrite.getWriteOptions() public;
• a public fragment accessor on TaskCommit;
• CTAS, RTAS, streaming, sharded, blob, or row-level write offload; or
• the separate read-path Arrow C stream integration.

As confirmed in the Gluten follow-up, WriteFragmentBuilder.schema(...) may later avoid an executor-side dataset open, but it is an optional performance optimization and is not required for this writer to be correct.

Compatibility and ownership

The factory is additive. Existing LanceDataWriter behavior and the BatchWrite contract do not change.

The ownership boundary remains the same as the current writer:

• executors create uncommitted fragments and return commit messages;
• Spark transports those messages to the driver; and
• LanceBatchWrite owns the transaction and final commit.

The API does not expose Arrow Java vectors across the Gluten/lance-spark boundary. Fragment data continues to cross through the Arrow C interface used by the existing Lance API.

Testing

The implementation PR should include focused tests that verify:

• code outside org.lance.spark.write can create a commit message through the factory;
• the returned message is accepted by LanceBatchWrite.commit(...);
• messages containing multiple fragments are aggregated and committed correctly; and
• existing LanceDataWriter commit behavior remains unchanged.

The Gluten-side integration will separately test its supported write modes and deterministic fallback for unsupported tables.

Delivery

After agreement, this can be delivered independently in two repositories:

1. lance-spark: add the public factory and focused tests.
2. gluten: add the columnar Lance writer and its validation/fallback rules.

The lance-spark change does not depend on the read-path work or on a new Lance Java API.

Questions for maintainers

1. Is LanceBatchWrite.taskCommit(List<FragmentMetadata>) the preferred public factory shape?
2. Should its return type remain the opaque WriterCommitMessage, or should it return TaskCommit?

[LuciferYang]

also cc @hamersaw

[LuciferYang]

Thanks for the well-scoped proposal — I read it against the current write path on main and it matches the code. A public factory on LanceBatchWrite is the right seam: steps 1/2/4 already exist, and the only missing link is the commit-message bridge, since TaskCommit's constructor is package-private and a writer in another package can't build one.

Q1 — factory shape. taskCommit(List<FragmentMetadata>) works and reads symmetrically with commit(...). I'd make it static (it needs no instance state) and have it defensively copy the list, matching what LanceDataWriter already does internally (new ArrayList<>(...)), plus reject a null list so misuse fails at the call site.

Q2 — return type. Return the opaque WriterCommitMessage and build a TaskCommit inside. That keeps TaskCommit an implementation detail (consistent with your "no public fragment accessor" out-of-scope item), and commit(...) casts back to TaskCommit, so the runtime type is all that matters. The hop is serialization-safe: WriterCommitMessage is Serializable, TaskCommit inherits it, and FragmentMetadata is too, which is what lets the message travel back to the driver as the task commit result.

Two small things for the implementation PR:

• commit(...) already normalizes row-id metadata (it maps every message through stripRowIdMeta), so the Gluten side can pass its Fragment.write() output straight through without touching row-id metadata itself.
• Consider marking the new method with Spark's @Evolving plus a short javadoc noting the message must be committed through a LanceBatchWrite for the same target dataset. There's no stability-annotation precedent in the repo yet, so it's just a suggestion.

Splitting delivery across the two repos (factory + tests here, columnar writer + fallback in gluten) makes sense. cc @hamersaw for the API-shape call.

[hamersaw]

Doing a quick scan over this, I don't have any concerns. It sounds like @LuciferYang has been iterating on the discussion gluten-side for a bit (thanks!). I'm excited to review the PR.