[Feature](lance) Add vector and scalar index lifecycle management

[Gabriel39]

Parent

Tracked by #66340. #65730 provides read-only index consumption and automatic vector-index selection.

Scope

• Create vector indexes.
• Create scalar indexes.
• Show the number of indexes.
• Show index names, UUIDs, columns, types, and dataset versions.
• Replace or rebuild an index with the same name.
• Drop indexes by name.
• Define authorization, concurrency, versioning, failure recovery, and metadata refresh semantics.

Completion criteria

• FE statements and metadata APIs are implemented.
• BE or service-side index operations are integrated with Lance.
• Unit and regression tests cover create, inspect, replace, rebuild, drop, and failures.
• User documentation describes syntax, lifecycle semantics, and limitations.
• The behavior is verified for vector and scalar index types.

[Gabriel39]

@u70b3, would you be available to take ownership of this issue and implement it?

[u70b3]

Thanks @Gabriel39 — I'd be glad to take ownership of this issue. Please assign it to me.

[u70b3]

Superseded: This proposal has been replaced by Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle. This version is retained for review history.

Here is a revised design proposal for this issue. It builds on #65730, the Lance Java SDK / lance-c versions pinned by Doris, and the existing Doris pattern for mutable external lake-table catalogs (Iceberg/Hive/Paimon): Doris parses a common SQL surface, dispatches through a catalog-specific operation seam, lets the external format remain the source of truth, and refreshes Doris metadata after a successful external commit.

/cc @Gabriel39 @zhangstar333

1. Goals and Phase-1 boundary

Phase 1 provides index lifecycle management for writable filesystem-backed Lance catalogs:

• create vector and scalar indexes;
• inspect logical indexes, physical segment metadata, coverage, and the logical index count;
• replace/rebuild an index by name;
• drop an index by name;
• define privilege, concurrency, version, failure, and metadata-refresh behavior.

Lance dataset manifests remain the authoritative index metadata. Doris does not persist a second copy of Lance index definitions in its internal table/index metadata.

Phase 1 uses the Lance Java SDK in FE and is intentionally synchronous. A BE-distributed or asynchronous build path is a follow-up and should not change the SQL contract.

2. SQL surface

Reuse the existing Doris index statements and command chain, extended for Lance external tables:

-- Vector index
CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (vec_col)
    USING IVF_PQ
    PROPERTIES (
        "metric" = "l2",
        "num_partitions" = "256",
        "num_sub_vectors" = "16"
    );

-- Scalar indexes
CREATE INDEX idx ON lance_ctl.db.tbl (col) USING BTREE;
-- BITMAP / INVERTED are also candidates for Phase 1.

-- Full rebuild and atomic replacement by logical name
CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (vec_col)
    USING IVF_PQ
    PROPERTIES (...);

SHOW INDEX FROM lance_ctl.db.tbl;

DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;

Proposed semantics:

• USING is required for Lance tables; Doris's internal default-to-INVERTED behavior should not be applied implicitly to an external format.
• IF NOT EXISTS and OR REPLACE are mutually exclusive.
• CREATE INDEX fails when the logical name already exists; IF NOT EXISTS returns success without rebuilding; CREATE OR REPLACE INDEX performs a full rebuild and maps to Lance replace=true.
• Incremental indexing of newly appended fragments (for example via Lance index optimization APIs) is not part of Phase 1.
• Phase 1 supports one indexed column per statement. Multi-column/composite indexes can be added later without changing the lifecycle model.
• Index names are preserved and matched exactly, following the pinned Lance implementation. Doris should not silently lowercase external index names or merge case-only names.
• Only index types verified end-to-end by both index creation and the Doris query path are accepted. Unsupported types and unknown properties fail during analysis instead of being accepted syntactically and rejected later.
• The exact per-type property/type/metric matrix and SDK mapping will be documented and tested in the PR. The implementation should use a Lance-specific type/definition model rather than adding Lance physical types to Doris's persisted internal catalog.IndexType.

3. SHOW INDEX and the Lance index model

Lance has two levels of index metadata:

• a logical index, identified by name, columns, type, and aggregate row/fragment coverage;
• one or more physical index segments, each with its own UUID, dataset version, and fragment set.

A distributed or incrementally maintained logical index may therefore have multiple UUIDs and dataset versions. The logical count must not be derived from the number of physical manifest entries.

The proposal is to keep Doris's existing MySQL-compatible 13-column SHOW INDEX schema instead of returning a different column layout only for Lance tables:

• Key_name: logical index name;
• Column_name: indexed column;
• Index_type: logical Lance index type;
• Properties: aggregate coverage, index-specific details, and a JSON list of physical segments (uuid, dataset_version, and fragment coverage).

For example:

{
  "indexed_fragments": 8,
  "unindexed_fragments": 2,
  "indexed_rows": 100000,
  "unindexed_rows": 24000,
  "segments": [
    {
      "uuid": "...",
      "dataset_version": 12,
      "fragments": [0, 1, 2, 3]
    }
  ]
}

The index count is the number of distinct user-visible logical indexes, excluding Lance system indexes. It is obtained from logical index descriptions, not from getIndexes() / lance-c physical entry counts. No additional count statement is proposed.

If reviewers prefer UUIDs and dataset versions as first-class result columns, a dedicated Lance system table or statement is safer than changing the established SHOW INDEX result schema.

4. Execution model

The Phase-1 flow follows Doris external-catalog DDL precedent:

Doris SQL / privilege validation
    -> Lance-specific index operator
    -> Lance Java SDK (JNI)
    -> Lance manifest commit
    -> Doris external-table refresh/edit-log path

The pinned Lance Java SDK provides create, drop, logical description, physical metadata, and distributed segment/commit primitives. Therefore Phase 1 does not require a new FE-to-BE thrift/RPC protocol.

There is an important difference from lightweight Iceberg/Paimon metadata DDL: Lance index creation scans column data and performs native CPU, memory, and object-store I/O work. In Phase 1 this work runs in Lance native threads inside FE. The synchronous implementation should therefore be documented with the following limitations:

• build latency scales with the dataset and index parameters;
• the pinned Java API does not expose a reliable cancellation token or Java progress callback for createIndex;
• client disconnect or KILL QUERY does not guarantee that the native build has stopped;
• FE failover can make the client-visible completion result ambiguous.

The operator boundary keeps a future implementation free to move segment building to BE workers or an external service while FE retains orchestration and the final commit. Lance exposes the required distributed model, but Lance itself does not provide the scheduler; the current lance-c v0.1.2 surface should not be assumed to contain the same distributed APIs as the Java SDK.

5. Authorization

• CREATE INDEX, CREATE OR REPLACE INDEX, and DROP INDEX require the table-level ALTER privilege.
• SHOW INDEX requires the table-level SHOW privilege.

This matches Doris internal index DDL and existing external-table ALTER behavior. The remote filesystem/object-store access still uses the catalog credential; Doris table privilege is the user-facing authorization boundary.

6. Lifecycle semantics

Versioning and visibility

• A successful create/replace/drop is committed by Lance as a new dataset version with an atomic manifest switch.
• Queries already pinned to an older dataset version continue to see that version.
• A failed build before manifest commit does not make the new index visible. It may leave unreferenced physical files under the Lance index directory; later Lance cleanup/VACUUM reclaims them according to retention policy.
• A catalog/table pinned to a historical dataset version is read-only for index mutation. SHOW INDEX reads the pinned snapshot; mutation statements fail instead of silently switching to latest.
• For an ordinary writable table, mutation and live inspection target the latest dataset version.

Concurrency

• Concurrent append is compatible with an index build; appended fragments may remain uncovered and are reported as unindexed.
• Overwrite, restore, data replacement, or fragment rewrite may conflict according to Lance transaction rules.
• Although current Lance format documentation describes compatible same-name commits, the pinned v9.1.0-beta.3 implementation tests same-name concurrent create/replace as a retryable commit conflict. Phase 1 should surface that failure and ask the user to retry instead of promising last-committer-wins.
• The pinned JNI mapping does not expose a distinct Java exception class for commit conflict, so Phase 1 should not implement automatic retry by parsing exception text.
• Doris metadata locks should not be held for the duration of native index training; Lance optimistic concurrency controls the external commit.

Failure recovery and retries

• FE crash or client timeout before the response is ambiguous: the Lance commit may already have succeeded. SHOW INDEX is the reconciliation mechanism.
• IF NOT EXISTS provides no-op retry behavior once the logical index is visible.
• CREATE OR REPLACE INDEX is convergent by name but is not strictly idempotent: retrying may rebuild work and produce a new physical UUID/dataset version.
• DROP INDEX on a missing name fails unless IF EXISTS is specified.

Drop and metadata refresh

• Drop removes the logical index from a new manifest version. Physical files are reclaimed later according to Lance cleanup retention, preserving access through retained historical versions.
• After a successful mutation, Doris follows the existing external DDL bookkeeping path: emit the refresh edit log and invalidate/refresh the table metadata on FE nodes.
• Failed mutations do not run the post-DDL refresh path.
• SHOW INDEX reads live index metadata from the target Lance snapshot rather than persisting index information in the Doris catalog cache.

7. Initial supported types and catalogs

Proposed initial set:

• Vector: IVF_PQ. IVF_FLAT is included only if the PR verifies that the [feature](lance) support lance catalog in doris #65730 Doris query path actually consumes it end-to-end.
• Scalar: BTREE, BITMAP, and INVERTED, each gated by schema/property validation and an end-to-end query test.
• Other Lance vector/scalar types are rejected until their create and query paths are verified.

Catalog scope:

• Filesystem-backed Lance datasets: local/file and supported object stores such as S3.
• REST Namespace create/replace/drop: rejected with a clear unsupported error in Phase 1. REST index mutation is a separate asynchronous transaction/capability model and does not have the same request surface as direct Dataset.createIndex.
• Read-only REST SHOW INDEX can be supported through namespace list/stats capabilities if reviewers want it in Phase 1; otherwise all REST lifecycle statements are rejected consistently and documented.

8. Implementation and test orientation

Implementation details will be reviewed in the PR. The intended direction is:

• reuse AlterTableCommand + create/drop index ops and the existing ShowIndexCommand;
• add explicit Lance external-table gates and catalog-specific dispatch;
• place SDK calls and Lance-specific validation/mapping behind a LanceIndexOperator-style seam;
• keep Lance logical/physical index DTOs separate from Doris internal index serialization;
• run post-DDL refresh only after a successful Lance commit.

Tests should cover parser/to-SQL behavior, privileges, supported/unsupported types and properties, duplicate/replace/drop behavior, logical versus physical metadata, append coverage, concurrency conflicts, historical-version mutation rejection, REST behavior, and vector/scalar query use. Regression tests should use an isolated writable Lance dataset rather than mutating a shared read-only fixture.

Questions for reviewers

• Q1: Is a filesystem-only, FE-side synchronous Java SDK implementation acceptable for Phase 1 with the resource, cancellation, and failover limitations above, while keeping distributed BE execution as a follow-up?
• Q2: Is the explicit CREATE OR REPLACE INDEX syntax acceptable? The alternative PROPERTIES("replace"="true") avoids grammar work but hides lifecycle semantics.
• Q3: Should Lance reuse the existing 13-column SHOW INDEX schema with physical segment metadata in Properties, or should this issue introduce a dedicated Lance index result surface?
• Q4: Is IVF_PQ + BTREE/BITMAP/INVERTED the right initial end-to-end set, with IVF_FLAT added only after query-side verification?
• Q5: For REST Namespace in Phase 1, should read-only SHOW INDEX be supported through list/stats while mutation remains unsupported, or should all index lifecycle statements be rejected consistently?

References

• Doris external-catalog mutation precedent: Iceberg/Paimon catalog-specific metadata operations and the external DDL refresh/edit-log path
• Doris internal index DDL chain: AlterTableCommand, create/drop index ops, and ShowIndexCommand
• [feature](lance) support lance catalog in doris #65730 and tracking issue [Tracking] Lance feature matrix for Doris 4.2 #66340
• Lance transaction semantics
• Lance index logical/physical model
• Lance distributed indexing
• Lance Spark SHOW INDEXES logical result model
• Lance Namespace index operations
• Pinned Java SDK sources: Dataset, IndexOptions, and IndexDescription

[Gabriel39]

Thanks for the detailed proposal. The source-of-truth model, logical/physical index distinction, version semantics, and post-commit metadata refresh direction are sound. After comparing the proposal with the current Doris command path, the pinned Lance SDK/Namespace API, and the Lance Spark, StarRocks, ClickHouse, DuckDB, and Trino designs, I think the following points should be resolved before implementation.

Blocking concerns

1. A synchronous native build in FE is not a safe production execution model

The proposal already lists the essential problems: index creation performs data scans and native CPU/memory/object-store I/O in FE, while the pinned API has no reliable cancellation/progress mechanism and FE failover can leave the result ambiguous. This also bypasses Doris workload management and allows a user with ALTER to trigger unbounded native work in the control plane.

The closest Lance integration distributes fragment builds to Spark executors and leaves the driver to coordinate/finalize the commit. ClickHouse similarly separates index metadata operations from the heavy MATERIALIZE INDEX mutation.

• https://lance.org/integrations/spark/operations/ddl/create-index/
• https://clickhouse.com/docs/reference/statements/alter/skipping-index

My recommendation is an asynchronous job with FE orchestration and BE or external-service execution, including status, cancellation, concurrency limits, and retry/reconciliation. If FE-side execution must exist temporarily, it should be explicitly experimental, disabled by default, and protected by dataset-size/fragment/concurrency limits. It should not be presented as the final production lifecycle implementation.

2. Reusing the current Doris index command chain requires a larger refactor than the proposal describes

The current parser only accepts NGRAM_BF, INVERTED, and ANN; it does not accept IVF_PQ, BTREE, BITMAP, or OR REPLACE:

doris/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4

Lines 237 to 240 in e3289c1

	| CREATE INDEX (IF NOT EXISTS)? name=identifier
	ON tableName=multipartIdentifier identifierList
	(USING (NGRAM_BF | INVERTED | ANN))?
	properties=propertyClause? (COMMENT STRING_LITERAL)? #createIndex

More importantly, CreateIndexOp.validate() always translates the definition into the persisted internal catalog.Index, while AlterTableCommand currently rejects CreateIndexOp for external tables:

doris/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateIndexOp.java

Lines 73 to 89 in e3289c1

	@Override
	public void validate(ConnectContext ctx) throws UserException {
	if (indexDef == null) {
	throw new AnalysisException("index definition expected.");
	}
	if (tableName != null) {
	tableName.analyze(ctx);
	}
	indexDef.validate();
	index = indexDef.translateToCatalogStyle();
	}
	@Override
	public AlterTableClause translateToLegacyAlterClause() {
	return new CreateIndexClause(tableName != null ? tableName.transferToTableName() : null,
	indexDef.translateToLegacyIndexDef(), index, alter);

That conflicts with the stated goal of keeping Lance index DTOs separate from Doris internal index serialization. The design should explicitly introduce a neutral parsed index definition, resolve the target table/catalog first, and then choose internal-index validation/serialization or Lance-specific validation/dispatch. Adding only a LanceIndexOperator after the existing validation path will not be sufficient.

For vector syntax, I would prefer reusing the Doris category-level surface:

CREATE INDEX idx ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    ...
);

This also follows the StarRocks pattern of USING VECTOR plus an index_type property and avoids adding a grammar keyword for every Lance algorithm.

Other required revisions

3. Incremental maintenance is necessary for a usable external index lifecycle

External writers can append fragments immediately after an index is created. Coverage will then decay. CREATE OR REPLACE INDEX performs an expensive full rebuild and is not an adequate maintenance operation.

The pinned Java SDK exposes Dataset.optimizeIndices(), and Lance documents it as the incremental path for indexing newly appended fragments. I suggest adding a separate operation such as:

BUILD INDEX idx ON lance_ctl.db.tbl;
-- or
ALTER TABLE lance_ctl.db.tbl OPTIMIZE INDEX idx;

The tests must also prove that queries include unindexed fragments correctly. Otherwise partial coverage becomes a correctness issue, not merely a performance issue.

4. Do not put unbounded physical segment metadata in SHOW INDEX.Properties

A JSON segments array can become extremely large for a fragmented or distributed index. It can create FE memory, MySQL result packet, and client usability problems.

Lance Spark intentionally returns only logical summaries from SHOW INDEXES and excludes per-segment metadata:

https://lance.org/integrations/spark/operations/ddl/show-indexes/

I recommend keeping the existing 13-column SHOW INDEX surface for logical name, columns, type, and aggregate coverage only. UUIDs, dataset versions, and fragment sets should be exposed through a structured and filterable system table/TVF such as lance_index_segments, not embedded in one string field.

5. REST behavior should be capability-based, not rejected by catalog type

The pinned lance-namespace-core 0.7.7 interface already defines createTableIndex, createTableScalarIndex, listTableIndices, describeTableIndexStats, and dropTableIndex:

https://github.com/lance-format/lance-namespace/blob/v0.7.7/java/lance-namespace-core/src/main/java/org/lance/namespace/LanceNamespace.java#L394-L447

For a Directory Catalog, direct Dataset operations are appropriate. For a REST Catalog, index operations should go through the Namespace service when the server supports them; Doris should return a clear unsupported error when it does not. Directly mutating the dataset with vended credentials would bypass service-side authorization, managed versioning, and transaction policy. At minimum, REST SHOW INDEX should use list/stats operations in Phase 1.

6. Narrow and rename the initial type set

I suggest Phase 1 support:

• vector: IVF_PQ;
• scalar: BTREE and BITMAP;
• IVF_FLAT only after the Doris query path is verified end to end;
• no INVERTED until the Lance FTS/BM25 query path is implemented.

Lance INVERTED is an FTS index with different tokenizer/query semantics from a Doris internal inverted index. Exposing the same keyword before Doris can query it is misleading. A future SQL surface should call it FTS.

The complete type/property matrix should be part of the design, not deferred entirely to the PR. It should include column types, vector element type/dimension/nullability, metric compatibility, parameter ranges, and build/query ABI compatibility.

7. Define SQL identifier semantics for index names

Allowing both idx and IDX while matching names exactly makes DROP INDEX idx ambiguous under Doris/MySQL identifier behavior. We should preserve display case but define quoted/unquoted resolution and reject new case-only duplicates, or provide an unambiguous quoted-name rule for pre-existing external collisions.

8. Refresh/recovery must handle post-commit failures

“Failed mutations do not refresh metadata” is insufficient when the Lance commit succeeds but FE fails before the edit log or client response. A later IF NOT EXISTS retry may be a no-op and still leave Doris caches stale.

The implementation should distinguish pre-commit failure from unknown/post-commit outcome, reopen the latest snapshot to verify the postcondition, and invalidate/refresh Doris metadata even for a reconciled no-op. SHOW INDEX helps the user observe state but does not repair the query path cached metadata.

Also, local/file catalogs need an explicit shared-filesystem or single-node restriction; otherwise a master failover can move execution to an FE that cannot see the same path.

Answers to the review questions

• Q1: No for the production implementation. FE-side synchronous execution is acceptable only as a tightly gated experimental fallback.
• Q2: CREATE OR REPLACE INDEX is reasonable if replacement is atomic, but incremental BUILD/OPTIMIZE INDEX is more important. For vector indexes, prefer USING ANN plus index_type.
• Q3: Keep SHOW INDEX as a logical summary; expose physical segments through a system table/TVF.
• Q4: Start with IVF_PQ + BTREE + BITMAP; add IVF_FLAT only after end-to-end verification, and exclude INVERTED until FTS querying is available.
• Q5: Support REST inspection through Namespace list/stats. Support REST mutation only through Namespace index operations and server capability detection, never by bypassing the metadata service.

The proposal demonstrates a good understanding of the Lance index model, but the execution placement and the split between Doris internal index objects and external Lance operations need to be redesigned before implementation starts.

[u70b3]

Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle

This revision supersedes the previous proposal and incorporates the review feedback from @Gabriel39.

Major changes from v1

• Heavy index creation, replacement, and incremental maintenance no longer run synchronously in FE. Production execution is a durable asynchronous job: FE orchestrates and finalizes commits, Directory catalogs use distributed BE segment construction, and REST catalogs use Namespace-managed execution.
• The SQL path introduces a neutral parsed index specification. Doris resolves the target table and catalog before choosing internal-index serialization or Lance-specific validation and dispatch.
• Vector syntax reuses the Doris category-level surface: USING ANN with index_type=IVF_PQ.
• BUILD INDEX is added to the Phase-1 lifecycle as the incremental operation for newly appended or otherwise uncovered fragments.
• SHOW INDEX returns only bounded logical summaries. A filterable physical-segment TVF exposes UUIDs, dataset versions, and aggregate segment coverage; explicit segment-to-fragment membership is deferred.
• REST behavior is capability-based. REST mutations go through Lance Namespace APIs and never bypass the metadata service with vended credentials.
• The initial type set is narrowed to IVF_PQ, BTREE, and BITMAP. Lance INVERTED/FTS and IVF_FLAT are deferred until the Doris query path is verified end to end.
• Index-name resolution, post-commit reconciliation, metadata refresh retries, and local-filesystem restrictions are defined explicitly.

1. Goals and Phase-1 boundary

Phase 1 provides a production-oriented lifecycle for user-visible Lance indexes:

• create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes;
• inspect logical index definitions, aggregate coverage, and logical index count;
• inspect physical index segments through a separate metadata surface;
• incrementally index newly appended or uncovered fragments;
• atomically replace/rebuild an index by logical name;
• drop an index by logical name;
• provide durable status, cancellation, concurrency limits, retry/reconciliation, and metadata refresh;
• support Directory Catalog operations and capability-based REST Namespace operations.

Lance dataset manifests or the Namespace service remain the authoritative index metadata. Doris does not persist a second copy of Lance index definitions in its internal table/index metadata. Doris persists only its own asynchronous job state and the information needed to reconcile an external operation.

Historical-version tables remain read-only for index mutation. Inspection uses the selected snapshot; mutation targets only the latest writable table state.

An FE-side synchronous native build is not part of the production implementation. If retained temporarily for development, it must be experimental, disabled by default, and protected by strict dataset-size, fragment-count, and concurrency limits. It does not satisfy Phase-1 completion.

2. SQL surface

Vector creation

CREATE INDEX [IF NOT EXISTS] idx
ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    "metric" = "l2",
    "num_partitions" = "256",
    "num_sub_vectors" = "16"
);

Scalar creation

CREATE INDEX idx
ON lance_ctl.db.tbl (scalar_col)
USING BTREE
PROPERTIES (
    "zone_size" = "4096"
);

CREATE INDEX idx_bitmap
ON lance_ctl.db.tbl (category_col)
USING BITMAP;

Full replacement and incremental maintenance

CREATE OR REPLACE INDEX idx
ON lance_ctl.db.tbl (vec_col)
USING ANN
PROPERTIES (
    "index_type" = "IVF_PQ",
    "metric" = "l2",
    "num_partitions" = "256",
    "num_sub_vectors" = "16"
);

BUILD INDEX idx ON lance_ctl.db.tbl;

CREATE OR REPLACE INDEX performs a full rebuild and atomically replaces the logical index. BUILD INDEX indexes only fragments not covered by the current logical index and maps to the Lance incremental optimization path.

Inspection, status, cancellation, and drop

SHOW INDEX FROM lance_ctl.db.tbl;

SHOW BUILD INDEX FROM db
WHERE TableName = "tbl";

CANCEL BUILD INDEX ON lance_ctl.db.tbl (job_id);

DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;

The existing Doris SHOW BUILD INDEX and CANCEL BUILD INDEX surfaces are extended to Lance external-index jobs instead of introducing a second job-control vocabulary.

Statement semantics

• USING is required for Lance tables.
• IF NOT EXISTS and OR REPLACE are mutually exclusive.
• Plain CREATE INDEX fails if the logical name exists.
• IF NOT EXISTS verifies the visible logical definition and returns a reconciled no-op when it already matches. A conflicting definition with the same name is an error rather than a silent no-op.
• CREATE OR REPLACE INDEX performs a full rebuild. It is convergent by name but not strictly idempotent: a retry can produce new segment UUIDs and a new dataset version.
• BUILD INDEX is incremental and does not change the logical definition.
• Phase 1 supports one indexed column per statement. Composite scalar indexes are deferred.
• Unknown index types, unknown properties, unsupported column types, and invalid property values fail during analysis.
• Create, replace, and build return after a durable job is accepted, not after native index construction completes.
• Drop is a metadata operation. It may complete synchronously when the selected provider exposes an atomic drop, but it still uses the same post-commit reconciliation and refresh rules.

3. Neutral command model and catalog dispatch

The current internal path must not translate every parsed definition into persisted catalog.Index before the target catalog is known.

The revised analysis flow is:

parsed SQL
    -> ParsedIndexSpec
    -> resolve catalog / database / table
    -> select InternalIndexProvider or LanceIndexProvider
    -> provider-specific validation
    -> internal schema-change operation or Lance external-index job

ParsedIndexSpec contains only SQL-level information:

• logical name;
• target columns;
• category from USING;
• properties;
• IF NOT EXISTS / OR REPLACE;
• operation kind: create, replace, incremental build, or drop.

For an internal OLAP table, the existing provider validates and translates the specification into Doris catalog.Index and the legacy schema-change path.

For a Lance external table, LanceIndexProvider produces a Lance-specific immutable definition and operation request. Lance types and physical metadata are never added to Doris internal IndexType serialization.

The provider boundary is also used by SHOW, BUILD, DROP, reconciliation, and capability checks so lifecycle behavior does not drift across separate command implementations.

4. Asynchronous execution model

Durable job

Create, replace, and incremental build create a persistent LanceIndexJob before heavy work starts. The minimum states are:

PENDING
  -> PREPARING
  -> BUILDING
  -> COMMITTING
  -> REFRESHING
  -> FINISHED

Terminal or recovery states:
FAILED
CANCELLING
CANCELLED
COMMITTED_REFRESH_PENDING
OUTCOME_UNKNOWN

The persisted job records:

• catalog, table identity, dataset URI or Namespace table ID;
• logical index name and normalized definition;
• operation kind and starting dataset version;
• old segment UUIDs for replace/reconciliation;
• provider and executor identity;
• worker task IDs or remote transaction ID;
• current state, progress, timestamps, and sanitized failure information.

FE failover reconstructs unfinished jobs from the edit log and resumes polling, reconciliation, commit, or refresh. It does not blindly restart native training.

Responsibility split

FE performs:

• SQL analysis and table-level privilege checks;
• durable job creation and state transitions;
• executor selection and concurrency admission;
• optimistic commit/finalization when the selected execution model separates build from commit;
• outcome reconciliation;
• metadata invalidation, refresh edit log, and retry of post-commit refresh.

For Directory Catalog, BE workers perform:

• fragment-scoped column scans;
• vector training;
• scalar/vector segment construction;
• native CPU, memory, and object-store I/O work;
• periodic progress reporting and cooperative cancellation.

For REST Namespace Catalog, the Namespace service owns execution and commit according to its advertised capabilities. Doris tracks the returned operation or transaction identifier and reconciles the result through the service; it does not substitute a direct Dataset mutation.

FE metadata locks are not held during native training. Doris serializes conflicting jobs for the same dataset and logical index name, while Lance transaction rules remain authoritative for conflicts with external writers.

Resource and cancellation behavior

• Jobs are admitted through explicit per-catalog and cluster-wide concurrency limits.
• BE execution is associated with workload-management accounting where the executor supports it.
• Cancellation is persisted before workers are signaled and immediately fences the job from entering COMMITTING.
• Every BE task carries the durable job ID and generation. FE accepts physical segment metadata only from the active generation.
• The lance-c extension used by the production path must expose cooperative cancellation and progress. The production Directory executor remains disabled until those APIs are available.
• A native task that finishes after cancellation may leave unreferenced physical files, but it cannot publish them because final commit is fenced in FE. The job becomes CANCELLED only after active tasks stop or their results are safely discarded.
• Native exception strings are not parsed to implement automatic conflict retry. Typed conflict results are used where the provider exposes them.
• Same-name concurrent create/replace conflicts are surfaced as retryable failures unless a later reconciliation proves that this job's requested postcondition was committed.

5. Provider and catalog behavior

Directory Catalog

For Directory Catalog, this proposal chooses Doris BE execution rather than introducing an external Java/Rust build service. Because BE is C++, this choice requires lance-c to expose the distributed Rust Lance APIs that already exist in core.
Directory-backed datasets use a distributed BE build pipeline. Phase 1 extends lance-c with the native capabilities needed by that pipeline:

• fragment-scoped uncommitted vector and scalar segment construction;
• serialization of physical segment metadata back to FE;
• progress and cooperative cancellation;
• merge support where the selected index type requires compatible segments.

FE selects disjoint fragment groups, dispatches BE tasks, validates the returned coverage, and performs the single atomic logical-index commit through the pinned Lance Java SDK. BE workers never publish the logical index independently.

CREATE and CREATE OR REPLACE build all fragments visible in the starting snapshot. BUILD INDEX computes the uncovered fragments from the latest logical description and sends only those fragments through the same segment pipeline. Concurrent appends after planning remain unindexed and can be covered by a later BUILD INDEX.

For object-store-backed datasets, workers use the catalog credentials and the dataset URI resolved from the latest table metadata.

A local/file dataset is mutable only when the deployment guarantees that the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation. Otherwise mutation is rejected. Local filesystem support is intended primarily for single-node development and tests; it must not silently depend on the current master FE's local disk.

REST Namespace Catalog

REST behavior is capability-based:

• SHOW INDEX uses Namespace list/stats operations.
• Create, replace, and drop use the Namespace index operations when supported by the server.
• Managed-versioning and service-side transaction policy are preserved.
• Doris never performs a direct Dataset mutation with vended credentials as a fallback for an unsupported Namespace mutation.
• An unsupported Namespace operation returns a clear capability error.
• If the Namespace service returns an asynchronous transaction or operation ID, the Doris job tracks and polls it.
• Physical-segment inspection is exposed only when the Namespace capability supplies the required metadata; Doris does not bypass the service to obtain it.

6. Initial type and property matrix

Only types verified by both creation and the Doris query path are enabled.

IVF_PQ

• SQL category: ANN
• Required property: index_type=IVF_PQ
• Column count: exactly one
• Column shape: fixed-size list
• Initial element types: FLOAT16, FLOAT32, UINT8, and INT8; FLOAT64 is deferred because the pinned lance-c creation surface does not advertise it
• Nullability: the vector field and its elements must be non-null in Phase 1
• Metrics: L2, COSINE, and DOT
• num_partitions: optional positive integer; SDK default is used when absent
• num_sub_vectors: optional positive integer; must divide the vector dimension
• num_bits: fixed to 8 in Phase 1
• Advanced centroids, codebooks, sample rate, training iterations, and HNSW/RQ/SQ properties are not user-visible in Phase 1
• Build/query compatibility is verified against the exact Lance Java SDK and lance-c versions pinned by Doris

BTREE

• Column count: exactly one
• Initial column types: integral numeric, floating numeric, DECIMAL, STRING, DATE, DATETIME, and TIMESTAMPTZ types already supported by the Lance predicate pushdown path
• Nullable columns are supported only with end-to-end coverage for equality, range, IS NULL, and IS NOT NULL over indexed and unindexed fragments
• Optional zone_size must be a positive integer
• Distributed-only/internal fields such as range identifiers are not exposed as user properties

BITMAP

• Column count: exactly one
• Initial column types: BOOLEAN, integral numeric, STRING, and DATE
• Nullable columns are supported with explicit NULL predicate tests
• No user-visible properties in Phase 1
• Distributed shard identifiers remain executor-internal

Deferred types

• IVF_FLAT remains disabled until an index built by the lifecycle path is consumed by the Doris vector query path in an end-to-end test.
• Lance INVERTED is not exposed as Doris INVERTED. It is an FTS/BM25 index with different query and tokenizer semantics. A later design should expose it as FTS together with Doris FTS query support.
• Other Lance scalar and vector types are rejected rather than passed through.

7. Logical and physical metadata surfaces

Lance has a logical index identified by name, columns, type, definition, and aggregate coverage. It may contain multiple physical segments with independent UUIDs, dataset versions, and fragment sets.

SHOW INDEX

The existing 13-column Doris schema is retained. It returns one row per logical index/column:

• Key_name: logical name;
• Column_name: indexed column;
• Index_type: ANN, BTREE, or BITMAP;
• Properties: bounded logical details only, such as Lance physical type, metric, indexed/unindexed row and fragment counts, and supported index-specific parameters.

The Properties field never contains a segment array or fragment-ID list.

Logical index count is the number of user-visible logical descriptions, excluding system indexes. It is not derived from physical manifest entries. No separate count statement is added.

Physical metadata

A filterable lance_index_segments TVF exposes one row per physical segment:

• catalog, database, table;
• logical index name;
• segment UUID;
• physical index type and version;
• dataset version;
• indexed row and fragment counts;
• creation timestamp when available.

Phase 1 does not expose the complete segment-to-fragment mapping. Aggregate logical coverage is available from SHOW INDEX, and per-segment fragment counts are available from the TVF. Explicit fragment-ID membership is deferred until there is a demonstrated diagnostic use case, avoiding another potentially unbounded result surface.

The TVF reads live metadata from the selected Lance snapshot. It does not persist a copy in Doris catalog metadata.

8. Authorization and identifier semantics

• CREATE, CREATE OR REPLACE, BUILD, DROP, and CANCEL require table-level ALTER.
• SHOW INDEX, SHOW BUILD INDEX, and physical metadata inspection require table-level SHOW.
• Remote storage still uses catalog credentials; Doris table privilege remains the user-facing authorization boundary.

Doris preserves the display case returned by Lance but resolves user-supplied index names case-insensitively, matching existing Doris/MySQL identifier behavior.

• New indexes whose names differ only by case are rejected.
• Create, build, replace, and drop first resolve a unique case-insensitive logical match, then pass the exact physical name to Lance.
• If a pre-existing external dataset contains case-only collisions, SHOW displays all of them, while mutation by that ambiguous name fails with an explicit ambiguity error.
• Phase 1 does not claim quoted exact-name resolution because the current parsed identifier path does not reliably preserve a distinct quoted-name matching mode.

9. Versioning, concurrency, and visibility

• A successful create, replace, build, or drop commits a new table version atomically.
• Queries already pinned to an older version continue to use that version.
• Mutation of a historical-version table is rejected.
• Ordinary mutation and live inspection target the latest version.
• Concurrent append is compatible with a build; newly appended fragments may remain uncovered and are reported as unindexed.
• BUILD INDEX later adds coverage for uncovered fragments.
• Overwrite, restore, fragment rewrite, or data replacement follows Lance transaction-conflict rules.
• Doris does not hold table metadata locks for the lifetime of a build.
• Doris prevents two active jobs for the same dataset and case-insensitive logical name, but it does not assume it controls external writers.
• Queries over a partially indexed table must combine indexed results with scans of unindexed fragments. End-to-end tests must prove result equivalence with index use disabled.

10. Failure recovery and metadata refresh

The implementation distinguishes three outcomes.

Confirmed pre-commit failure

The external manifest or Namespace transaction was not committed. The job becomes FAILED. Doris does not run post-DDL refresh. Unreferenced artifacts are reclaimed later by Lance cleanup according to retention policy.

Confirmed commit

The requested postcondition is visible in the latest snapshot:

• create: a matching logical definition is present;
• replace: the definition matches and the old segment identity has been replaced;
• incremental build: coverage has advanced or is already complete;
• drop: the logical name is absent.

Doris invalidates and refreshes table metadata and emits the normal external-DDL refresh edit log. A refresh failure does not turn the job into an ordinary build failure; the job becomes COMMITTED_REFRESH_PENDING and retries refresh.

Unknown or post-commit failure

After FE failover, timeout, lost worker response, or an ambiguous commit result, Doris reopens the latest snapshot or queries the Namespace transaction and verifies the operation-specific postcondition.

• If the postcondition holds, Doris records the commit, runs refresh, and finishes the job.
• If it does not hold and the provider proves no commit occurred, the job fails or is safely retried according to the operation policy.
• If the outcome remains unknowable, the job remains OUTCOME_UNKNOWN and exposes a diagnostic rather than silently rebuilding.

A reconciled IF NOT EXISTS or IF EXISTS no-op still invalidates/refreshes Doris metadata when it observes external state that may be newer than the local cache.

SHOW INDEX is the user-visible reconciliation tool, but it is not a substitute for repairing Doris metadata caches.

11. Test plan

Tests cover:

• parser and to-SQL behavior for ANN/index_type, BTREE, BITMAP, OR REPLACE, BUILD, SHOW, CANCEL, and DROP;
• internal-table behavior remaining unchanged after neutral command refactoring;
• ALTER/SHOW privileges;
• the complete type/property matrix and rejection of unknown values;
• duplicate, IF NOT EXISTS, case-only duplicate, ambiguous pre-existing name, replace, incremental build, and drop semantics;
• durable job state transitions, FE restart/failover recovery, concurrency limits, cancellation, and retryable conflicts;
• successful commit followed by edit-log/refresh failure and subsequent reconciliation;
• logical versus physical metadata and multi-segment logical counts;
• append after create, coverage decay, incremental BUILD, and query correctness across indexed plus unindexed fragments;
• historical-version mutation rejection;
• local/file visibility restrictions;
• REST list/stats, supported mutation, unsupported capability, managed-versioning, and asynchronous transaction polling;
• end-to-end IVF_PQ, BTREE, and BITMAP query use through the Doris BE reader.

Regression tests use isolated writable datasets and generate expected result files through the normal regression-test scripts.

12. Confirmed design decisions and implementation decomposition

The review questions from v1 are resolved as follows:

1. Production execution: FE-side native construction is rejected. Directory catalogs use durable asynchronous jobs with distributed BE segment construction; REST catalogs use Namespace-managed operations.
2. Native prerequisite: Phase 1 extends lance-c with fragment-scoped uncommitted build, physical metadata transfer, progress, and cooperative cancellation. BUILD INDEX uses this distributed uncovered-fragment path rather than running Dataset.optimizeIndices inside FE.
3. Commit ownership: BE workers produce uncommitted physical segments. FE validates disjoint coverage and performs the single atomic logical-index commit. A persisted cancellation or stale job generation fences commit.
4. SQL and job control: Vector creation uses USING ANN with index_type=IVF_PQ. CREATE OR REPLACE performs a full rebuild, BUILD INDEX performs incremental coverage, and the existing SHOW BUILD INDEX / CANCEL BUILD INDEX surfaces are extended for Lance jobs.
5. Metadata: SHOW INDEX remains a bounded logical summary. Phase 1 adds one lance_index_segments TVF for physical UUIDs, versions, and aggregate segment coverage. Full fragment membership is deferred.
6. REST: Inspection uses Namespace list/stats. Mutation uses Namespace index operations only when supported and never bypasses the service with vended credentials.
7. Initial types: Phase 1 enables IVF_PQ, BTREE, and BITMAP only. IVF_FLAT requires a separate end-to-end query verification, and Lance INVERTED remains deferred until Doris exposes FTS semantics.
8. Recovery: A confirmed or reconciled external commit always enters the refresh path. Refresh failures remain durable as COMMITTED_REFRESH_PENDING and are retried.

Implementation is split into the following dependency-ordered work items:

1. lance-c APIs: add fragment-scoped uncommitted builds, segment metadata transfer, progress, cancellation, and the required merge support.
2. FE command model: introduce ParsedIndexSpec and catalog-specific providers without changing internal OLAP index serialization.
3. Durable job framework integration: persist Lance job state, generation fencing, admission limits, failover recovery, SHOW, and CANCEL behavior.
4. BE execution: schedule fragment groups, build segments under workload management, report progress, and discard cancelled/stale results.
5. Commit and reconciliation: validate segment coverage, commit once, reconcile unknown outcomes, and retry metadata refresh.
6. Catalog and metadata surfaces: implement Directory and REST providers, logical SHOW INDEX, and the lance_index_segments TVF.
7. Tests and documentation: complete the matrix in Section 11 and document lifecycle guarantees and limitations.

These items may be delivered as separate dependent PRs, but the production Directory create/build path remains disabled until the native build, cancellation, durable job, commit-fencing, reconciliation, and end-to-end query requirements are all present.

Remaining implementation-planning question

The technical direction above is fixed, but the delivery boundary still needs reviewer agreement:

Is direct BE execution through an extended lance-c the preferred integration model, or should Doris use an external Java/Rust index-build service?

References:

• Doris existing CREATE/BUILD/SHOW/CANCEL index command paths
• [feature](lance) support lance catalog in doris #65730 and [Tracking] Lance feature matrix for Doris 4.2 #66340
• Lance distributed indexing: https://lance.org/guide/distributed_indexing/
• Lance Spark CREATE INDEX: https://lance.org/integrations/spark/operations/ddl/create-index/
• Lance Spark SHOW INDEXES: https://lance.org/integrations/spark/operations/ddl/show-indexes/
• Lance Namespace operations: https://lance.org/format/namespace/operations/
• Pinned Lance Java SDK v9.1.0-beta.3 and lance-c v0.1.2

[Gabriel39]

Thanks for the detailed revision. V2 addresses most of the architectural concerns from the first review: synchronous FE construction is no longer the production path, the neutral command model is clearer, BUILD INDEX is separated from full replacement, logical and physical metadata are separated, and the recovery states are substantially better defined.

I think the Directory Catalog direction is now viable, but I do not think the design is implementation-ready yet. The following issues still need to be resolved.

1. The REST design currently promises more than the pinned Namespace API can express

Doris pins Lance Namespace 0.7.7. Its CreateTableIndexRequest supports the column, index type, name, distance type, and FTS-related fields, but it does not carry num_partitions, num_sub_vectors, zone_size, replace semantics, or an uncovered-fragment/incremental-build request:

https://github.com/lancedb/lance-catalog/blob/v0.7.7/java/lance-namespace-apache-client/docs/CreateTableIndexRequest.md

There is also no separate Namespace BUILD INDEX operation. In addition, ListTableIndices exposes only name, UUID, columns, and status, while index stats expose only type, distance, indexed/unindexed row counts, and physical index count. That is not sufficient to verify that an existing REST index has the complete normalized definition required by the proposed IF NOT EXISTS reconciliation.

Please either:

• make the required Namespace specification/API changes an explicit prerequisite; or
• define a Phase-1 REST capability matrix and reject unsupported replace, build, and property combinations instead of promising uniform behavior.

“Advertised capabilities” should also be made concrete. In the pinned Java interface, optional operations are primarily represented by typed UnsupportedOperationException; there is no general capability-advertisement API.

2. Reusing SHOW/CANCEL BUILD INDEX requires a real external-job integration design

The current CancelBuildIndexCommand explicitly rejects external catalogs and eventually looks up an internal OlapTable and SchemaChangeHandler.IndexChangeJob:

https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CancelBuildIndexCommand.java

Similarly, SHOW BUILD INDEX reads the internal /jobs/<dbId>/build_index proc data and has internal-index-specific columns such as PartitionName and AlterInvertedIndexes:

https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowBuildIndexCommand.java

Extending the SQL vocabulary is reasonable, but the design should specify the common job registry/proc source, catalog qualification, filtering, privilege checks, output schema, and state mapping for internal and Lance jobs. Otherwise the SQL examples in this proposal cannot work through the existing implementation.

3. REST reconciliation needs job-specific provenance, not only a matching postcondition

The remote transaction_id is optional. If the service commits the operation but FE fails before persisting the returned transaction ID, a later “same name and matching definition” observation does not prove that this Doris job performed the commit; an external writer may have produced the same postcondition.

This is particularly important because Lance allows concurrent CreateIndex transactions. For two operations on the same logical name, the later operation can replace the earlier one rather than returning the retryable conflict assumed by this proposal:

https://lance.org/format/table/transaction/

Please persist a client operation ID before dispatch and propagate it through Namespace transaction properties or another queryable server-side idempotency/provenance mechanism. For Directory builds, expected segment UUIDs can provide this proof. Without job-specific evidence, the REST job should remain OUTCOME_UNKNOWN instead of being marked successful merely because the logical definition matches.

4. DROP also needs a durable operation record

The proposal creates durable jobs for create, replace, and build, but allows DROP to complete synchronously while still promising failover-safe post-commit reconciliation and refresh.

If DROP commits externally and FE fails before refresh/edit-log completion, there is no durable state from which to resume. DROP should therefore use the same durable Lance job/operation record, even if it normally moves through the states very quickly.

5. Please make the distributed segment contract explicit

The pinned Java SDK already exposes mergeExistingIndexSegments and commitExistingIndexSegments, so BE construction followed by one FE commit is technically plausible:

https://github.com/lance-format/lance/blob/v9.1.0-beta.3/java/src/main/java/org/lance/Dataset.java

However, the design should require:

• one pinned and verified Lance core/index-metadata compatibility contract across lance-c and the Java SDK;
• validation of field IDs, physical index type and parameters, starting dataset version, exact assigned fragment coverage, UUID uniqueness, and artifact location before FE accepts a worker result;
• for incremental BUILD, committing the complete final logical segment set (existing compatible segments plus new segments), not only the newly built segments;
• an explicit IVF model strategy. Independent per-segment models are valid for fan-out query, but physical merge requires compatible/shared model artifacts such as centroids and PQ codebooks.

See: https://lance.org/guide/distributed_indexing/

Credentials should not be persisted in the job edit log. Workers should obtain or refresh catalog credentials when a task or retry is dispatched.

6. Metadata details

I recommend reporting IVF_PQ, rather than ANN, in SHOW INDEX.Index_type. ANN is the SQL category, while IVF_PQ is the actual index type; reporting only ANN becomes ambiguous when additional ANN implementations are added.

REST metadata should also be explicitly capability-dependent: the pinned stats model does not expose fragment counts or the complete creation properties, so those SHOW fields cannot always be populated uniformly.

Integration-model recommendation

For the remaining planning question, I prefer direct BE execution through an extended lance-c for Directory Catalog. An external Java/Rust build service would introduce a new deployment, authentication, versioning, availability, and operational dependency for a core Apache Doris feature. Keeping execution in BE is a better fit as long as the production path remains gated until the required lance-c build, cancellation, progress, segment-metadata, and compatibility APIs are available.

Section 5 already says that the proposal chooses BE execution, while the final section presents this as unresolved. Please make the conclusion consistent.

Once the REST capability boundary, existing job-command integration, job-specific reconciliation, and durable DROP semantics are clarified, I think the design will be in much better shape for approval and decomposition into implementation issues.

[u70b3]

Thank you @Gabriel39 for the patient and precise guidance across both review rounds — each round has made this design substantially stronger. Below is the v3 full revision, addressing every round-2 point directly and incorporating several self-identified corrections.

Revised Design Proposal v3 — Asynchronous Lance Index Lifecycle

Draft for #66497, prepared 2026-08-07. Incorporates Gabriel39's round-2 review (2026-08-06 13:15 UTC) + self-identified errata. This revision has passed three independent adversarial reviews (research-consistency, review-coverage, source-level fact-check); every load-bearing claim carries pinned-source evidence, and the distributed build path is verified by an end-to-end PoC against lance-core 9.1.0-beta.3 (fragment-scoped uncommitted build → coordinator commit → k-NN consumption). Post as a full standalone revision.

This revision incorporates the second-round review from @Gabriel39: REST is now an explicit capability matrix, SHOW/CANCEL BUILD INDEX get a concrete external-job integration design, reconciliation requires job-specific provenance, DROP uses the same durable job record, and the distributed segment contract is specified. It also contains several self-identified corrections to v2.

/cc @Gabriel39 @zhangstar333

Major changes from v2

1. REST capability matrix replaces uniform-behavior language: a per-operation/per-property support table; unsupported combinations are rejected at analysis time where statically known, otherwise surfaced as a clear capability error.
2. Job-specific provenance: every mutation job persists a client-generated operation_id before dispatch. Reconciliation requires job-specific evidence — for Directory, the expected segment UUID set plus the operation_id recorded in the Lance transaction properties at commit time; for REST, the server-returned transaction_id (see the honest boundary in §10). A matching postcondition alone is never accepted as success — including for DROP.
3. Concurrency semantics stated in both layers: the Lance format documentation defines same-name concurrent CreateIndex as last-committer-wins; the pinned implementation enforces it via a retryable commit conflict + rebase on the first attempt. Either way, a name/definition match never proves attribution.
4. DROP is durable: it uses the same LanceIndexJob record (it usually transitions quickly), closing the post-commit failover gap — and its reconciliation follows the same provenance rules.
5. Distributed segment contract: a pinned lance-c/Java-SDK compatibility requirement, a worker-result validation checklist, full-final-segment-set commits for incremental BUILD, an explicit shared IVF model strategy, a defined BE→FE segment-metadata schema, and no credentials in the edit log.
6. SHOW/CANCEL BUILD INDEX integration design for external jobs: registry, proc source, catalog qualification, filtering, privileges, output schema, and state mapping (current implementations verified internal-only on both master and branch-4.1).
7. Metadata: SHOW INDEX.Index_type reports the physical type (IVF_PQ); REST SHOW fields are capability-dependent.
8. Type matrix narrowed: vector element types are FLOAT16/FLOAT32 only (see corrections).
9. Execution model locked: Directory = BE via extended lance-c; no external build service. The FE-side build remains an experimental, disabled-by-default development fallback using the Java SDK's existing uncommitted-build primitives.
10. Provenance-carrying commits use the general Transaction path: the one-shot Dataset.createIndex / commitExistingIndexSegments / dropIndex APIs do not carry transaction properties in the pinned SDK; commits that must record operation_id go through Transaction.Builder.transactionProperties + operation.CreateIndex + CommitBuilder (verified end-to-end through JNI in the pinned sources).

Corrections to v2 (self-identified)

• Concurrent same-name CreateIndex — both layers, precisely. The Lance format transaction documentation states: two concurrent CreateIndex operations are allowed; with the same name, "the second operation will win and replace the first" (docs/src/format/table/transaction.md:172-173). The pinned implementation reaches that semantics via conflict + rebase: check_create_index_txn returns a retryable conflict for a same-name regular index on the first attempt (rust/lance/src/io/commit/conflict_resolver.rs:577-617), and the Java CommitBuilder.maxRetries defaults to 0, so without retries the caller observes the conflict error. v2 saw only the implementation layer; the design must assume the doc layer for external writers (retried/ rebased commits can replace Doris-created indexes) — hence provenance in §10.
• UINT8 element type: the Rust core treats UInt8 vectors as binary vectors and allows only Hamming distance (rust/lance/src/index/vector/utils.rs:193,204-211). v2's matrix (UINT8 with L2/COSINE/DOT) was inconsistent. UINT8 is deferred until a binary-vector/HAMMING decision is made. INT8 is deferred: it is advertised by the pinned lance-c creation surface and accepted by the Rust core (L2/COSINE/DOT allowed), but it is an undocumented branch there and lacks end-to-end verification. FLOAT64 remains deferred — not advertised by the pinned lance-c creation surface (lance.h:926 restricts to float32/float16/uint8/int8), although the Rust core accepts it.
• REST monitoring channel: index progress is tracked through the Namespace list/stats operations — the spec states "Index creation is handled asynchronously. Use the ListTableIndices and DescribeTableIndexStats operations to monitor index creation progress" (spec.yaml:1633-1634). The optional transaction_id is a generic long-running-operation handle, not the index progress channel.
• FTS deferral rationale: the BE reader already has a full-text search path (be/src/format_v2/table/lance_reader.cpp:521 on branch-4.1). Lance INVERTED is deferred because Doris has no SQL-level FTS semantics for it yet — not because the query path is absent.
• TVF per-segment coverage dropped from Phase 1 (declared downgrade): v2 promised per-segment row/fragment counts in the physical-segment TVF. No pinned API exposes them: the lance-c list JSON is hard-coded to {name, uuid, columns, type, dataset_version} per entry (lance-c src/index.rs), and the Namespace stats model exposes only row counts (spec.yaml:3206-3234). This becomes lance-c extension item ⑥; the TVF ships without those columns until then. Reviewer approval for this deferral is requested explicitly.
• Index-name case behavior (research correction): an earlier internal research note claimed Lance lowercases index names. The pinned sources show Lance index names are case-sensitive, exactly matched, and stored as-is (rust/lance/src/index/create.rs:202, rust/lance/src/index/api.rs:211; no normalization in builder/Python/Java layers). The §8 design — Doris-layer case-insensitive resolution over Lance's exact-match storage — stands precisely because idx/IDX coexistence is a real external state.

1. Goals and Phase-1 boundary

Phase 1 provides a production-oriented lifecycle for user-visible Lance indexes:

• create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes (Directory full; REST restricted to the request model per §5);
• incrementally index uncovered fragments and atomically replace an index by name (Directory);
• drop an index by name (Directory and REST);
• inspect logical definitions, aggregate coverage, and the logical index count (Directory full; REST capability-dependent);
• durable status, cancellation, concurrency limits, provenance-based reconciliation, and metadata refresh;
• REST behavior follows the capability matrix in §5; everything outside it is rejected explicitly.

Lance dataset manifests or the Namespace service remain the authoritative index metadata. Doris persists only its own job state and reconciliation evidence. Historical-version tables remain read-only for index mutation; inspection uses the selected snapshot, mutation targets only the latest writable state.

An FE-side synchronous build is not part of the production implementation. A development fallback (experimental, disabled by default, limited by dataset-size/fragment/concurrency settings) MAY use the pinned Java SDK's existing uncommitted primitives (IndexOptions.withFragmentIds / withIndexUUID, javadoc-verified uncommitted + fragment-scoped). The production objection is compute placement in the control plane, not missing APIs.

2. SQL surface

Unchanged from v2 except the property-key question (Q2):

• vector: CREATE INDEX ... USING ANN PROPERTIES ("index_type"="IVF_PQ", "metric_type"="l2", "num_partitions"="256", "num_sub_vectors"="16") (key naming is Q2 below; example uses the proposed key);
• scalar: USING BTREE / USING BITMAP;
• CREATE OR REPLACE INDEX = full rebuild + atomic name replacement;
• BUILD INDEX = incremental coverage of uncovered fragments;
• SHOW INDEX FROM t; SHOW BUILD INDEX FROM db WHERE ...; CANCEL BUILD INDEX ON ctl.db.tbl (job_id); DROP INDEX [IF EXISTS].

Statement semantics (delta from v2):

• Create, replace, build, and drop each return after a durable job is accepted. Drop typically completes within one commit cycle but is equally durable.
• REST IF NOT EXISTS reconciliation compares only the fields observable through the Namespace stats model (name, columns, index type, distance type); properties the REST path cannot express are rejected at analysis for REST tables, so no hidden definition skew is possible (§5).
• All other semantics (OR REPLACE convergent-but-not-idempotent, single column, analysis-time rejection of unknown types/properties) unchanged.

3. Neutral command model and catalog dispatch

Unchanged from v2: ParsedIndexSpec → resolve catalog/database/table → InternalIndexProvider or LanceIndexProvider → provider validation → internal schema change or Lance external-index job. Lance definitions never enter Doris internal catalog.Index serialization.

4. Durable job framework and SHOW/CANCEL BUILD INDEX integration

Job record (persisted via edit log)

• job id, operation kind (CREATE / REPLACE / BUILD / DROP), catalog/database/table identity, dataset URI or Namespace table id;
• logical index name and normalized definition;
• operation_id (client-generated UUID), persisted before any dispatch;
• starting dataset version; expected/accepted segment UUID set; old segment UUIDs (replace);
• executor identity, worker task ids, or Namespace transaction id (persisted when returned);
• state, progress, timestamps, sanitized failure information;
• no credentials — workers obtain or refresh catalog credentials when a task or retry is dispatched.

States

Unchanged from v2: PENDING → PREPARING → BUILDING → COMMITTING → REFRESHING → FINISHED, plus FAILED / CANCELLING / CANCELLED / COMMITTED_REFRESH_PENDING / OUTCOME_UNKNOWN. DROP traverses the same machine (typically PENDING → COMMITTING → REFRESHING → FINISHED in one cycle). FE failover reconstructs unfinished jobs from the edit log and resumes polling, reconciliation, commit, or refresh; it does not blindly restart native training.

Resource admission

Unchanged from v2, restated here because it gates the production path: jobs are admitted through explicit per-catalog and cluster-wide concurrency limits; BE execution is associated with workload-management accounting where the executor supports it.

Integration with the existing job commands

The grammar already accepts the required shapes on both master and branch-4.1: SHOW BUILD INDEX ((FROM|IN) db)? wildWhere? sortClause? limitClause? and CANCEL BUILD INDEX ON <multipart table> (job_ids) (DorisParser.g4: master fe-sql-parser :392-393/:644-646; branch-4.1 fe-core :390-391/:640-642). The current implementations are internal-only on both branches: CancelBuildIndexCommand.validate calls Util.prohibitExternalCatalog (:80) and resolves an OlapTable + SchemaChangeHandler.IndexChangeJob; ShowBuildIndexCommand builds the internal proc path /jobs/<dbId>/build_index (:188-191) with internal-specific columns (PartitionName, AlterInvertedIndexes, TITLE_NAMES :68-74). The design:

• Registry: a catalog-agnostic external-index job registry in FE, edit-log persisted and rebuilt on replay, sibling to the internal IndexChangeJob registry. Lance jobs register here; the internal path is untouched.
• Proc/SHOW source: SHOW BUILD INDEX unions internal and external jobs. A new JobType column (INTERNAL / EXTERNAL_LANCE) distinguishes rows. Internal-only columns stay empty for external rows (Lance jobs are table-level, not partition-level). External rows populate the generic columns: JobId, DbName (catalog-qualified), TableName, IndexName, State, Progress, CreateTime, FinishTime, Msg.
• Catalog qualification: CANCEL's multipart table name already parses catalog.db.tbl (grammar + LogicalPlanBuilder.visitCancelBuildIndex are catalog-aware). SHOW's FROM clause currently takes a single identifier only; it is extended to accept a catalog-qualified catalog.db (preferred, consistent with the table qualification) — exact grammar detail in the PR.
• Filtering and privileges: external rows are visible with table-level SHOW on the resolved external table; CANCEL requires table-level ALTER.
• State mapping: external-index job states are displayed verbatim (PENDING … OUTCOME_UNKNOWN); WHERE filtering on State uses these same job-state values; documentation maps them to the internal lifecycle.
• Cancellation semantics: unchanged from v2 (persisted before workers are signaled; fences COMMIT; generation-checked results). For REST-tracked operations there is no Namespace cancel operation: cancellation fences the Doris-side commit/refresh and stops polling; a later external completion is handled by the provenance rules in §10.
• Conflict reporting: automatic conflict retry decisions are never made by parsing native exception strings; typed conflict results are used where the provider exposes them (unchanged from v2).

5. Execution model

Directory Catalog: BE pipeline + segment contract

Production execution requires extending lance-c (prerequisite, §12). FE orchestrates; BE workers build; FE performs the single atomic logical-index commit through the pinned Java SDK.

Commit path (verified against pinned sources): provenance-carrying commits go through the general transaction path — Transaction.Builder.transactionProperties(Map) (Transaction.java:176-179) carrying the job's operation_id, with operation.CreateIndex, executed via CommitBuilder (JNI wiring: lance-jni/src/transaction.rs:830-846). The one-shot Dataset.createIndex(IndexOptions), commitExistingIndexSegments, and dropIndex paths do not carry transaction properties (they build the commit internally with None properties; create/commit-existing: rust/lance/src/index.rs:1538-1547, drop: rust/lance/src/index.rs:1177-1194) and are therefore not used for provenance commits.

Segment contract:

• Compatibility: the pinned lance-c and the pinned Java SDK must be verified against the same Lance core index-metadata format. The commit-side API surface is verified in the pinned sources: Index.Builder (Index.java:187-258), commitExistingIndexSegments (Dataset.java:1176), IndexOptions.withFragmentIds/withIndexUUID (IndexOptions.java:135-157), and the commit path's own validation (non-empty set, unique UUIDs, non-overlapping fragment coverage; rust/lance/src/index.rs:100-131). The full loop is additionally verified end to end by a PoC on lance-core 9.1.0-beta.3: fragment-scoped uncommitted builds over disjoint fragment subsets → single coordinator commit → dataset version increment by exactly one → k-NN query consumes the resulting multi-segment logical index. An end-to-end Doris test will prove the same for a BE-built (lance-c) segment committed via the Java SDK.
• BE→FE segment metadata schema: per segment, workers report: uuid, field IDs, name, datasetVersion, indexVersion, indexType, fragment coverage, and the indexDetails protobuf bytes (createdAt/baseId optional; createdAt is filled by the commit side). This is the exact field set the commit path requires — validated statically (JNI index_metadata_to_segment, lance-jni/src/blocking_dataset.rs:1218-1243, rejects segments missing fragment coverage or index details; segment.fields is validated against the indexed column's field id, rust/lance/src/index.rs:1469-1475) and empirically (PoC: a Builder-reconstructed Index missing any of these fails before commit). Bare UUIDs are not sufficient.
• Segment identity: every segment carries its own UUID — commitExistingIndexSegments rejects duplicate segment UUIDs (PoC-verified); the "shared UUID" wording in withIndexUUID's javadoc belongs to the separate merge flow. FE pre-assigns per-worker UUIDs (via withIndexUUID where the executor supports it) so the expected UUID set is known before dispatch; this set backs the provenance rule in §10.
• Worker result acceptance: before accepting a worker result, FE validates per segment: field IDs, physical index type and parameters, starting dataset version, exact assigned fragment coverage (disjoint across workers and equal to the assignment), UUID uniqueness, and artifact location beneath the dataset index directory.
• Full final segment set: CREATE/REPLACE commit the complete segment set for the starting snapshot. Incremental BUILD commits the complete final logical segment set — existing compatible segments plus newly built ones — not only the new segments (PoC-verified; note that a follow-up commit on an already-committed name uses replace semantics in the pinned SDK).
• IVF model strategy (Phase 1): shared model. A dedicated training task derives IVF centroids and PQ codebooks once from a sample of the starting snapshot; all build workers use the shared model — concretely, the pinned Java SDK supports exactly this split (centroids trained once via VectorTrainer.trainIvfCentroids, distributed to builders through IvfBuildParams.setCentroids; PoC-verified across workers); on the BE path this depends on lance-c extension ① accepting caller-supplied centroids/codebooks (§12). This keeps segments mutually compatible for a future physical merge and yields uniform recall across segments. Independent per-segment models (valid for fan-out query) are rejected for Phase 1 because they preclude later merge. Physical segment merge itself is not performed in Phase 1.
• Credentials: never persisted in the job record; obtained/refreshed at task dispatch (§4).
• Local/file restriction (restored from v2): a local/file dataset is mutable only when the deployment guarantees that the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation; otherwise mutation is rejected. Local filesystem support is intended primarily for single-node development and tests; it must not silently depend on the current master FE's local disk.

FE experimental fallback: unchanged gating (experimental, disabled by default, size/fragment/concurrency limits). Implementation uses the Java SDK's existing uncommitted primitives; no lance-c dependency.

REST Namespace Catalog: capability matrix

Capability detection is behavioral: the pinned Java interface represents unsupported operations as default methods throwing a typed UnsupportedOperationException bound to spec error code 0 (LanceNamespace.java:402-448, errors/ErrorCode.java:22); there is no capability-advertisement API. Unsupported combinations fail during analysis where statically known, otherwise with a clear capability error. Doris never falls back to direct Dataset mutation with vended credentials. All REST mutations use the durable job record.

Operation	REST Phase-1 (pinned Namespace 0.7.7)
SHOW INDEX	Yes — via ListTableIndices + DescribeTableIndexStats. Fields limited to the stats model: name, UUID, columns, status; index type, distance, indexed/unindexed row counts, physical index count (spec.yaml:3149-3234). Fragment counts and creation properties are unavailable → corresponding SHOW fields are empty.
CREATE INDEX	Restricted — only what CreateTableIndexRequest carries: column, index_type, name, distance_type (FTS fields unused; spec.yaml:3049-3108). IVF build properties (num_partitions, num_sub_vectors) and BTREE zone_size are rejected for REST tables.
CREATE OR REPLACE INDEX	Rejected — the request model has no replace semantics and the spec defines TableIndexAlreadyExists (error code 7); a non-atomic drop+create pair is not offered under OR REPLACE.
BUILD INDEX	Rejected — no incremental/uncovered-fragment operation exists in the Namespace API.
DROP INDEX	Yes — via DropTableIndex (LanceNamespace.java:445); still uses the durable job record.
lance_index_segments TVF	Rejected — the stats model exposes no per-segment metadata.

Monitoring and provenance: progress is polled via ListTableIndices / DescribeTableIndexStats (the spec-sanctioned channel). If the server returns a transaction_id, it is persisted as the REST job's only server-side evidence (§10). The pinned request model cannot carry a client operation ID or transaction properties; adding a namespace-level idempotency/provenance key is listed as a spec-evolution follow-up.

(Alternative considered: make Namespace spec/API changes — replace semantics, build parameters, incremental build — an explicit prerequisite and offer uniform REST behavior. The matrix above is the Phase-1 proposal; the spec-evolution path can be a follow-up with the Lance community.)

6. Initial type and property matrix

• IVF_PQ: SQL category ANN + index_type=IVF_PQ; exactly one fixed-size-list column; element types FLOAT16, FLOAT32 (others deferred, see corrections); field and elements non-null; metrics L2/COSINE/DOT; num_partitions, num_sub_vectors optional positive integers (num_sub_vectors must divide the dimension); num_bits fixed to 8.
• BTREE: one column; integral/floating numeric, DECIMAL, STRING, DATE/DATETIME/TIMESTAMPTZ per the existing Lance pushdown support; nullable columns gated by the NULL-predicate tests; optional zone_size positive integer (Lance default 4096, DEFAULT_BTREE_BATCH_SIZE, rust/lance-index/src/scalar/btree.rs:91).
• BITMAP: one column; BOOLEAN, integral numeric, STRING, DATE; explicit NULL predicate tests; no user-visible properties.
• Deferred: IVF_FLAT (end-to-end query verification), Lance INVERTED (needs Doris FTS SQL semantics; to be exposed as FTS), UINT8/INT8/FLOAT64 (see corrections).

7. Logical and physical metadata surfaces

• SHOW INDEX: unchanged 13-column schema (ShowIndexCommand.java:52-66, identical on master and branch-4.1). One row per logical index/column: the pinned Lance list APIs return one entry per physical segment — multiple same-named entries for a multi-segment logical index (PoC-verified) — so SHOW INDEX groups by logical name. Index_type reports the physical Lance type (IVF_PQ, BTREE, BITMAP) rather than the SQL category. Properties carries bounded logical details only (physical type, metric, parameters, aggregate indexed/unindexed rows and fragments from the logical description) — never segment arrays. The logical index count is the number of user-visible logical index descriptions (i.e., distinct grouped names), excluding Lance system indexes, and is not derived from physical manifest entries; no separate count statement is added.
• lance_index_segments TVF: catalog, database, table, logical index name, segment UUID, physical index type, dataset version, creation timestamp when available. Per-segment row/fragment counts are deferred (see corrections): the lance-c list JSON carries per-entry {name, uuid, columns, type, dataset_version} and no fragment coverage or row counts; this becomes lance-c extension item ⑥.
• REST: SHOW fields are capability-dependent per §5; the TVF is rejected.

8. Authorization and identifier semantics

Unchanged from v2: mutation + CANCEL require table-level ALTER; SHOW surfaces require table-level SHOW. Doris resolves user-supplied index names case-insensitively (quoted and unquoted identifiers alike) and preserves display case; new case-only duplicates are rejected; pre-existing external collisions surface an explicit ambiguity error on mutation. Note the layered reality (corrected research): Lance itself is case-sensitive and exact-matching (create.rs:202, api.rs:211), so Doris's case-insensitive resolution is a Doris-layer policy over exact-match storage — which is exactly why external case-only collisions are a real state that must be handled explicitly.

9. Versioning, concurrency, and visibility

• Successful create/replace/build/drop commits a new dataset version atomically; queries pinned to older versions are unaffected; historical-version mutation is rejected.
• Concurrent same-name CreateIndex (external writers): the format documentation defines last-committer-wins; the pinned implementation reaches it via retryable conflict + rebase (see corrections). Doris serializes same-name jobs internally (case-insensitive) but does not control external writers, which may retry and therefore replace; reconciliation never treats a name/definition match as proof of attribution (§10).
• Conflicts (per transaction.md:175-194): Overwrite/Restore/UpdateMemWalState conflict with index creation. Rewrite conflicts only with overlapping fragments and no stable row ids and no fragment reuse index; DataReplacement conflicts only with overlapping fragments when the replaced column is being indexed — these are surfaced as retryable. Concurrent append is compatible; appended fragments are reported as unindexed and covered by a later BUILD.
• Doris does not hold table metadata locks for the lifetime of a build.
• Conflict-retry decisions are never made by parsing native exception strings (typed conflict results only).
• Queries over a partially indexed table must combine indexed results with scans of unindexed fragments; end-to-end tests must prove result equivalence with index use disabled.

10. Failure recovery, provenance, and metadata refresh

The three outcome classes from v2 are retained, with a strengthened and honestly-bounded success criterion.

Confirmed pre-commit failure

Unchanged: job FAILED; no post-DDL refresh; unreferenced artifacts reclaimed by Lance cleanup per retention.

Confirmed commit — requires job-specific evidence

• Directory (create/replace/build): the logical index in the reopened latest snapshot is backed by exactly the job's expected segment UUID set — pre-assigned by FE before dispatch where the executor supports it (§5), so the expectation is deterministic rather than observed; the commit itself carries the job's operation_id in the Lance transaction properties via the general Transaction path (§5; runtime-verified readable back from the new version's transaction).
• Directory (drop): the pinned SDK has no DropIndex operation class, and the one-shot Dataset.dropIndex commits internally with None properties (rust/lance/src/index.rs:1177-1194), so it is not used for provenance commits. A provenance-carrying drop goes through the same general Transaction path, expressed as operation.CreateIndex with an empty new-index list and removedIndices set to the index metadata loaded for the name before dispatch (withRemovedIndices, operation/CreateIndex.java:92-94) — mirroring the Rust one-shot's internal operation — with transactionProperties carrying the job's operation_id. That recorded operation_id is the job-specific evidence; the exact removed segment UUID set is further corroboration. The postcondition "name is absent" is corroboration only — an external writer can independently drop the same name (false positive) or recreate it after our drop (false negative). Without transaction-properties evidence, a Directory drop follows the unknown-outcome rule below.
• REST: the pinned Namespace API cannot carry a client operation ID in index requests. The only job-specific evidence is the server-returned transaction_id (persisted when returned), matched via the generic transaction API (DescribeTransaction, LanceNamespace.java:754) where the server supports it. A matching logical definition observed through list/stats is corroboration only.

On a confirmed commit, Doris invalidates/refreshes metadata and emits the external-DDL refresh edit log; a refresh failure transitions the job to COMMITTED_REFRESH_PENDING with retry (unchanged).

Unknown or post-commit failure

Reconcile by reopening the latest snapshot / querying the Namespace state, as in v2 — but if no job-specific evidence is available (expected: FE failover before persisting the returned transaction_id; a drop without transaction-properties evidence; any REST operation on a server that returns no transaction_id), the job remains OUTCOME_UNKNOWN with a diagnostic. It is never marked successful from a matching postcondition alone — this is required precisely because same-name CreateIndex is last-committer-wins for external writers.

A reconciled IF NOT EXISTS / IF EXISTS no-op still invalidates/refreshes Doris metadata when newer external state is observed (unchanged).

11. Test plan

v2 coverage, plus:

• capability matrix: every rejected REST combination fails with the designed error (analysis-time vs capability error);
• REST IF NOT EXISTS compares only stats-observable fields (name, columns, type, distance);
• provenance: Directory commit carries operation_id in transaction properties (read back from the new version's transaction); commit-with-lost-response reconciled via UUID set; drop commits are expressed as operation.CreateIndex with removedIndices over the general Transaction path (verified through JNI) and reconciled only via transaction-properties evidence; matching-postcondition-without-evidence stays OUTCOME_UNKNOWN (including REST with no transaction_id returned);
• external same-name replacement during a Doris job (retried create replacing ours) is detected and attributed correctly;
• DROP failover: drop commits externally, FE fails before refresh → resumed from the durable record;
• incremental BUILD commits the full final segment set (existing + new), validated against duplicate-UUID / overlapping-coverage rejection;
• BE→FE segment metadata schema: segments missing fragment coverage or index details are rejected before commit (mirroring the JNI conversion requirements);
• shared-model build: segments built by different workers are merge-compatible and fan-out query recall is uniform;
• SHOW/CANCEL BUILD INDEX: union display, JobType column, catalog qualification, privilege filtering, external-index job states, cancellation fencing;
• resource admission: per-catalog and cluster-wide concurrency limits enforced;
• local/file visibility restrictions (mutation rejected when the path is not uniformly visible);
• job record contains no credentials (edit-log content test);
• REST monitoring via list/stats polling; transaction_id persisted when present.

12. Confirmed design decisions and implementation decomposition

1. Execution: Directory = BE via extended lance-c, production-gated until the required APIs land; REST = Namespace capability matrix; no external build service (deployment/auth/ops dependency for a core feature is not justified); FE sync build = experimental fallback only.
2. lance-c extensions (prerequisite): ① fragment-scoped uncommitted index builds, including caller-supplied shared IVF centroids / PQ codebooks (so the §5 shared-model strategy reaches BE workers); ② segment metadata serialization/transfer (BE→FE per §5 schema); ③ progress reporting; ④ cooperative cancellation; ⑤ segment merge / commit-existing; ⑥ per-segment coverage/stats exposure (feeds the TVF deferral). Upstream engagement starts now — lance-c is actively landing related distributed APIs (v0.1.3 added segment-selection for distributed search).
3. Segment contract: pinned compatibility, BE→FE metadata schema, validation checklist, full-final-set commits, shared IVF model, no persisted credentials.
4. Commit path: provenance commits use the general Transaction path (Transaction.Builder.transactionProperties + operation.CreateIndex + CommitBuilder); one-shot convenience APIs are not used where provenance is required.
5. Provenance: operation_id persisted pre-dispatch; success requires job-specific evidence; otherwise OUTCOME_UNKNOWN. REST boundary per §10 is stated explicitly.
6. DROP durable; SHOW/CANCEL BUILD INDEX integration per §4; resource admission per §4.
7. SQL: USING ANN + index_type=IVF_PQ; OR REPLACE = full rebuild; BUILD = incremental; SHOW Index_type = physical type.
8. Types: IVF_PQ (FLOAT16/FLOAT32) + BTREE + BITMAP only.
9. REST: capability matrix; mutation never bypasses the service.

Work items (dependency-ordered; deliverable as separate PRs):

1. lance-c extensions (upstream; tracked prerequisite subtask, then pin bump);
2. FE neutral command model (ParsedIndexSpec + providers; internal behavior unchanged);
3. Durable job framework + SHOW/CANCEL integration (registry, edit log, proc union, privileges, resource admission);
4. BE build pipeline (training task, build workers, result validation, credential handling);
5. Commit, provenance, reconciliation (general Transaction path, segment contract enforcement, refresh retry);
6. Metadata surfaces (SHOW INDEX physical type, TVF, REST capability matrix);
7. Tests and documentation per §11 + lifecycle guarantees/limitations doc.

Questions for reviewers

• Q1 — target branch: [feature](lance) support lance catalog in doris #65730 landed on branch-4.1 only; master currently has no Lance catalog/FE code, and the branches have diverged structurally (the SQL grammar lives in fe/fe-sql-parser on master vs fe/fe-core on branch-4.1; both verified). Should [Feature](lance) Add vector and scalar index lifecycle management #66497 implementation target branch-4.1 only, master-first with a 4.1 backport, or dual-track?
• Q2 — metric property key: since vector creation reuses the Doris category surface (USING ANN), I propose adopting the internal ANN property key metric_type (verified required by AnnIndexPropertiesChecker). One caveat: internal ANN currently accepts only l2_distance and inner_product — COSINE would be a newly introduced value for this key, and Lance value names differ (l2/cosine/dot). Options: (a) metric_type with Lance value names (l2/cosine/dot), documented as Lance-table-specific values; (b) metric_type with Doris value names mapped to Lance (l2_distance→L2, inner_product→DOT, plus new cosine→COSINE); (c) fully Lance-native key metric. My preference is (a) for key consistency with internal ANN while keeping Lance-native values. Build parameter names stay Lance-native (num_partitions, num_sub_vectors, num_bits) in all options.
• Q3 — lance-c packaging: confirm the lance-c extension is tracked as an explicit prerequisite subtask (upstream issue + PRs, then a pin bump in Doris), with [Feature](lance) Add vector and scalar index lifecycle management #66497 remaining the parent lifecycle issue until the native and Doris work is integrated end to end. Related: the TVF per-segment coverage columns (see corrections) ride on extension item ⑥ — please confirm the Phase-1 TVF ships without them.

References

• v2 proposal and round-1/round-2 review threads on this issue
• [feature](lance) support lance catalog in doris #65730, [Tracking] Lance feature matrix for Doris 4.2 #66340
• Lance transaction semantics: https://lance.org/format/table/transaction/ (pinned: docs/src/format/table/transaction.md:162-194); implementation conflict path: rust/lance/src/io/commit/conflict_resolver.rs:577-617 (pinned v9.1.0-beta.3)
• Lance distributed indexing: https://lance.org/guide/distributed_indexing/
• Lance Namespace operations + spec: https://lance.org/format/namespace/operations/ (pinned: spec.yaml:1633-1634, 3049-3234)
• Pinned Lance Java SDK v9.1.0-beta.3: Dataset.java (:590/:607/:1106/:1176), IndexOptions.java (:135-157), Index.java (:187-258), Transaction.java (:176-179), operation/CreateIndex.java (:92-94), CommitBuilder.java (:75,:273); JNI: lance-jni/src/transaction.rs:830-846, lance-jni/src/blocking_dataset.rs:1218-1243
• Pinned lance-c v0.1.2 (include/lance/lance.h); upstream lance-c v0.1.3 segment-selection APIs
• Doris (master + branch-4.1): CancelBuildIndexCommand.java (:80), ShowBuildIndexCommand.java (:68-74,:188-191), ShowIndexCommand.java (:52-66), DorisParser.g4 (master fe-sql-parser :392-393/:644-646; 4.1 fe-core :390-391/:640-642), AnnIndexPropertiesChecker.java (:42-46)

[Gabriel39]

Thanks for the v3 revision. After checking the confirmed 4.2 release boundary and re-reading the baseline implementation in #65730, I need to correct part of my previous review and narrow the proposal.

There are two confirmed product constraints:

1. Doris 4.2 is based on branch-4.1, not master. Therefore this work should target branch-4.1. There is no need for a master-first or dual-track implementation in this issue.
2. No functionality that requires extending lance-c will be included in this release. Such work is explicitly outside the 4.2 scope.

These constraints materially change the appropriate Phase-1 design.

Query syntax: keep the existing vector_search() contract

I rechecked #65730. The baseline deliberately exposes Lance vector search only through the vector_search() TVF:

#65730

The implementation and regression tests use:

SELECT row_id, _distance
FROM vector_search(
    "table" = "catalog.db.table",
    "column" = "embedding",
    "query_vector" = "[0,0,0,0]",
    "metric" = "l2",
    "top_k" = "5"
);

• Implementation: https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
• Regression coverage: https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy

There is no implementation or stated plan in the baseline to translate internal-table ANN query forms such as l2_distance_approximate(...) or inner_product_approximate(...) into a Lance search.

Therefore:

• Lance queries should continue to use vector_search().
• This index-management issue should not add compatibility with the internal-table ANN query syntax.
• Reusing USING ANN as a category in CREATE INDEX is only DDL/parser reuse; it does not imply query-syntax reuse.
• For Q2, I now recommend the Lance-native property already used by the existing query surface: "metric" = "l2|cosine|dot" (option c), rather than introducing metric_type with internal-table values.

Query behavior is relevant to this issue only as end-to-end verification: an index created by the lifecycle API must be consumed correctly by the existing vector_search(..., "use_index"="true") path, with a compatible metric and correct results. This issue should not redesign the query syntax or optimizer rules.

Scope the implementation to the existing lance-c 0.1.2 APIs

The pinned lance-c 0.1.2 already exposes the one-shot lifecycle APIs required by the original issue:

• lance_dataset_create_vector_index
• lance_dataset_create_scalar_index
• lance_dataset_drop_index
• lance_dataset_index_count
• lance_dataset_index_list_json

https://github.com/lance-format/lance-c/blob/v0.1.2/include/lance/lance.h#L490-L541

The 4.2 design should use those existing APIs, with the heavy direct-Dataset operation executed on a selected BE for Directory Catalog, and use the existing Namespace operations for the supported REST cases. The limitations of the one-shot API—no native progress reporting, cooperative cancellation, distributed fragment build, or job-specific transaction provenance—should be documented honestly rather than solved by expanding lance-c in this release.

The following v3 items depend on new lance-c capabilities and should be removed from the 4.2 implementation scope:

• fragment-scoped distributed builds;
• shared IVF centroid/PQ-codebook distribution to BE workers;
• BE-to-FE uncommitted segment metadata transfer;
• coordinator commit of worker-built segments;
• physical segment merge;
• native progress and cooperative cancellation;
• per-segment coverage/statistics extensions;
• incremental BUILD over uncovered fragments.

In particular, the original #66497 scope requires replacing or rebuilding an index with the same name. The existing replace=true path provides a full rebuild. Incremental BUILD INDEX was added during review, but it is not required by the original issue and should be deferred because the intended distributed/incremental implementation requires APIs outside the 4.2 boundary.

Revised Phase-1 lifecycle

For 4.2, I recommend limiting the lifecycle to:

• CREATE INDEX for the verified vector and scalar types;
• CREATE OR REPLACE INDEX mapped to the existing full-build replace=true behavior;
• SHOW INDEX, using the existing list/count metadata and grouping physical entries by logical name where necessary;
• DROP INDEX;
• restricted REST behavior based only on the pinned Namespace request/response models;
• metadata refresh, privileges, version rules, concurrency limitations, and failure behavior appropriate to these existing one-shot operations.

Do not promise a reliable native CANCEL operation if the underlying call cannot be interrupted. Likewise, SHOW should not promise coverage fields that require a new lance-c API. UUID, columns, type, and dataset version are already available and satisfy the original metadata scope.

The query-side acceptance test should be limited to proving that a created IVF_PQ index is used by the existing vector_search() path and that scalar indexes are consumed by the existing Lance predicate path. FTS SQL design, internal ANN query syntax, distributed recall/model strategy, and future physical merge behavior are outside this issue.

Answers to the v3 questions

• Q1: target branch-4.1 only. This is the confirmed base for Doris 4.2.
• Q2: use the existing Lance-facing metric key and values (l2, cosine, dot), matching vector_search(). Do not require internal ANN query/property syntax compatibility.
• Q3: lance-c extensions are not prerequisites for this version; they should be tracked as future work. Any feature that requires such an extension, including per-segment coverage, distributed BUILD, progress, cancellation, or segment merge, must be deferred rather than gating [Feature](lance) Add vector and scalar index lifecycle management #66497.

With these scope corrections, v3 should be simplified substantially. The goal of this issue is index lifecycle management using the capabilities already present in the 4.1 baseline and pinned SDKs, not a distributed index-build framework or a unified internal/external vector-query syntax.

[u70b3]

Quick update on the prerequisite track (v3 §12 decision 2 / Q3): the upstream engagement has started — I've filed lance-format/lance-c#55, an RFC covering all six lance-c extensions, split into three independently-mergeable waves:

• Wave 1 (blocking): fragment-scoped uncommitted segment builds with shared-model injection (caller-supplied IVF centroids / PQ codebook) + segment metadata serialization (pb::IndexMetadata bytes as the BE→FE transfer format).
• Wave 2 (production quality): build progress reporting + cooperative cancellation.
• Wave 3 (symmetry/observability): commit-existing-segments C API + per-segment coverage/stats in listing.

One finding worth surfacing here, since it de-risks the schedule: the Rust core (9.1.0-beta.3) already implements nearly everything behind the C boundary. CreateIndexBuilder already carries fragments / index_uuid / train / progress / transaction_properties and execute_uncommitted() -> IndexMetadata (rust/lance/src/index/create.rs:50-141), with scalar fragment-scoped builds supported as well (:270); the injection points IvfBuildParams.centroids and PQBuildParams.codebook both exist. The only genuine core-side gaps are the cancellation hook and per-segment listing stats — the rest is FFI exposure.

This does not presume the Q3 answer: if round-3 prefers a different tracking form, the upstream issue adapts. Round-3 feedback welcome whenever convenient — no urgency imposed by this step.

[Gabriel39]

Thanks for starting the upstream discussion. The Rust-core findings are useful for future distributed-index work, and lance-format/lance-c#55 can remain as that future track.

However, this does not change the confirmed scope for Doris 4.2, and Q3 is no longer open: lance-c extensions are not a prerequisite for #66497 in 4.2. Features that depend on those extensions must be deferred. Therefore, Wave 1 should not be described as blocking the 4.2 implementation or as de-risking its current schedule; it is outside the release's critical path.

For #66497, the next design revision should be simplified around the existing branch-4.1 baseline and the pinned lance-c 0.1.2 one-shot APIs:

• lance_dataset_create_vector_index and lance_dataset_create_scalar_index;
• full rebuild/replacement through the existing replace=true path;
• lance_dataset_index_count and lance_dataset_index_list_json for the currently available metadata;
• lance_dataset_drop_index;
• execution of the heavy direct-Dataset call on a selected BE for Directory Catalog.

The 4.2 design should explicitly exclude fragment-scoped distributed builds, shared-model distribution, BE-to-FE uncommitted segment transfer, coordinator segment commit, incremental BUILD INDEX, physical merge, native progress, cooperative cancellation, and per-segment coverage/statistics extensions. It should also avoid promising reliable CANCEL BUILD INDEX semantics when the underlying one-shot native call cannot be interrupted.

The REST path is separate: lance-c additions do not expand REST Catalog capabilities. REST index lifecycle operations must continue to use the pinned Namespace request/response models and the server's implemented capabilities. In particular, newer lance-c APIs would not provide REST OR REPLACE, incremental build, cancellation, or additional request parameters.

One future-facing note for lance-c#55: pb::IndexMetadata bytes are not directly accepted by the pinned Java SDK. Index.Builder takes the metadata as individual fields and exposes no public parseFrom(byte[]) entry point, so the cross-SDK design still needs an explicit Java decoder/adapter and a version-compatibility contract. This does not need to be solved for #66497 because that distributed path is out of scope for 4.2.

Please make the next revision a 4.2 design based only on the existing APIs, covering BE selection/execution, supported index types and properties, replace=true semantics, privileges, concurrency, versioning, failure behavior, metadata refresh, local-path visibility, and the restricted Namespace 0.7.7 REST capability matrix. Query-side work should remain limited to validating index consumption through the existing vector_search() and scalar predicate paths.