[#8838] feat(catalogs): Support create/load/list table operation for lance table. by yuqi1129 · Pull Request #8879 · apache/gravitino

yuqi1129 · 2025-10-22T12:40:29Z

What changes were proposed in this pull request?

Add support create and load table operations for lance table.

Why are the changes needed?

It's a need.

Fix: #8838
Fix: #8837

Does this PR introduce any user-facing change?

N/A.

How was this patch tested?

Currently, I have only tested it locally

➜  [/Users/yuqi/Downloads] curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "lance_table14",
  "comment": "This is an example table",
  "columns": [
    {
      "name": "id",
      "type": "integer",
      "comment": "id column comment",
      "nullable": false,
      "autoIncrement": true,
      "defaultValue": {
        "type": "literal",
        "dataType": "integer",
        "value": "-1"
      }
    }
  ],
  "indexes": [
    {
      "indexType": "primary_key",
      "name": "PRIMARY",
      "fieldNames": [["id"]]
    }
  ],
  "properties": {
    "format": "lance",
    "location": "/tmp/lance_catalog/schema/lance_table14"
  }
}' http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables
{"code":0,"table":{"name":"lance_table14","comment":"This is an example table","columns":[{"name":"id","type":"integer","comment":"id column comment","nullable":false,"autoIncrement":true,"defaultValue":{"type":"literal","dataType":"integer","value":"-1"}}],"properties":{"format":"lance","location":"/tmp/lance_catalog/schema/lance_table14/"},"audit":{"creator":"anonymous","createTime":"2025-10-23T03:18:39.123151Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[{"indexType":"PRIMARY_KEY","name":"PRIMARY","fieldNames":[["id"]]}]}}
➜  [/Users/yuqi/Downloads] curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables/lance_table14
{"code":0,"table":{"name":"lance_table14","comment":"This is an example table","columns":[{"name":"id","type":"integer","comment":"id column comment","nullable":false,"autoIncrement":false,"defaultValue":{"type":"literal","dataType":"integer","value":"-1"}}],"properties":{"format":"lance","location":"/tmp/lance_catalog/schema/lance_table14/"},"audit":{"creator":"anonymous","createTime":"2025-10-23T03:18:39.123151Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[{"indexType":"PRIMARY_KEY","name":"PRIMARY","fieldNames":[["id"]]}]}}            ➜  [/Users/yuqi/Downloads] curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables
{"code":0,"identifiers":[{"namespace":["test","lance_catalog","schema"],"name":"lance_table10"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table11"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table12"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table13"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table14"}]}                      
➜  [/Users/yuqi/Downloads]

And the lance location

➜  [/tmp/lance_catalog/schema] ls
lance_table10 lance_table11 lance_table12 lance_table13 lance_table14
➜  [/tmp/lance_catalog/schema] cd lance_table14
➜  [/tmp/lance_catalog/schema/lance_table14] ls -al
total 0
drwxr-xr-x@ 4 yuqi  wheel  128 10 23 11:18 .
drwxr-xr-x@ 7 yuqi  wheel  224 10 23 11:18 ..
drwxr-xr-x@ 3 yuqi  wheel   96 10 23 11:18 _transactions
drwxr-xr-x@ 3 yuqi  wheel   96 10 23 11:18 _versions
➜  [/tmp/lance_catalog/schema/lance_table14] ls -al _versions
total 8
drwxr-xr-x@ 3 yuqi  wheel   96 10 23 11:18 .
drwxr-xr-x@ 4 yuqi  wheel  128 10 23 11:18 ..
-rw-r--r--@ 1 yuqi  wheel  225 10 23 11:18 1.manifest
➜  [/tmp/lance_catalog/schema/lance_table14]

yuqi1129 · 2025-10-22T13:08:44Z

#8847 should be merged first.

…tino into issue_8838

mchades · 2025-10-23T10:10:23Z

...house/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceCatalogOperations.java

+    }
+  }
+
+  private org.apache.arrow.vector.types.pojo.Schema convertColumnsToSchema(Column[] columns) {


Why use a fully qualified name?

We already have classes Schema in Gravitino

mchades · 2025-10-23T10:17:07Z

core/src/main/java/org/apache/gravitino/catalog/TableOperationDispatcher.java

+      } catch (IOException ioe) {
+        throw new RuntimeException("Failed to load table entity " + ident, ioe);
+      }
+    }


I think you should place the table operations of GenericLakehouseCatalog in the corresponding catalog, rather than scattering them in TableOperationDispatcher.

Let me check, there should be many places that need to be adjusted

Check how ManagedSchemaOperations is implemented, don't do lots of special if..else check.

jerryshao · 2025-10-24T06:59:54Z

api/src/main/java/org/apache/gravitino/rel/indexes/Indexes.java

        .build();
  }

+  /** Custom JSON serializer for Index objects. */


It's not proper to add the JSON serde in the API module. Should if be in the class like IndexDTO, and then convert the IndexDTO to the IndexImpl here?

jerryshao · 2025-10-24T07:25:18Z

api/src/main/java/org/apache/gravitino/rel/GenericTable.java

+  default boolean external() {
+    return false;
+  }
+}


We can extend the current table interface, it's not good to add a new interface.

jerryshao · 2025-10-24T07:31:01Z

...house/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceCatalogOperations.java

+  public NameIdentifier[] listTables(Namespace namespace) throws NoSuchSchemaException {
+    return new NameIdentifier[0];
+  }
+
+  @Override
+  public Table loadTable(NameIdentifier ident) throws NoSuchTableException {
+    // Should not come here.
+    return null;
+  }


If this will not be happened, we'd better throw an exception here, to avoid an unexpected call.

jerryshao · 2025-10-24T07:32:57Z

...house/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceCatalogOperations.java

+      return builder
+          .withName(ident.name())
+          .withColumns(columns)
+          .withComment(comment)
+          .withProperties(properties)
+          .withDistribution(distribution)
+          .withIndexes(indexes)
+          .withAuditInfo(
+              AuditInfo.builder()
+                  .withCreator(PrincipalUtils.getCurrentUserName())
+                  .withCreateTime(Instant.now())
+                  .build())
+          .withPartitioning(partitions)
+          .withSortOrders(sortOrders)
+          .withFormat("lance")
+          .build();


We can move this to the general table operation, and in this class we could mainly focus on the lance related operations.

jerryshao · 2025-10-24T07:37:31Z

catalogs/catalog-generic-lakehouse/build.gradle.kts

@@ -43,6 +43,7 @@ dependencies {
  implementation(libs.commons.lang3)
  implementation(libs.guava)
  implementation(libs.hadoop3.client.api)


I think hadoop3 is not required.

jerryshao · 2025-10-24T11:34:23Z

core/src/main/java/org/apache/gravitino/meta/GenericTableEntity.java

+import org.apache.gravitino.rel.indexes.Index;
+
+@Getter
+public class GenericTableEntity extends TableEntity {


We don't have to create a new GenericTableEntity, we can extend the current TableEntity.

jerryshao · 2025-10-24T11:53:50Z

...house/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceCatalogOperations.java

+                        col.name(),
+                        org.apache.arrow.vector.types.pojo.FieldType.nullable(
+                            converter.fromGravitino(col.dataType())),
+                        null);


What's meaning of null here?

jerryshao · 2025-10-24T12:01:10Z

.../src/main/java/org/apache/gravitino/catalog/lakehouse/GenericLakehouseCatalogOperations.java

+    LakehouseCatalogOperations lakehouseCatalogOperations =
+        SUPPORTED_FORMATS.compute(
+            format,
+            (k, v) ->
+                v == null
+                    ? createLakehouseCatalogOperations(
+                        format, properties, catalogInfo, propertiesMetadata)
+                    : v);


This map seems have threading issue if the request coming in concurrently.

jerryshao · 2025-10-24T12:11:52Z

.../src/main/java/org/apache/gravitino/catalog/lakehouse/GenericLakehouseCatalogOperations.java

+      HasPropertyMetadata propertiesMetadata) {
+    LakehouseCatalogOperations operations;
+    switch (format.toLowerCase()) {
+      case "lance":


We'd better not hardcode "lance" here and above, using a constant and enum value is better.

jerryshao · 2025-10-24T12:13:36Z

...kehouse/src/main/java/org/apache/gravitino/catalog/lakehouse/LakehouseCatalogOperations.java

+import org.apache.gravitino.connector.CatalogOperations;
+import org.apache.gravitino.rel.TableCatalog;
+
+public interface LakehouseCatalogOperations extends CatalogOperations, TableCatalog {}


What's the difference between LakehouseCatalogOperations and GenericLakehouseCatalogOperations?

### What changes were proposed in this pull request? This PR trys to resolve the comments that have not been addressed in #8879 ### Why are the changes needed? It's an improvement. Fix: #8915 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Test locally, and we will add ITs in #8921

…n for lance table. (apache#8879) ### What changes were proposed in this pull request? Add support create and load table operations for lance table. ### Why are the changes needed? It's a need. Fix: apache#8838 Fix: apache#8837 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Currently, I have only tested it locally ```shell ➜ [/Users/yuqi/Downloads] curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ "name": "lance_table14", "comment": "This is an example table", "columns": [ { "name": "id", "type": "integer", "comment": "id column comment", "nullable": false, "autoIncrement": true, "defaultValue": { "type": "literal", "dataType": "integer", "value": "-1" } } ], "indexes": [ { "indexType": "primary_key", "name": "PRIMARY", "fieldNames": [["id"]] } ], "properties": { "format": "lance", "location": "/tmp/lance_catalog/schema/lance_table14" } }' http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables {"code":0,"table":{"name":"lance_table14","comment":"This is an example table","columns":[{"name":"id","type":"integer","comment":"id column comment","nullable":false,"autoIncrement":true,"defaultValue":{"type":"literal","dataType":"integer","value":"-1"}}],"properties":{"format":"lance","location":"/tmp/lance_catalog/schema/lance_table14/"},"audit":{"creator":"anonymous","createTime":"2025-10-23T03:18:39.123151Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[{"indexType":"PRIMARY_KEY","name":"PRIMARY","fieldNames":[["id"]]}]}} ➜ [/Users/yuqi/Downloads] curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables/lance_table14 {"code":0,"table":{"name":"lance_table14","comment":"This is an example table","columns":[{"name":"id","type":"integer","comment":"id column comment","nullable":false,"autoIncrement":false,"defaultValue":{"type":"literal","dataType":"integer","value":"-1"}}],"properties":{"format":"lance","location":"/tmp/lance_catalog/schema/lance_table14/"},"audit":{"creator":"anonymous","createTime":"2025-10-23T03:18:39.123151Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[{"indexType":"PRIMARY_KEY","name":"PRIMARY","fieldNames":[["id"]]}]}} ➜ [/Users/yuqi/Downloads] curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables {"code":0,"identifiers":[{"namespace":["test","lance_catalog","schema"],"name":"lance_table10"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table11"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table12"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table13"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table14"}]} ➜ [/Users/yuqi/Downloads] ``` And the lance location ```shell ➜ [/tmp/lance_catalog/schema] ls lance_table10 lance_table11 lance_table12 lance_table13 lance_table14 ➜ [/tmp/lance_catalog/schema] cd lance_table14 ➜ [/tmp/lance_catalog/schema/lance_table14] ls -al total 0 drwxr-xr-x@ 4 yuqi wheel 128 10 23 11:18 . drwxr-xr-x@ 7 yuqi wheel 224 10 23 11:18 .. drwxr-xr-x@ 3 yuqi wheel 96 10 23 11:18 _transactions drwxr-xr-x@ 3 yuqi wheel 96 10 23 11:18 _versions ➜ [/tmp/lance_catalog/schema/lance_table14] ls -al _versions total 8 drwxr-xr-x@ 3 yuqi wheel 96 10 23 11:18 . drwxr-xr-x@ 4 yuqi wheel 128 10 23 11:18 .. -rw-r--r--@ 1 yuqi wheel 225 10 23 11:18 1.manifest ➜ [/tmp/lance_catalog/schema/lance_table14] ```

…pache#8922) ### What changes were proposed in this pull request? This PR trys to resolve the comments that have not been addressed in apache#8879 ### Why are the changes needed? It's an improvement. Fix: apache#8915 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Test locally, and we will add ITs in apache#8921

…n for lance table. (apache#8879) Add support create and load table operations for lance table. It's a need. Fix: apache#8838 Fix: apache#8837 N/A. Currently, I have only tested it locally ```shell ➜ [/Users/yuqi/Downloads] curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ "name": "lance_table14", "comment": "This is an example table", "columns": [ { "name": "id", "type": "integer", "comment": "id column comment", "nullable": false, "autoIncrement": true, "defaultValue": { "type": "literal", "dataType": "integer", "value": "-1" } } ], "indexes": [ { "indexType": "primary_key", "name": "PRIMARY", "fieldNames": [["id"]] } ], "properties": { "format": "lance", "location": "/tmp/lance_catalog/schema/lance_table14" } }' http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables {"code":0,"table":{"name":"lance_table14","comment":"This is an example table","columns":[{"name":"id","type":"integer","comment":"id column comment","nullable":false,"autoIncrement":true,"defaultValue":{"type":"literal","dataType":"integer","value":"-1"}}],"properties":{"format":"lance","location":"/tmp/lance_catalog/schema/lance_table14/"},"audit":{"creator":"anonymous","createTime":"2025-10-23T03:18:39.123151Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[{"indexType":"PRIMARY_KEY","name":"PRIMARY","fieldNames":[["id"]]}]}} ➜ [/Users/yuqi/Downloads] curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables/lance_table14 {"code":0,"table":{"name":"lance_table14","comment":"This is an example table","columns":[{"name":"id","type":"integer","comment":"id column comment","nullable":false,"autoIncrement":false,"defaultValue":{"type":"literal","dataType":"integer","value":"-1"}}],"properties":{"format":"lance","location":"/tmp/lance_catalog/schema/lance_table14/"},"audit":{"creator":"anonymous","createTime":"2025-10-23T03:18:39.123151Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[{"indexType":"PRIMARY_KEY","name":"PRIMARY","fieldNames":[["id"]]}]}} ➜ [/Users/yuqi/Downloads] curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" http://localhost:8090/api/metalakes/test/catalogs/lance_catalog/schemas/schema/tables {"code":0,"identifiers":[{"namespace":["test","lance_catalog","schema"],"name":"lance_table10"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table11"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table12"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table13"},{"namespace":["test","lance_catalog","schema"],"name":"lance_table14"}]} ➜ [/Users/yuqi/Downloads] ``` And the lance location ```shell ➜ [/tmp/lance_catalog/schema] ls lance_table10 lance_table11 lance_table12 lance_table13 lance_table14 ➜ [/tmp/lance_catalog/schema] cd lance_table14 ➜ [/tmp/lance_catalog/schema/lance_table14] ls -al total 0 drwxr-xr-x@ 4 yuqi wheel 128 10 23 11:18 . drwxr-xr-x@ 7 yuqi wheel 224 10 23 11:18 .. drwxr-xr-x@ 3 yuqi wheel 96 10 23 11:18 _transactions drwxr-xr-x@ 3 yuqi wheel 96 10 23 11:18 _versions ➜ [/tmp/lance_catalog/schema/lance_table14] ls -al _versions total 8 drwxr-xr-x@ 3 yuqi wheel 96 10 23 11:18 . drwxr-xr-x@ 4 yuqi wheel 128 10 23 11:18 .. -rw-r--r--@ 1 yuqi wheel 225 10 23 11:18 1.manifest ➜ [/tmp/lance_catalog/schema/lance_table14] ```

…pache#8922) This PR trys to resolve the comments that have not been addressed in It's an improvement. Fix: apache#8915 N/A. Test locally, and we will add ITs in apache#8921

yuqi1129 added 2 commits October 22, 2025 20:37

Support create/load table operations for lance

f2de31b

fix

30921b0

yuqi1129 added 3 commits October 22, 2025 21:45

support list operations.

a52d9fb

Optimize location logic.

95c7c18

Merge branch 'branch-lance-namepspace-dev' of github.com:apache/gravi…

a2b0735

…tino into issue_8838

yuqi1129 changed the title ~~[#8833] feat(catalogs): Support create/load table operation for lance table.~~ [#8833] feat(catalogs): Support create/load/list table operation for lance table. Oct 23, 2025

yuqi1129 changed the title ~~[#8833] feat(catalogs): Support create/load/list table operation for lance table.~~ [#8838] feat(catalogs): Support create/load/list table operation for lance table. Oct 23, 2025

yuqi1129 added 3 commits October 23, 2025 11:36

fix

f0da361

Fix sql format problem

a64ab69

fix

e9fb50d

yuqi1129 requested review from jerryshao and mchades October 23, 2025 09:07

mchades reviewed Oct 23, 2025

View reviewed changes

yuqi1129 mentioned this pull request Oct 23, 2025

[#8839] feat(catalogs): Support add index for lance table #8900

Closed

jerryshao assigned yuqi1129 Oct 24, 2025

jerryshao reviewed Oct 24, 2025

View reviewed changes

jerryshao merged commit d093824 into apache:branch-lance-namepspace-dev Oct 24, 2025
31 of 32 checks passed

yuqi1129 added a commit to yuqi1129/gravitino that referenced this pull request Oct 27, 2025

Polish code for apache#8879

3fce8cd

yuqi1129 mentioned this pull request Oct 27, 2025

[#8915] improvment(catalogs): Polish code for PR #8879 #8922

Merged

Conversation

yuqi1129 commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yuqi1129 commented Oct 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryshao Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuqi1129 commented Oct 22, 2025 •

edited

Loading

jerryshao Oct 24, 2025 •

edited

Loading