Question about the delta catalog managed commit

[horizonzy]

When I try to create a table following the demo code.  https://github.com/unitycatalog/unitycatalog/blob/2e4376a7ed6b506315fa8c54de273e48[…]park/src/main/scala/io/unitycatalog/spark/UCSingleCatalog.scala

Step 1. Create a staging table
Step 2. Get the tmp credential via staging table, and commit the 0000.json to the staging table location.
Step3. Create the normal table(External) using the same table name, staging table location.

But at the Step3, it throw an exception.

com.databricks.sdk.core.error.platform.InvalidParameterValue: Input path url 's3://xxxxxxx/unity-catalog/xxxxxx/__unitystorage/catalogs/00216c35-b906-4425-ba63-5ccbeab49acd/tables/fb1bb278-34e3-4cc2-9f4c-736c46d70efa' overlaps with managed storage within 'CreateTable' call. 

Because I try to create a External  table with the managed storage, the External  table location is the staging table location.

But if I try to change the table type to Managed at Step3, it will throw another exception.
com.databricks.sdk.core.error.platform.InvalidParameterValue: Request to create table

'streamnative_workspace.public_default.test__unity__catalog__yan__7' from outside of Databricks Unity Catalog enabled compute environment is denied. Table type must be EXTERNAL to create table externally, instead it is 'MANAGED'.

Anyone can help to check this case? Thanks  (edited)

[horizonzy]

@yili-db Could you help to check it. Thanks.

[yili-db]

@horizonzy Are you trying to create an external table or managed table?

• To create an external table, you should not create staging table. Instead, call generateTemporaryPathCredentials to get credentials to write files, then your step 3 (External).
• To create a managed table, you just need to change the EXTERNAL in step 3 to MANAGED and you are already there. The error you got is from Databricks. You need to have your workspace to turn on a private preview setting for this. This can be done by Databricks support engineer.

[horizonzy]

@yili-db Thanks for you reply.

I am verifying the delta kernel managed catalog commit feature.

To create an external table, you should not create staging table. Instead, call generateTemporaryPathCredentials to get credentials to write files, then your step 3 (External).

For this case, if I use generateTemporaryPathCredentials, I can't get the unTableId, so the managed catalog commit table type must be MANAGED? So the managed catalog commit feature only support MANAGED table, right?

And I have another question, If the user's existing table type is EXTERNAL, how do we upgrade the existing table to managed catalog commit.

[yili-db]

In this case, you will have to create MANAGED table using the MANAGED table option I mentioned earlier. Managed catalog commit feature only supports MANAGED tables.

Due to fundamental differences of how the storage is assigned and managed between EXTERNAL and MANAGED tables, you can not upgrade EXTERNAL table to MANAGED table. You need to create a new MANAGED table then do the data migration.

@yili-db Thanks for you reply.

I am verifying the delta kernel managed catalog commit feature.

> To create an external table, you should not create staging table. Instead, call generateTemporaryPathCredentials to get credentials to write files, then your step 3 (External).

For this case, if I use generateTemporaryPathCredentials, I can't get the unTableId, so the managed catalog commit table type must be MANAGED? So the managed catalog commit feature only support MANAGED table, right?

And I have another question, If the user's existing table type is EXTERNAL, how do we upgrade the existing table to managed catalog commit.

[horizonzy]

Thanks.Help me a lot.