Unity Catalog Support

[dhirschfeld]

I'm opening this to discuss the state of Unity Catalog support in delta-rs.

In my ideal world I'd be able get a DeltaTable object from Unity Catalog, with permissions controlled by UC access policies.

[dhirschfeld]

xref:

• fix: remove the get_data_catalog() function #1941

[dhirschfeld]

Extract the session credentials from your cluster

• https://kb.databricks.com/en_US/notebooks/access-s3-temp-session-token

[rtyler]

I have had a number of discussions with Databricks on the topic. The 🦀 code has support for table lookup but it's kind of useless IMHO without credential vending support from the Unity APIs. Basically a user today would need to have an access token for Unity and their own separate storage credentials for accessing the table in object storage.

What is missing is a public API from Databricks which would allow the actor running delta-rs to retrieve credentials for use in accessing storage.

So yes, I would like to see this too, but I'm marking this on-hold until Databricks provides us the necessary REST APIs for Unity Catalog

[dhirschfeld]

Thanks for the info @rtyler. I'll continue to monitor this thread in the hope that something changes on the Databricks side...

[wmoreiraa]

Hi sir, @rtyler. Did Databricks gave any updates on the last months on this topic? I think we'all on databricks ecosystem are waiting for this.

[nicornk]

https://github.com/duckdb/uc_catalog/blob/4a2ba143eb8686becf69f759caa731742c2d7c81/src/uc_api.cpp#L120

There seems to be an API now to get temporary storage credentials?

[ion-elgreco]

Those APIs aren't public yet if you check the docs. But it's promising

[ion-elgreco]

Actually they seems already to be working : ) I'll start working on it soon ^^

[nicornk]

I also found zero google results on the APIs, just wanted to share this here.

[ion-elgreco]

No luck:

{'error_code': 'UNAUTHENTICATED',
 'message': "Request to generate access credential for table '<redacted>' from outside of Databricks Unity Catalog enabled compute environment is denied for security. Please contact Databricks support for integrations with Unity Catalog.",
 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo',
   'reason': 'UNITY_CATALOG_EXTERNAL_GENERATE_TABLE_CREDENTIALS_DENIED',
   'domain': 'unity-catalog.databricks.com',
   'metadata': {'tableId': '<redacted>'}},
  {'@type': 'type.googleapis.com/google.rpc.RequestInfo',
   'request_id': '<redacted>',
   'serving_data': ''}]}

[abhiaagarwal]

Unity Catalog open-source is available:
https://github.com/unitycatalog/unitycatalog/

[ion-elgreco]

Yup, but we likely only can support the OSS version and not databricks unity

[abhiaagarwal]

I didn't realize there was a difference :(

[dhirschfeld]

An interesting blog-post from DuckDB:

• https://duckdb.org/2024/06/10/delta.html

They're working with Databricks to implement native Delta Lake support in DuckDB.

It seems very cool but the examples are all using direct access to a local filesystem or S3. IIUC DuckDB still wouldn't have access to Unity Catalog managed tables.

[ion-elgreco]

@abhiaagarwal see this comment: #2066 (reply in thread)

I already gave the new API credentials fetching api a go, but it didn't allow me to fetch them

[roeap]

since unity is now oss and should have some features that were holding us back in adpoting this, i am looking to soon spend some time on reviving our integration - unless someone else is faster :)

[ion-elgreco]

There probably isn't much value to it though as most people want databricks UC and not UC OSS, as of now the databricks UC APIs are not working for credentials fetching

[roeap]

they are the same. the reason we may be unblocked is more realted to some features that were associated with teh release as well.

[roeap]

the api surface is nove much clearer (and better documented) and fully accessible to clients.

[ion-elgreco]

They are the same indeed, but the main thing we need is fetching credentials, you can see a couple posts above that I already tried this, but it wasn't working unfortunately.

[roeap]

I'll try anyhow 😀