add read only support for Delta table

[djouallah]

something like this will be nice

ctx_bend.sql(f"""CREATE view IF NOT EXISTS {tbl} AS SELECT * FROM 'fs:///Tables/{tbl}/' (pattern => '.*.Delta' ) """).collect()

[BohuTANG]

Yes, this needs add a delta format like parquet.

[djouallah]

@BohuTANG does databend support scanning arrow dataset as an alternative ?

[BohuTANG]

@BohuTANG does databend support scanning arrow dataset as an alternative ?

No, what's the use case under that?

[djouallah]

this is how I am querying delta table using datafusion

import datafusion
from deltalake import DeltaTable
ctx = datafusion.SessionContext()
for tbl in ['nation','region','customer','supplier','lineitem','orders','partsupp','part'] :
 ctx.register_dataset(tbl, DeltaTable(f'{sf}/{tbl}').to_pyarrow_dataset())
datafusion=execute_query(ctx, sql)

[BohuTANG]

The better way would be to natively support Delta using native-rs. We're mapping out plans for it.

[djouallah]

any update on this ?

[sundy-li]

I think we already support.

Test examples: https://github.com/datafuselabs/databend/blob/12a9186b78780fcb6d79be1b4d2aa4f9856f78c2/tests/suites/1_stateful/12_delta/11_0000_delta_engine.sh

Documents: https://docs.databend.com/guides/access-data-lake/delta

[djouallah]

Thank you