How to use kyuubi(kyuubi-1.2.0-bin-spark-3.1-hadoop3.2) with delta-lake (Azure blob storage)

[zhichaoleo]

Hi Team.
We are trying to use kyuubi(kyuubi-1.2.0-bin-spark-3.1-hadoop3.2) with delta-lake (Azure blob storage). But I can't find any documents to set up related configurations on kyuubi offical doc site or google search.
My kyuubi-defaults.conf:

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
fs.azure.account.key..blob.core.windows.net=

But it doesn't work as expected. Am I configure kyuubi right? Is there any docs and examples about how to use kyuubi with delta-lake (Azure blob storage) ?
Thanks!

[pan3793]

1.2.0 is not the latest version, we move the release to the https://kyuubi.apache.org/releases.html after entering apache incubator.

For delta integration, please follow the delta official doc https://docs.delta.io/latest/quick-start.html to setup detla jar and spark confs, the basic setup steps should be:

1. Download delta jar, and put the jar into $SPARK_HOME/jars/
2. Modify $SPARK_HOME/conf/spark-defaults.conf and put the following items.

spark.sql.extensions o.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
# azure confs ...

3. Use spark-shell to verify it works.
4. Deploy Kyuubi
5. Use client (e.g. beeline) to connect Kyuubi and run SQL to opreate delta tables.

For deployment, we recommand persist the required confs into $SPARK_HOME/conf/spark-defaults.conf, which will take effect for spark-submit, spark-shell and Kyuubi, and overwrite those confs specfic to Kyuubi in $KYUUBI_HOME/conf/kyuubi-defaults.conf.

[yaooqinn]

Hi @zhichaoleo thank you try Apache Kyuubi and the discussion here looks amazing!

As you mentioned that

But I can't find any documents to set up related configurations on kyuubi offical doc site or google search.

So, can you help us build a doc for integrating with delta lake?

[zhichaoleo]

Hi @yaooqinn,
Sure. I think I can write down the steps. But I don't know how to make this contribution. Could you show me the steps to build this doc?
Thanks

[yaooqinn]

You can fork this repo and clone it to your local machine, then you can open it in any IDE or editor software you like.

The newly add doc should be written in markdown form and docs/integrations might be a good place to put it.

The content of your article can be how to integrate Kyuubi and delta lake step by step.

There is also an example for kudu integration in https://github.com/apache/incubator-kyuubi/tree/master/docs/integrations

FYi, it's also easy to verify how the doc looks like in local by https://kyuubi.apache.org/docs/latest/develop_tools/build_document.html

[zhichaoleo]

Great. I will write this doc and make a PR. Hope you and team members can give feedback and suggestions.
Kyuubi is a great tool, and I wish it will get better and better.

[yaooqinn]

thanks @zhichaoleo