Apache Kyuubi(Incubating) 1.4.0-incubating Release Notes

[turboFei]

In this release, Kyuubi supports the latest Spark 3.2, Z-Order to optimize sql, long running SQL engine without principal and keytab provided, mysql protocol frontend and mysql client. Other major updates includes GROUP engine.share.level support, asynchronous opening session, user-friendly kyuubi beeline and zookeeper authentication enhancement.

Highlight

Fully verified with the latest Spark 3.2

Support Z-Order to optimize SQL

Support long running SQL engine without principal and keytab provided

Support GROUP engine share level

Enhance USER engine share level support engine pool

Enhance zookeeper authentication and ACLs

Add more Kyuubi defined functions

Fully tested with all data lake platform Delta Lake, Iceberg and Hudi

Support both KERBEROS and PLAIN authentication at the same time

Support MySQL protocol frontend and MySQL client(experimental)

Support to launch query engine asynchronously during opening session and user-friendly Kyuubi Hive JDBC and BeeLine

Functionality

[KYUUBI #1225][KYUUBI #1228] Add kyuubi-extension-spark-3-2 module and profile to support latest spark-3.2

[KYUUBI #939] Add Z-Order extensions to support optimize SQL

[KYUUBI #913] Support long running SQL engine without principal and keytab provided

[KYUUBI #1219] Support MySQL Protocol Frontend

[KYUUBI #1349] Add new conf kyuubi.frontend.protocols

[KYUUBI #1206] Support GROUP engine share level

[KYUUBI #1018] Set spark.sql.execution.topKSortFallbackThreshold to 10000

[KYUUBI #1059] Add Plan Only Operations

[KYUUBI #1039] Application stop log redirect append to engineLog

[KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page

[KYUUBI #1198] Support to incremental collect statement result

[KYUUBI #1248] Change kyuubi.engine.session.initialize.sql default value to Nil

[KYUUBI #1286] Increase kyuubi.session.engine.initialize.timeout default to 3min

[KYUUBI #1323] Introduce EngineType to distinguish multiple engines

[KYUUBI #1262] Support both KERBEROS and PLAIN authentication at the same time

[KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides

[KYUUBI #1204] Enhance zookeeper authentication and ACLs

[KYUUBI #1039] Application stop log redirect append to engineLog

[KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page

Integration

[KYUUBI #703] Integrate test with Hudi 0.9.0

[KYUUBI #1338] Integrate test with Iceberg 0.12.1

Hive JDBC

[KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver

[KYUUBI #1346] Support launch query engine asynchronously during opening session

[KYUUBI #1377] Support to get launch engine log asynchronously with KyuubiConnection

[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement

Others

[KYUUBI #1047] Use deployment to run kyuubi server HA on K8s

[KYUUBI #1223] Remove dead module kyuubi-spark-monitor

[KYUUBI #1277] Add UDF engine_id

[KYUUBI #660] Add UDF session_user

Credits

Last but not least, this release would not have been possible without the following contributors: AnybodyHome, Brian Yue, Cheng Pan, Fei Wang, Fu Chen, Jagadesh Adireddi, Kent Yao, Min Zhao, Guoqing Yang, h, happytofly, hongdongdong, jiaoqingbo, qiuliang, qphien, senmiaoliu, simon, sunfangbin, sychen, timothy65535, ulysses-you, wFoget, xiebo03, xiongyinke, yanghua, yanyu34946, zhang1002, zhenjiaguo, zhouyifan279, zwangsheng, 张宇翔