Enable hyperspace in spark thriftserver #499
-
|
We have server code which connects to spark jdbc(thrift) server to execute sql. We can have batch jobs which can create index for our data lake. But we are not getting how to enable hyperspace for spark thrift server, where we do not have control in initialising sparksession. So question is how to enable hyperspace in spark thriftserver ? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
For now, there's no way to enable hyperspace in spark thriftserver. One thing we can do is creating a SparkSessionExtension for Hyperspace, and support ENABLE/DISABLE HYPERSPACE sql or enable hyperspace for all sessions when the extension is added. For example, in Delta Lake: |
Beta Was this translation helpful? Give feedback.
-
|
@ashokblend We're going to implement a SparkSessionExtension for Hyperspace soon. I think it would be a good addition. Thanks! 😄 |
Beta Was this translation helpful? Give feedback.
For now, there's no way to enable hyperspace in spark thriftserver.
One thing we can do is creating a SparkSessionExtension for Hyperspace, and support ENABLE/DISABLE HYPERSPACE sql or enable hyperspace for all sessions when the extension is added.
For example, in Delta Lake:
https://github.com/delta-io/delta/blob/master/core/src/main/scala/io/delta/sql/DeltaSparkSessionExtension.scala
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension"