Could add listeners to Databricks Cluster config - spark.sql.queryExecutionListeners za.co.absa.spline.harvester.listener.SplineQueryExecutionListener

[SP-ARUN]

Hi All,

I tried adding the below to the databricks cluster config to avoid adding an additional cell in each notebook, i couldn't add it ()

The one i added in databricks cluster spark config is below : this doesnt work and when added the cluster is not running throwing an error.

spark.sql.queryExecutionListeners za.co.absa.spline.harvester.listener.SplineQueryExecutionListener

The one i'm using in each cell is as below,

%scala
System.setProperty("spark.sql.queryExecutionListeners", "za.co.absa.spline.harvester.listener.SplineQueryExecutionListener")

Could someone help me here ??

[wajda]

This is a known Databricks issue

Please read this post for a workaround