Hi,
I have cloned the code from spark-agent repo and packaged the jar locally without any modifications (used Spark 2.4 bundle), and installed jar on databricks cluster.
As soon as I run any command , its failing with following error. After googling about this, found that it will occur when we use incompatible versions for (io.delta -2.11) and scala (2.12). Can you please check this if any of the dependencies referring to wrong scala version. I think databricks by default has (io.delta -2.11) based on cluster config.
Databricks cluster config: Spark 2.4.5 and Scala 2.11 (using same config, I was able to capture lineage with the already packaged jars provided by you)
com.google.common.util.concurrent.ExecutionError: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.delta.sources.DeltaDataSource could not be instantiated
If I uninstall jar from cluster, all works well.
PS: I was able to capture lineage from the pre built artifacts provided by you in Maven repo. I am facing this error when I packaged the jar locally.
Hi,
I have cloned the code from spark-agent repo and packaged the jar locally without any modifications (used Spark 2.4 bundle), and installed jar on databricks cluster.
As soon as I run any command , its failing with following error. After googling about this, found that it will occur when we use incompatible versions for (io.delta -2.11) and scala (2.12). Can you please check this if any of the dependencies referring to wrong scala version. I think databricks by default has (io.delta -2.11) based on cluster config.
Databricks cluster config: Spark 2.4.5 and Scala 2.11 (using same config, I was able to capture lineage with the already packaged jars provided by you)
com.google.common.util.concurrent.ExecutionError: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.delta.sources.DeltaDataSource could not be instantiated
If I uninstall jar from cluster, all works well.
PS: I was able to capture lineage from the pre built artifacts provided by you in Maven repo. I am facing this error when I packaged the jar locally.