Not able to see UI lineage log from databricks job.

[tariniroutey]

Hi Team,

We have databricks as our compute we want to see the lineage of each job running in databricks .I followed each process mentioned in these below mentioned URL
https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-databricks

Configured the databricks cluster instructed in the above URL

spark.conf.set("spark.spline.lineageDispatcher", "http")
log4j-active.txt

spark.conf.set("spark.spline.lineageDispatcher.http.producer.url", "http://*************:8080/producer")
spark.conf.set("spline.mode","REQUIRED")

Deployed these dockers image in separate vm and able to access Rest Gateway and Data lineage web UI also.
Docker URL for the above components : https://github.com/AbsaOSS/spline-getting-started/tree/main/docker

After triggering the job I am not able to see the lineage. Attaching the execution log for reference.

Appreciate your help and guidance.

Thank You.

[wajda]

A minor correction - when setting Spline properties through the Spark config they must be prefixed with spark., otherwise they will be ignored by Spark. E.g. spark.conf.set("spark.spline.mode","REQUIRED"). But that's not related to your main question.
As for why you don't see any lineage please read the answer here - AbsaOSS/spline-spark-agent#394. If it doesn't help, open an issue and provide an example of the Spark job. We'll take a look.

[tariniroutey]

HI @wajda
Thank You very much It started working I am able to generate the lineage. I created a automated cluster and attached the library and it works like charm..

System.setProperty("spline.mode", "REQUIRED")
System.setProperty("spark.sql.queryExecutionListeners", "za.co.absa.spline.harvester.listener.SplineQueryExecutionListener")
System.setProperty("spline.producer.url","http://***.***.***.***:8080/producer")

import za.co.absa.spline.harvester.SparkLineageInitializer._
spark.enableLineageTracking()

We need one info the XSD of the json output of the lineage to understand it better so that we can integrate it in our Webapp. Would be great if you can provide some insight to start with. Thank you very much for your incredible support this really help @wajda.

[wajda]

JSON Schema is on our todo list. Meantime take a look at the Swagger definitions that you can find on the Spline Server index page - http://***.***.***.***:8080. If you have any specific question don't hesitate to ask, preferably in another discussion ticket to not mix different topics in one thread.