HDFS factory not working.

[larryxia-ddl]

I found za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory not working, because this line will throw exception.

override def createDataLineageReader(): DataLineageReader = throw new UnsupportedOperationException

even I bypass this issue by comment out call in DatalineageListener, I found the hdfs output is not correct, there is always only one json record export to the log file, it seem it was overwrite every time.

It seems this feature is not ready yet. please advise.

[wajda]

Yes, persistence to HDFS is experimental and isn't tested well. A lineage file is rewritten as intended. The original idea for HDFS persistence was to store a lineage file next to the parquet file containing the data. So that you always have data and its lineage in one place, easy to access. When the data file is rewritten, the lineage file is also rewritten respectively. We didn't consider appending data use case.
HDFS persistor didn't received enough attention recently. If you have ides how to improve it, we will appreciate a PR.

[Roshni1988]

Hi larryxia-ddl ,

Were you able to start the UI, when you are trying to use the data saved from HDFS?

I have my data stored in HDFS, but i am not able to start the spline UI using the below command

java -jar spline-web-0.2.5-exec-war.jar -Dspline.hdfs.file.name=hdfs://quickstart.cloudera:8020/user/root/spline -Dspline.hdfs.file.permissions=777

Am I Doing something wrong here.. Kindly Help as this is Little urgent for me.

Below is the logs i get and it's get stuck

[root@quickstart spline_jars]# java -jar spline-web-0.2.5-exec-war.jar -Dspline.hdfs.file.name=hdfs://quickstart.cloudera:8020/user/root/spline -Dspline.hdfs.file.permissions=777 -httpPort 1234
Jan 17, 2018 5:59:03 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-1234"]
Jan 17, 2018 5:59:03 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat
Jan 17, 2018 5:59:03 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.37
Jan 17, 2018 5:59:12 AM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Jan 17, 2018 5:59:12 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-1234"]

and while opening the localhost:1234 in the browser i get page not found issue.

[wajda]

Hi Roshni1988,
You are using incorrect command line to start Spline UI. It's strange that you don't see any error message in the console as they should be self explanatory. Currently Spline UI is hard coded to use MongoDB storage, and it requires the following two JVM props to be provided: spline.mongodb.url and spline.mongodb.name (see https://absaoss.github.io/spline/#web-ui-application)

Properties spline.hdfs.file.name and spline.hdfs.file.permissions that you provided are inapplicable here and have no effect. That are used in Spline core inside your tracked Spark application when HDFS factory is selected for persistence: spline.persistence.factory=za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory

But let me make it clear once again - at the moment HDFS persistence is write-only and serves more as a linage log rather than a reusable lineage storage. It will be improved in the future Spline versions.

[sergiobanos]

Hi,

I have being looking Spline and seems really nice. When I try to persist on HDFS, I am facing the following problem as larryxia-ddl ( details below )

Spline.properties

spline.persistence.factory=za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory
spline.hdfs.file.name= hdfs://10.0.2.15:8020/user/cloudera/OutputLineage.txt
spline.hdfs.file.permissions=777

Question1: Is there something incorrect in the above settings that prevent me to see the hdfs output?
Question 2: Is there a planned release where this issue can be addressed?. Is there a workaround I can make to make it work?

Help is appreciated -

Thanks very much for your contribution!

Details:

Exception in thread "main" java.lang.UnsupportedOperationException
at za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory.createDataLineageReader(HdfsPersistenceFactory.scala:61)
at za.co.absa.spline.core.DataLineageListener.(DataLineageListener.scala:43)
at za.co.absa.spline.core.SparkLineageInitializer$SparkSessionWrapper.attemptInitialization(SparkLineageInitializer.scala:72)
at za.co.absa.spline.core.SparkLineageInitializer$SparkSessionWrapper.liftedTree1$1(SparkLineageInitializer.scala:58)
at za.co.absa.spline.core.SparkLineageInitializer$SparkSessionWrapper.enableLineageTracking(SparkLineageInitializer.scala:57)

[wajda]

Hm, I was under impression that this bug has been fixed a long time ago. Sorry about that. I will take a look and push a fix asap. Most likely the fix has been made in our internal branch and has yet to be published on github.

[wajda]

Please try 0.2.6 release, should be working now.

[sergiobanos]

Working perfectly. Thanks Wajda!