Unexpected error using spline agent 0.7 with AWS Glue 3.0

[wajda]

Discussed in #345

^{Originally posted by araujo-joao October 29, 2021}
Hi guys,

I've been using a custom version of the spline agent for collecting data lineage from AWS Glue spark jobs.
Now AWS Glue released a new version that uses spark 3.1 so I decided to try to update the spline agent to 0.7.0, which supports it.

I'm getting an
ERROR SplineQueryExecutionListener: Unexpected error occurred during lineage processing for application ...
and after digging a bit I could retrieve a better error message:
java.lang.NumberFormatException: For input string: "op-0"

After looking at the code, this seems to be related with the idGenerators created on the latest versions of the spline agent, but unfortunately my knowledge of scala isn't sufficient to understand why this is breaking the spline agent.

Could someone give a hint on how to solve or further debug this issue?

Thanks in advance!

Edit: replacing "spline" with "spline agent" since that's the only component used to get this error.

[jaldaadithya]

Guys, could you please tell us how did you pass the listener and producer URLs in the AWS glue job
For us, its not even reflected in the logs that AWS Glue is using spline Listener

[wajda]

hopefully this helps
https://aws.amazon.com/blogs/big-data/build-data-lineage-for-data-lakes-using-aws-glue-amazon-neptune-and-spline/

[aman3389]

you can add job parameters in advanced options

spark.spline.lineageDispatcher.http.producer.url=http://:8080/producer --conf spark.spline.mode=REQUIRED --conf spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener