Performance affect on spark workload for using spline.

[rupesh3020]

Hi @wajda ,

Do we have any benchmark or data which shows what is affect of spline on spark workloads?
Does it increase the execution time?

If yes what factors should we keep in mind. I was under the impression that it will not have any noticable impact on performance.

[wajda]

No, Spline agent does not interfere with the Spark code execution, so it in principle cannot affect the main execution time of data processing. The only time it can affect though is the time of the spark driver running time. Spline agent works as a Spark driver listener. It adds small constant time overhead to initialise itself, and when the main job finishes (and the output data is written) it again takes some little time to collect the lineage metadata and send it out to the destination point (via the lineage dispatcher). That time overhead depends on the complexity of the execution plan and the amount of collected metadata. But usually we are talking about really small time delay (fractions of seconds to a few seconds for any sensible Spark job) that in comparison with the main data processing time is negligible and does not have any correlation to the amount of data being process (so is constant from that perspective).

[rupesh3020]

What if the lineage info in json format is 95 mb?

[wajda]

Ouch! Then the delay and memory consumption might not be negligible :) At least it is constant and doesn't grow with the amount of the data being processed. It only depends on the complexity of the job logic and the complexity and size of the schema.

Your 95 mb, is it zipped or plain text? Is it due to a large amount of operations or attributes?