RDD lineage support

[karanglass34]

Currently there is no one easy solution to provide lineage for RDDs, but there are several ways how to provide at lease some of it or define it manually. This ticket will group all of them.

Subtasks:

Task	Ticket
partial RDD support (Write - LogicalPlan, Read - RDD)	#498
RDD Read support
RDD manual lineage metadata enrichment

---

Original message:

I am using the latest spline version. When I enable spline, I see incomplete lineage, my code reads the file from parquet and creates a temporary view, performs some transformations and creates a new view which is then finally written to a parquet. However I am not able to see lineages for my transformations.

Are there any constraints like all the intermediate RDD's should be in memory? or any specific spark config apart from that mentioned in the documentation which needs to enabled?

[lukasnalezenec]

Hi, can you please add some script reproducin issue ?
We need to know how you read and write data, we are not interested in operations like filter, aggregations...

[wajda]

Are you using RDD or DataFrame? Spline can only track Spark transformations defined using Spark SQL / DataFrame API. Unfortunately RDD transformations are untrackable due to lack of meta information.

[ghost]

@wajda Do you think this issue of RDD transformations not having metadata could be potentially solvable through Spline eventually?

More specifically: I see that the entry point for the lineage data is through the QueryExecutionListeners, which is why the RDD transformations aren't getting picked up. Do you know of any ways/think it is possible for Spline to track RDD transformations through some other means?

[wajda]

Yes, we brainstormed this question a while ago, a few possible partial solution popped out. But they were either fragile or cumbersome to use, and neither was solving the problem completely.
We could capture a physical lineage, but the goal is to capture logical one, so we need to keep as close to a user source code as possible. We thought we could use AspectJ to intercept the RDD methods calls and capture the logical plan that way. But again, most of the RDD methods accept lambdas, and in JVM we don't know what is inside lambdas. This is the biggest issue. Perhaps the best solution would be the one that approaches the problem from both sides - utilizes a static source code analysis, and then link the static metadata with the dynamic metadata collected on runtime. IMO that sounds doable in theory, but quite complex in practice. But we never say never :)

[dan-coates]

In the event that Spline can't capture this lineage information automatically, we were considering options to augment the lineage data more or less manually. Ideally Spline has full visibility to all actions and can report them automatically, but where the operations are opaque or simply not captured at the sufficient detail in Spark's plans, business requirements would dictate that we still need to capture the actions somehow.

Do you think there would be any possibility in Spline's architecture to essentially inject user-defined lineage events into Spline? For context, we are calling DataFrame.foreachPartition and executing JDBC calls for each row here, so we would like to note that in the lineage stream, and could call a Spline method when we execute a DataFrame or RDD operation which Spline does not support.

I can see us having to have some kind of sidecar in any case to have complete visibility over lineage information, so if it can be part of Spline itself, that can at least contain all the lineage information in a single graph. If that doesn't make sense architecturally though, we can simply have two sets of lineage data and stitch them together outside of Spline.

[wajda]

Yes, it makes perfect sense. Thanks for the great idea. Need to think about it. So, basically, the way it could be done is something like this:

val df0: DataFrame = ... // my initial DataFrame
val rdd0 = df0.rdd       // switch to an RDD level. Automatic lineage is lost at this point!
val rdd1 = rdd0.map(myCoolRDDTransformFunction) // do some stuff on RDD
val df1 = rdd1.toDF()    // get back to DataFrame.
// From the business perspective df1 is derived from df0 via myCoolRDDTransformFunction, so...
df1.derivesLineageFrom(df0).as(spline.CustomOperation("my-cool-rdd-transform", ... /* other relevant info */ ))

I wonder if this approach could also work for tracking ML lineages.

[RavikumarRapolu]

Hi Wajda,

we are also facing issue with rdd, actually above idea is good. can we expext it in next release please , we want to use spline into our business model.

[Nikhith-kudumula]

Hi Wajda,

We are experiecing issue with generating lineage while there is a rdd logic in code. Can we expect fix for it in next version of Spline ?

[wajda]

We'll see. Not in 0.5, but maybe in one of the nearest future releases.

[Nikhith-kudumula]

Hi Wajda,

Can you please let us know if RDD lineage support is planned for release in near time. We are intergrating Spline in our business Model and we have rdd's in our code.

Thanks,
Nikhith

[wajda]

We'll try to include it in 0.6, aiming for the release at the beginning of June.

[ashyamala]

@wajda Do you still feel that Spline is still on track to support user-defined RDD lineage or any kind of RDD lineage within the 0.6 release or before the end of the year?

[wajda]

@ashyamala, it's still on the road map, the demand for this feature is growing. But apparently it won't fit into 0.6, as we've got plenty of stuff to do in 0.6 already.

[wajda]

End of the year - I'd say very likely.
Or, of course if anybody from the community could help us with that it could be released earlier.

[wajda]

It didn't make it to 0.6. It doesn't seem to be a priority so far...

[elymberopoulos]

Hello, I'm just wondering what the state of this issue is? Is it actively in progress?

[wajda]

No, it's not a priority at the moment, but is still on the road map.