-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support attribute and expression level lineage for MapElements, SerializeFromObject, DeserializeToObject #342
Comments
@cerveada what do you think is the priority on getting this implemented? |
We might try to solve it using custom annotations on the model case classes for instance to carry the missing compile time information to the runtime, but this isn't something that can be done quickly. I'd say this is a nice feature request that could be addressed in the scope of solving RDD lineage gaps in #33 |
I didn't quite get about |
The lineage of the example in this discussion: #341 currently outputs 5 operations:
I am not too certain on the details of what occurs in the DeserializeToObject and SerializeFromObject to be honest. When I dug into the collections in the ArangoDB, my biggest issue was trying to find a connection from fields in the |
That's because the connections between fields in |
@wajda Ah okay. So it sounds like it isn't possible to create this feature? |
Well, it's practically impossible to do it automatically. In theory we could try to decompile and reverse engineer the bytecode in attempts to recover the tracing between the fields, but you know, the amount of work is significant and the outcome is not really predictable or guaranteed. So I would prefer not going that route. |
@wajda ah I see. Looking at that outer feature request, while it isn't the most ideal solution, I think we could make that work for us. So I am fine with closing out this request for now. |
Leave it open please, for ease of tracking. |
Lineages generated from code showed in discussions/341 are missing connections between attributes and expressions.
Let's add support for that.
The text was updated successfully, but these errors were encountered: