DataSource information enchancement

[vishalag001]

Currently, the dataSource collection only contains URI and the name is dependent on the URI(anything after the '/'). However a dataSource should ideally have a tableName, schema and related details.

In Spline, such information is captured on the write operation. Say for a hiveTable write, we have params which contains the tableName, Schema name, etc. For BigQuery, we get datasetName, projectName and tableName.

Is it possible to leverage the operation collection to enhance the dataSource collection ?

Benefits of this approach:

• The UI could refer to the schema.tableName rather than the name(which is derived from URI) and make it more meaningful.
• It will help to list dataSource URI which fall under same tableName( ,i.e, same table but different partitions)
• On UI, the list of different tables can be displayed and from there on one can navigate to the lineageOverview (by the corresponding progress Event). In case of more than 10 partitions, we can use latest partitions to display the lineage

@wajda let me know your thoughts. I am happy to contribute to this.

[wajda]

Yes, I completely understand your points. It's certainly possible, but it should rather be done on the UI end (respectively the Consumer API) by queries Operation collection via respective writesTo edge, instead of storing those details directly in the DataSource collection.
The reason is the cardinality of WriteOp -> DataSource relation, which is many-to-one. Thus, the same data source can potentially be named differently in the context of different execution plans (think about aliases) while referring the very same data source instance (a table or a HDFS directory). The data source URI serves as an ID and it's name property is supposed to be a human readable name that can be automatically derived from some write operations and/or later changed on the UI. It's just that currently it's initially set to the last part of the URI, but it could be anything really.
We are planning to create a mechanism that would allow users to configure their own rules and expressions to automatically derive initial data source names from the context, as well as configure data source mapping rules to automatically resolve aliases or otherwise equivalent URIs as one. See #689

[wajda]

I've create a separate issue for this - #1095