ATLAS-3661 Create 'spark_column_lineage' type and relationship definition

[vladhlinsky]

What changes were proposed in this pull request?

Create spark_column_lineage type and relationship definition to add support of column level lineage for CREATE TABLE AS SELECT ... statements and views. Column level lineage refers to lineage created between the input and output columns.
For example:

hive > create table employee_ctas as select id from employee;

For the above query, lineage is created from employee to employee_ctas, and also from employee.id to employee_ctas.id.

How was this patch tested?

Manually using modified version of Spark Atlas Connector:

• Installed and started Atlas.
• 1100-spark_model.json is updated with proposed changes. Atlas is restarted.
• Executed the next statements using spark-shell:

spark.sql("create table sparkemployee_1_2(id int,name string)");
spark.sql("create table sparkemployee_ctas_1_2 as select id from sparkemployee_1_2");

• Verified that each table has column entities and spark_column_lineage entity is created.

[vladhlinsky]

Attaching screenshots.

[vladhlinsky]

cc @HeartSaVioR @sarathsubramanian

[sarathsubramanian]

Changes looks good. +1. Thanks @vladhlinsky.

[HeartSaVioR]

LGTM

[nixonrodrigues]

+1 for PR, thanks @vladhlinsky for PR.

[lyyprean]

what version is spark-atlas-conection used

[lyyprean]

What version does spark-atlas-connector use?

[CavalierHE]

What version does spark-atlas-connector use?

[pPanda-beta]

@lyyprean @CavalierHE

I think cloudera has not open sourced it yet, and completely hiding the implementation of column lineage harvester!

[CavalierHE]

@lyyprean @CavalierHE

I think cloudera has not open sourced it yet, and completely hiding the implementation of column lineage harvester!

maybe，how about apache？