[AIRFLOW-5912] Expose lineage API

[bolkedebruin]

Only lineage items obtained from XCom were rendered rather than
all. Additionally source tasks are recorded.

---

Issue link: AIRFLOW-5912

• Description above provides context of the change
• Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[mik-laj]

Can you also update the documentation?
https://github.com/apache/airflow/blob/master/docs/rest-api-ref.rst

[bolkedebruin]

Can you also update the documentation?
https://github.com/apache/airflow/blob/master/docs/rest-api-ref.rst

Mmm yes I will. I do think this should be autogenerated in the future from a swagger definition.

[mik-laj]

@bolkedebruin This month, I will start a new AIP on modernizing and hardening the API to make it fit for production use. Automatically generated documentation is one of the elements

[bolkedebruin]

Tests are passing (error seems transient). @mik-laj can you please review again? @potiuk maybe as well?

[potiuk]

Will do this weeknd. Seems tha the travis errors are not completely gone - we have just one thing left (CoreDNS configuration in Kind) to fix in order to be ready to move to Github Actions.

[codecov-io]

Codecov Report

Merging #7138 into master will increase coverage by 0.15%.
The diff coverage is 77.02%.

@@            Coverage Diff            @@
##           master   #7138      +/-   ##
=========================================
+ Coverage   85.24%   85.4%   +0.15%     
=========================================
  Files         683     755      +72     
  Lines       39155   39758     +603     
=========================================
+ Hits        33378   33954     +576     
- Misses       5777    5804      +27

Impacted Files	Coverage Δ
airflow/api/client/json_client.py	0% <0%> (ø)	⬆️
airflow/www/api/experimental/endpoints.py	89.81% <100%> (+1.09%)	⬆️
airflow/lineage/__init__.py	93.84% <100%> (ø)	⬆️
airflow/api/client/local_client.py	92% <50%> (-8%)	⬇️
airflow/api/client/api_client.py	63.15% <50%> (-1.55%)	⬇️
airflow/example_dags/example_papermill_operator.py	68.18% <68.18%> (ø)
airflow/api/common/experimental/get_lineage.py	89.47% <89.47%> (ø)
...rflow/contrib/sensors/sagemaker_training_sensor.py	0% <0%> (-100%)	⬇️
airflow/contrib/hooks/azure_data_lake_hook.py	0% <0%> (-93.11%)	⬇️
airflow/contrib/sensors/azure_cosmos_sensor.py	0% <0%> (-81.25%)	⬇️
... and 228 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d654d69...a55ee87. Read the comment docs.

[potiuk]

Just some minor nits. I think for now we should let it be added in the same way as other experimental APIs until we decide what's next for the API.