Support for showing lineage of table across ETL's in a data warehouse

[apoorva007]

Hey,
Wanted to understand if there is a way to push table lineage related metadata to amundsen.
Couldn't find any related documentation. If someone can point towards it, that would be really helpful

[feng-tao]

@apoorva007 we haven't build the support in databuilder for neo4j as metadata backend. But I know @verdan 's team has built support for pushing lineage info into atlas which is another proxy offering for metadata.

cc @verdan for further comment.

[verdan]

@apoorva007 you can use Amundsen with either Neo4j or Apache Atlas as the backend. We use Apache Atlas (and are constantly working on improving and adding new features in Amundsen) as our metadata proxy. Atlas supports the lineage out of the box, which is a one of a huge advantage of using Atlas as metadata proxy. It connects with many elements like spark, hive etc. using hooks and bridges, and thus records the whole lineage out of the box.
We have not yet started implementing lineage in Amundsen (via Atlas proxy), but please let us know what do you think of the this, your requirements, and if we can collaborate more on that part.

[jornh]

First option - link to Atlas or other 3rd party

As @verdan points to Atlas already has strong lineage modelling and UI - a shortcut to navigating to that from Amundsen might simply be to populate the TABLE LINEAGE sidebar link with URL data pointing to the right lineage page as shown here (this is what I understand Lyft is already doing with the 3rd party lineage tool they have deployed)

2nd option - "the younger brother" populate Neo4j

Another option which we could name "Atlas lineage's younger brother" would appear if you take a closer look at Amundsen's current Neo4j backend's data model. You'll notice the ingredients to model and store lineage are already in the diagram. It's the DAG - write_to -> Table and Table - replicated_from -> Table "links":

In fact you could claim that the - DAG - writes_to -> Table part is already visualized by the GENERATED BY section in the top screenshot in this comment. See further on near-term plans in https://github.com/lyft/amundsenfrontendlibrary/issues/202#issuecomment-504213401 [broken and outdated link]

---

Related issue in the frontend repo discussing "more ambitious" Amundsen lineage visualization: https://github.com/lyft/amundsenfrontendlibrary/issues/110

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[jornh]

2nd option above has moved a bit forward with merge of amundsen-io/amundsendatabuilder#126 as far as I can see this means with Neo4j #69 (comment) is essentially implemented so you can build a Databuilder job which will build lineage in Amundsen Metadata service.

Submitting a PR to ingest some sample data modeled after previous sample ingestion PRs seems like a good good first issue if you have Python skills and want to get your hands dirty and move this feature forward sooner rather than later!

If you have React skills https://github.com/lyft/amundsenfrontendlibrary/issues/110 might be what you want to look at

Roadmap link

https://github.com/lyft/amundsen/blob/master/docs/roadmap.md#native-lineage-integration