lineage

ERD

• as table relationship
• aws doc

Lineage

Limit

• Lineage is computed on a one-year rolling window
• Lineage is not captured when:
  • table/view, schema or catalog is renamed
  • you use Spark SQL dataset checkpointing
  • you want to cover Global temp views, Stack function
  • you want to cover tables under system.information_schema
• Column lineage is not captured when:
  • either source or target is not referenced by table name. such as select * from delta."s3://<bucket>/<path>"
• Immutable: You cannot delete table rows of system lineage tables

  • To delete lineage data, you must delete the metastore managing the Unity Catalog objects

  • data retention: Managed table data and metadata will be auto-deleted after 30 days.