DLT

Wiki in davidkhala/Delta

Declarative pipelines

• formerly known as Delta Live Table (DLT)

Compared to jobs, it add values in

• Data Quality: Built-in data quality features added
• Lineage Tracking: Automatically tracks data lineage
• Simplify pipeline management

built-in sample

• It uses legacy settings including
  • In Destination section. Storage options = Hive Metastore
  • In General section. Product edition still list classic options
• You cannot attach it to your existing compute
• The same job will resuse compute used in previous run if there is no setting change.
• These type of Job Compute is managed by Databricks

  • Error: dlt prefixed spark images cannot be used outside of the Delta Live Tables service

  • You cannot edit it
  • You cannot pause it. Once paused, new run will not reuse it. Instead, it will create another Job Compute.
• To save compute cost, configure
  • In Compute section
    • Cluster mode = Fixed size
    • Workers = 1
    • Don't enable Photon Acceleration
  • In Advanced section
    • Worker type = Standard_F4