Home

Jump to bottom

David Liu edited this page Dec 5, 2024 · 9 revisions

Welcome to the delta wiki!

Delta Lake

Github org, major repos

open-source data management for the lakehouse

Adding an ACID transactional storage layer on top of object storage
data versioning, rollback
handle both batch and streaming data in a unified way
provide table abstraction for SQL and DataFrame API

Donoted Linux Foundation

LF announcement
Databricks annoucement
it was launch in October 2017
from Databricks, by Oct 2019

DeltaLogs

Delta Lake transaction Log

Table Ledger: single source of truth trackcing table changes
guarantees atomicity of Delta Lake

Delta Table

Delta Table: (in general) the table in Delta Lake

Delta engine

A query optimizer for Delta Lake

Proprietary to Databricks
for performance of Spark SQL, Databricks SQL and Dataframe operations
apply push down design

Delta Live Tables (DLT): Data pipeline framework

DLT is not a table

It manage the data flow, task orchestration, data quality and error handling
It offers declarative pipeline development
It contains pipeline, the main unit of execution
It manages many Delta tables
proprietary to Databricks

Delta Sharing

An open protocol for secure data sharing