lake

Delta Lake

Github org, major repos

open-source data management for the lakehouse

• Adding an ACID transactional storage layer on top of object storage
• data versioning and time travel(e.g. rollback)
• handle both batch and streaming data in a unified way

History

Donoted to Linux Foundation

• it originates in October 2017
• Donated from Databricks, by Oct 2019
  • LF announcement
  • Databricks annoucement

Runtime

option 1: runs on top of existing data lake

option 2: runs on top of Apache Spark

ACID

concurrently multi-writes: Delta Lake supports multiple writes simultaneously

• Delta Lake does locking to make sure that queries writing to a table from multiple requester at the same time won’t corrupt the table.