CN Doc

LakeSoul

LakeSoul is a unified streaming and batch table storage for fast data processing built on top of the Apache Spark engine by the DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and streaming & batch unification.

LakeSoul implements incremental upserts for both row and column and allows concurrent updates on the same partition. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales meta data management by using distributed NoSQL DB Cassandra.

More detailed features please refer to our wiki page: Wiki Home

Some features and performance comparisons: Data Lake Comparison

Usage Documentations

Please find usage documentations in project's wiki: Usage Doc

使用文档

Follow the Quick Start to quickly setup a test env.

Checkout the CDC Ingestion with Debezium and Kafka example on how to sync LakeSoul table with OLTP dbs like MySQL in a realtime manner.

Feature Roadmap

• Meta Management
  • Multiple Level Partitioning: Multiple range partition and at most one hash partition
  • Concurrent write with optimistic lock mechanism
  • MVCC with read isolation
  • Write atomicity through Cassandra's Light Weight Transaction
• Table operations
  • LSM-Tree style upsert for hash partitioned table
  • Merge on read for hash partition with upsert delta file
  • Copy on write update for non hash partitioned table
  • Compaction
• Spark Integration
  • Table/Dataframe API
  • SQL support with catalog except upsert
  • Query optimization
    • Shuffle/Join elimination for operations on primary key
  • Merge UDF (Merge operator)
  • Merge Into SQL support
    • Merge Into SQL with Primary Key
    • Merge Into SQL with non-pk
• Flink Integration
  • Table API
  • Flink CDC
• Realtime Data Warehousing
  • CDC ingestion and time travel
  • Event driven materialized view build and compaction
• Cloud Native
  • Object storage IO optimization
  • Multi-layer storage classes support with data tiering

Feedback and Contribution

Please feel free to open an issue or dicussion if you have any questions.

Join our slack user group

Contact Us

Email us at opensource@dmetasoul.com.

Opensource License

LakeSoul is opensourced under Apache License v2.0.