Data lake metadata and transaction log store.
Storing metadata about files allows you to intelligently skip data files when querying data to make computations run faster. You can also run some queries on the metadata itself.
Your metadata needs to be specific to your data and query patterns. I can't tell you what you should store in your metadata layer. You need to look at how you interact with your data to determine the metadata fields that will allow for data skipping.
The transaction log records files that are added and removed from the data lake. The transaction log allows for powerful features like time travel, versioned data, and backwards compatible compaction.
This project is inpired by Delta lake. It has the following important differences:
- It's not intended to be compatible with Hive, so it can implement "disk partitioning" more efficiently
- Appa doesn't support streaming, so it's less complex
- Appa is intended to work with different types of technologies, not just Spark. The Delta philosophy is "do all your data lake management processes with Spark". The Appa philosophy is "use Spark, Dask, Pandas, or pure Python to manage the different operations". Spark is a good technology for certain data lake operations, but not the best for operations like deleting files in S3.
- Small file compaction is a first class citizen in Appa
Suppose you have the following data files.
file1.csv
full_name,birth_year,country
Confucius,551,china
Deng Xiaoping,1904,china
Fan Bingbing,1982,china
file2.csv
full_name,birth_year,country
Mahatma Gandhi,1948,india
Amartya Sen,1933,india
Priyanka Chopra,1982,india
file3.csv
full_name,birth_year,country
Shaggy,1968,jamaica
Usain Bolt,1986,jamaica
Chetan Bhagat,1974,india
Diego Maradona,1960,argentina
Let's create a metadata store for these files:
file_name,countries,max_birth_year
file1.csv,[china],1982
file2.csv,[india],1982
file3.csv,[jamaica,india,argentina],1986
Here's how the metadata store allow us to run queries faster:
- For
where country = 'china', we can query file1 and skip file2 and file3 - For
where birth_year > 1985, we can query file3 - For
where country in ('jamaica', 'india')', we can query file2 or file 3
Appa gives you the flexibility to choose which metadata you'll store. Generating metadata takes computational resources so you don't want to store metadata needlessly.
TODO