Data pipelines for Spark structured streaming and Spark batching jobs.
- spark_jobs: Spark jobs to ingest data from Kafka and process it in micro batches.
- spark_libs: Spark common libraries for all jobs to process data such as flattening nested json data, .etc