ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

A tool for data sampling, data generation, and data diffing

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

A collection of Magnolia add-on modules

Big Data Toolkit for the JVM

Spark SQL index for Parquet tables

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

Read SparkSQL parquet file as RDD[Protobuf]

GCS support for avro-tools, parquet-tools and protobuf

A set of connectors for Monix. 🔛

Apache Spark is a fast, in-memory data processing engine with elegant and expressive development API's to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.This project will have sample programs for Spark in Scala language .

A light Kafka to HDFS/S3 ETL library based on Apache Spark

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

Bucketing and partitioning system for Parquet

A collection of Apache Parquet add-on modules

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Cloud Data Access client

Parquet file generator

Code examples for my blog posts

Embulk (https://github.com/embulk/embulk/) output plugin to dump records as Apache Parquet (https://parquet.apache.org/) files on S3.