Apache Avro is a data serialization system.

Confluent Schema Registry for Kafka

Iceberg is a table format for large, slow-moving tabular data

MongoDB Kafka Connector

A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

Divolte Collector

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

⛈️ RumbleDB 1.21.0 "Hawthorn blossom" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

eCommerce application is a platform where customers can find products, shop around using a cart, check out the products and initiate payments. It is used for the didactic purpose only, as a support project for my training at https://www.ionutbalosin.com/training

Project goal: Explore Kafka, Kafka Connect, and Kafka Streams. Components: store-api: Inserts/updates MySQL records. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors: Listen to Kafka, insert/update Elasticsearch docs. store-streams: Listens to Kafka, processes with Kafka Streams, pushes new messages to Kafka.

Fluent Kafka Streams Test with Java

Wrangler Transform: A DMD system for transforming Big Data

Experiment with Kafka, Debezium, and ksqlDB. research-service: Performs MySQL record manipulation. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors and kafka-research-consumer: Listen to Kafka, insert/update Elasticsearch. ksqlDB-Server: Listens to Kafka, performs joins, and pushes new messages to new Kafka topics.

A user friendly API for checking for and reporting on Avro schema incompatibilities.

JDK 11 examples and demo projects.

Tablasco is a JUnit rule for comparing tables and Spark module for comparing large data sets