What's in your data? Extract schema, statistics and entities from datasets

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

Generate avro schemas from python dataclasses, Pydantic models and Faust Records. Code generation from avro schemas. Serialize/Deserialize python instances with avro schemas.

☀️ A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats

Avrotize is a command-line tool for converting data structure definitions between different schema formats, using Apache Avro Schema as the integration schema model.

Generate Apache Avro schemas for Python types including standard library data-classes and Pydantic data models.

ffmpeg for market data

🎼 Docker compose files for various kafka stacks

Use the AWS Glue Schema Registry in Python projects.

A pure python avro schema validator

An Avro SerDe implementation that integrates with the confluent schema registry and serializes and deserializes data according to the defined confluent wire format

☀️ Avro, Protobuf, Thrift on Swagger

This repository provides Python scripts to generate simulated data and produce it into Kafka topics, facilitating testing and development of Kafka-based applications and pipelines.

Utility generating avro files from postgres

Streaming Data from Kafka to Postgres with Kafka Connect, AVRO, Schema Registry and Python

CLI tool for consuming and producing Avro messages on Kafka.

Kafka streams processing in Python.

Convert AVRO events to JSON and perform schema validation using EventBridge Pipes and Confluent Schema Registry.

ECE4721J (VE472) "Methods and Tools for Big Data" Coursework @UM-SJTU Joint Institute