Construct a streaming event pipeline around Apache Kafka and its ecosystem. Using public data from the Chicago Transit Authority, we will construct an event pipeline around Kafka that allows us to simulate and display the status of train lines in real time.

Make avro-tools available via a Docker container.

Samples related to data engineering, e.g. spark, embulk, airflow, etc.

Simulate alert streams for the Fink broker

A pure python avro schema validator

Apache Avro <-> pandas DataFrame

Generate avro schemas from python dataclasses, Pydantic models and Faust Records. Code generation from avro schemas. Serialize/Deserialize python instances with avro schemas.