goal

The main goal is to publish some recreatives projects for me to learn, exchange with people knowing more than me and maybe offer me new opportunities for data engineering projects.

TODO : Repaint my white hair because of this cluster and make my producer work.

LittleMeteoProject

A Kafka project with K8s.

K8S :

https://github.com/apache-spark-on-k8s/kubernetes-HDFS/tree/master

rust producer meteo

The cargo directory containing the Rust producer. Adjust the config files in rust_producer_meteo/config/apiparams.json Create rust_producer_meteo/config/apikey.json

{
    "keys": [
        {"OPENWEATHERMAP": "KEY ..." },
        {"OTHER_API ": "OTHER KEY ..."}
    ]
}

Configs folder

Contains configs read by minikube to start the services, pdbs, deployments and pvcs.

This cluster would contain :

• HDFS
• Kafka + Schema Registry & Zookeeper
• Spark consumer with Scala

sh src/minikube/start-env.sh

Spark

Not created yet. Would be a script made to "normalize" different sources.

Data sources / dashboard

Graphs comparing air quality and weather in quiet zones like Le Vésinet vs a urban zone like Saint-Denis

openweathermaps.com