Data-stack

Attempt to create a complete architecture to handle meteorological data.

short-term (WIP):

• upload grib/hdf5 DataFile (FastAPI)
• archive DataFile (S3 / MinIO / Posix)
• manage DataFile lifecycle (PostgreSQL, Kafka)
• ingest DataFile as a set of Measurements
• Efficient temporal access to Measurements (InfluxDB)
• Efficient spatial query (PostGIS)

middle-term (TODO):

• Generate graphics, data visualisation, frontend (?)
• Prediction, machine learning, experiments on data
• DevOps: packaging, monitoring, scalability (Prometheus, Grafana, Kubernetes)

personal goals:

• benchmark different Techs (message queues, databases)
• experiments with code structure / technics

Documentation

• Features
• Glossary
• Technologies

Setup

# create virtual env
uv venv

# synchronize environment
uv sync --all-packages

# lint, format (pre-commit)
uv run pre-commit run --all-files

# generate docs
uv run scripts/analyze.py

# run server
cd projects/meteo-backend

# test project
uv run coverage run -m pytest

# generate coverage
uv run coverage html -d docs/coverage