Python-DevOps

Gathers Python stack for DevOps, everything is Docker!

Table of contents

• Basic Python
• Basic Backend
• Scaling-up Backend
• Big data piping
• Apache big data
• Unit test
• Stress test
• Miscellaneous

Basic Python

1. AutoPEP8
2. Graph function dependencies

Basic Backend

1. Flask
2. Flask with MongoDB
3. REST API Flask
4. Flask Redis PubSub
5. Flask Mysql with REST API
6. Flask Postgres with REST API
7. Flask Elasticsearch
8. Flask Logstash with Gunicorn
9. MLFlow with Nginx reversed proxy

Scaling Backend

1. Flask SocketIO with Redis
2. Multiple Flask with Nginx Loadbalancer
3. Multiple Flask SocketIO with Nginx Loadbalancer
4. RabbitMQ and multiple Celery with Flask

Big data piping

1. Streaming Tweepy to Elasticsearch
2. Scheduled crawler using Luigi Spotify to Elasticsearch
3. Airflow to Elasticsearch

Apache big data

1. Flask with Hadoop
2. Flask with Kafka
3. Flask with Hadoop Hive
4. PySpark with Jupyter and Hadoop
5. Flink with Jupyter

Unit test

1. Pytest

Stress test

1. Locust

Miscellaneous (huseinhouse collection)

1. Elasticsearch + Kibana
2. Elasticsearch + Cerebro
3. Jupyter notebook
4. Jupyterhub
5. Hadoop cluster + Luigi + Jupyter Notebook

How-to Docker

Every folders contain .yml for docker-compose. You need to install Docker-Compose first.

To run,

compose/build

To safely close

compose/down

To remove all images

docker rmi $(docker images -q)

To remove none images

docker rmi $(docker images -f "dangling=true" -q)

To remove all containers

docker rm $(docker ps -aq)