Gathers Python stack for DevOps, everything is Docker!
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
apache-bigdata
basic-backend
basic
misc
piping
scaling-backend
stresstest/1.Locust-Stresstest
unit-test/1.pytest-flask
.gitignore
LICENSE
README.md

README.md

Python-DevOps

Gathers Python stack for DevOps, everything is Docker!

Table of contents

Basic Python

  1. AutoPEP8
  2. Graph function dependencies

Basic Backend

  1. Flask
  2. Flask with MongoDB
  3. REST API Flask
  4. Flask Redis PubSub
  5. Flask Mysql with REST API
  6. Flask Postgres with REST API
  7. Flask Elasticsearch
  8. Flask Logstash with Gunicorn
  9. MLFlow with Nginx reversed proxy

Scaling Backend

  1. Flask SocketIO with Redis
  2. Multiple Flask with Nginx Loadbalancer
  3. Multiple Flask SocketIO with Nginx Loadbalancer
  4. RabbitMQ and multiple Celery with Flask

Big data piping

  1. Streaming Tweepy to Elasticsearch
  2. Scheduled crawler using Luigi Spotify to Elasticsearch
  3. Airflow to Elasticsearch

Apache big data

  1. Flask with Hadoop
  2. Flask with Kafka
  3. Flask with Hadoop Hive
  4. PySpark with Jupyter and Hadoop
  5. Flink with Jupyter

Unit test

  1. Pytest

Stress test

  1. Locust

Miscellaneous (huseinhouse collection)

  1. Elasticsearch + Kibana
  2. Elasticsearch + Cerebro
  3. Jupyter notebook
  4. Jupyterhub
  5. Hadoop cluster + Luigi + Jupyter Notebook

How-to Docker

Every folders contain .yml for docker-compose. You need to install Docker-Compose first.

To run,

compose/build

To safely close

compose/down

To remove all images

docker rmi $(docker images -q)

To remove none images

docker rmi $(docker images -f "dangling=true" -q)

To remove all containers

docker rm $(docker ps -aq)