DM-5194: Dockerize LTD Keeper #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Flask-Script solves a lot of problems with building an application launcher, and will be useful in the future. See http://flask-script.readthedocs.org/en/latest/ See Miguel Grinberg on how Flask-Script can be further used to setup debugging shell environments, and commands for DB migrations. See https://github.com/miguelgrinberg/flasky/blob/master/manage.py A key change is that we can configure the app object without running it. With uWSGI we need to provide a module with a flask application callable object. The run.py runserver command now runs a development server. I've tried to integrate test.py into Flask-Script, but there's a weird test discovery configuration glitch with py.test that I cannot yet solve. Also moved DB initialization to a specialized sub-command: ./run.py init In a Kubernetes context the right thing to do would run a specialized container that initializes the database and then exists. The same would happen for DB migrations.
A better move would be to test if this is a development profile or not; only do the initialization in a development case.
uWSGI is used as an application server that binds the Flask web app to Nginx. uwsgi talks to nginx over TCP socket on the 3031 port. I've found that I don't need to specify the host name at all. I think :3031 is equivalent to 0.0.0.0:3031 uWSGI connects to the Flask app via the module and object within the module. Here run.py is where the Flask app object is created. This is in prep for running dual container pods to separate the nginx and uwsgi processes.
The image is based on python:3.5.1 and installs ltd-keeper by first copying the local source into the container, and then installing dependencies via requirements.txt We then run the Flask app via uWSGI (configured via the uwsgi.ini file) so that nginx (in a separate container) can be used as a reverse proxy. This gives us the robustness to deploy ltd-keeper in production. dockerignore prevents all __pycache__ from the local environment from being copied into the container; this allows us to run test.py.
This Compose file runs two services: the uWSGI+Flask ltd-keeper app from this repo and the nginx-python container from https://github.com/lsst-sqre/nginx-python-docker You can deploy this setup by cloing both nginx-python-docker and this repo into the same base directory (side by side). The build fields will build the necessary images (these build fields can be removed and replaced with just an image field pointing to a pre-built image that's either local or on Docker Hub. Deployment is done via `docker-compose up`. Note that the nginx service links to the uwsgi server. This allows nginx to talk to uwsgi over a http://uwsgi:3031 socket.
This image is on Docker Hub lsstsqre/nginx-python:compose Since that image is fairly stable there shouldn't be any need to rebuild it locally by default.
Use **/ syntax to do global globs. See https://docs.docker.com/engine/reference/builder/#dockerignore-file
jonathansick
force-pushed
the
tickets/DM-5194
branch
from
b4b6151
to
e4341a7
Compare
This replication controller includes a templates for creating keeper pods that behave similarly to the docker-compose set up where separate nginx and uwsgi containers work together. Rather than create a bare pod, it's smart to use a replication controller. This way we can ensure the health of keeper, do rolling updates, and eventually, scale the number of keeper pods. The pod configuration is done through Kubernetes secrets (keeper-secrets.template.yaml). The pod template configures environment variables in the uwsgi container based on these secrets. The secrets config file is only a template since we obviously don't want to check the real secrets into the repo. The trouble here is we can't use this replication controller until we adopt a networked SQL database since we can't attached persistent storage to pods managed by a replication controller.
This Service provides a stable external IP for all pods matching the keeper selector. If there are multiple keeper pods, it will load balance traffic.
Ingress provides the stable IP in front of the Keeper service and is also used to terminate TLS. The TLS certificates are supplied via a keeper-ingress-secrets.yaml file (again, this is a template to prevent checking in secrets in the repo) Note that Ingress is in Beta and requires setup on Google Container Engine. See http://kubernetes.io/docs/user-guide/ingress/#prerequisites
Use this to create bare pods for maintenance activities, like running a migration or initializing a database. This pod overrides the default command in the uwsgi container to simply run sleep for 1 day rather than start the uWSGI+Flask app. This way an operator can log into the container and use run.py init or run.py migrate, etc.. Also overrides the securityContext to run as root (uuid=0). I found this necessary because the persistent storage from Google Compute Engine is owned by root, so a root user needs to initially setup the database before chown-ing it to the uwsgi user that we normally run.
This persistent storage is a disk on Google Container Engine that is resiliant to a pod or even node going down. We'll use for now to host the sqlite database. The sqlite URI to this database is stored in the keeper-secrets.
This makes it easy to inspect the db directly, or through the model classes, when in a run.py shell. Based on http://flask-script.readthedocs.org/en/latest/#shell docs
Never used this functionality; suppresses a warning until Flask-SQLAlchemy 3 See http://stackoverflow.com/a/33790196
These notes document my initial experiments to deploy Keeper on Google Container Engine. Eventually these notes will be refined into more formal deployment documentation in the Sphinx docs/ dir.
It turns out we can't use the keeper-controller replication controller since we can't attach persistent storage to a pod managed by a replication controller. In the meantime, we can deploy a bare pod looked up to persistent storage. This yaml pod definition does it. This pod is very similar to the keeper-mgmt-pod, except that all the hacks to prevent uwsgi from running, and also running as root are removed. The label is also configured so that the keeper-service load balancer will send traffic to the pod. Effectively this is a drop-in replacement for keeper-controller.
jonathansick
force-pushed
the
tickets/DM-5194
branch
from
e4341a7
to
e81db0d
Compare
I know that tests/__init__.py isn't recommended, but this seems to be the only way to enable test recovery given that the app isn't setup.py install'd. I think this is better than test.py since it allows you to run py.test direct (e.g., to run extra plugins).
- Added docs to test with cov and flake8 plugins - Travis uses cov and flake8 plugins. Note that only flake8 will generate errors; cov will just print results to the log.
- Install the CLIs - Create the cluster - Create the persistent storage.
jonathansick
force-pushed
the
tickets/DM-5194
branch
from
425f082
to
02abbbd
Compare
The environment variables that configure the keeper app are set via secrets resources deployed to kubernetes. This page provides background on how secrets work, how to use secrets, and a reference on the secrets the configure the Keeper Flask app specifically.
This is based mostly on my original notes kubernetes/README.rst. Still needs to include the Ingress resource deployment.
Running the keeper pods as a Deployment gives us the replication controller for free with the bonus that I can now connect the persistent disk.
Instead of trying to use Kubernetes Ingress to terminate SSL, which I can't get to work, I've adapted the deployment to use an Nginx SSL proxy. This setup is based on https://github.com/GoogleCloudPlatform/kube-jenkins-imager/blob/master/LICENSE Basically it takes traffic on ports 80 and 443, terminates SSL, and sends traffic to the keeper service on port 8080. The nginx proxy also redirects port 80 to use HTTPS instead. - ssl-proxy-service exposes ports 80 and 443 to the world gives the external IP - keeper-service sets a cluster IP and port that the SSL proxy can send traffic to. - ssl-proxy runs an Nginx container from Google that is configured to terminate SSL
jonathansick
force-pushed
the
tickets/DM-5194
branch
from
02abbbd
to
75e8155
Compare
This deployment guide covers the new architecture with an Nginx SSL proxy service and with Keeper being run as a Kubernetes deployment.
These configurations included running Keeper not as a Deployment, and also using Ingress instead of the Nginx SSL proxy.
jonathansick
force-pushed
the
tickets/DM-5194
branch
from
75e8155
to
9d813da
Compare
|
See operations documentation at http://ltd-keeper.lsst.io/en/tickets-dm-5194/ |
Original file is assets/kubernetes_arch.ai and the output rendering is _static/kubernetse_arch.svg
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dockerizes the Keeper app such that it can be deployed into production on Kubernetes and also developed locally with Docker Compose.
The architecture is to have an nginx container (see https://github.com/lsst-sqre/nginx-python-docker) that acts as a reverse proxy to a container running Keeper. Keeper is run through uWSGI.
In the production (Kubernetes) deployment there are two layers: