Repository navigation
Switch Azure deployment to Kubernetes - #1301
Merged
Merged
Conversation
Sumukh
pushed a commit
that referenced
this pull request
Jul 2, 2018
CircleCI version `1.0` is deprecated and will be start rejecting jobs on August 30th. This PR migrates our workflow to `2.0`. This loosely follows the the [1.0 -> 2.0 Migration Guide](https://circleci.com/docs/2.0/migrating-from-1-2/). This incidentally fixes the pip failures on recent CircleCI runs failing (#1299 #1301 etc) with the following error `_NamespacePath object has no attribute sort ` Signed-off-by: Colin Schoen <cschoen@berkeley.edu>
colinschoen
approved these changes
Jul 3, 2018
colinschoen
left a comment
Member
There was a problem hiding this comment.
Happy to merge this... here is the stamp. I'll let you merge in case you want someone else from Microsoft to review.
Contributor
Author
|
@marrobi @martinpeck Would you like to review this or are you happy to merge this as-is? There's an architecture diagram here: |
Previously we were hosting the ok-server, ok-worker, autopy-manager and autopy-worker in AppService for containers. However, we were seeing some issues with this approach as workers were killed without death notifications and rq/Redis was getting confused. After this change, we now deploy the compute (workers and servers) to Azure Kubernetes service. In order to make it easier to orchestrate the setup of the various resources in Azure as well as the Kubernetes deployments, we're using bash scripts. First, we create a bunch of resources using `setup.sh` and then we configure them using `install.sh`. The setup scripts are wrapped in a Docker container so that the environment in which the scripts execute is well known, e.g. Azure CLI is installed, helm is available, kubectl is available, etc. Most of the setup scripts fall-back to ARM templates to create the resources in Azure. The ARM templates can be modified to configure parameters such as resource SKUs, number of VMs to spin up for the cluster, type of VMs to use in the cluster, etc. The install scripts perform operations such as creating the database schema, deploying the application to Kubernetes via Helm and setting up the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego can configure a TLS certificate for the ingress via Let's Encrypt. Note that the ingress to the applications running in the Kubernetes cluster are configured as just one entrypoint so that a single TLS certificate and DNS entry can be used (Azure public IP resource currently only support a single DNS label per IP). To solve routing, we thus need to mount the two sub-applications (OKpy and AutoPY) at sub-urls of the ingress which requires some minor application changes which are submitted in a separate pull request: #1300 The setup scripts can be run either via the "Deploy to Azure" button (in which case the setup Docker container is run in an Azure Container Instance) or via a local `docker run` command. In the latter case, it's advised to also include a volume mount `docker run -v $PWD:/app/secrets` so that the access keys for the resources created during the deployment will be made available to the deployer. Note that in order to skip deploying specific resources (e.g. to re-use existing resources), you can place a file into the secrets folder that contains the access keys for your existing resources in which case the setup scripts will skip deploying that step.
Contributor
Author
|
@colinschoen Let's merge this. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Previously we were hosting the ok-server, ok-worker, autopy-manager and
autopy-worker in AppService for containers. However, we were seeing some
issues with this approach as workers were killed without death
notifications and rq/Redis was getting confused. After this change, we
now deploy the compute (workers and servers) to Azure Kubernetes
service.
In order to make it easier to orchestrate the setup of the various
resources in Azure as well as the Kubernetes deployments, we're using
bash scripts. First, we create a bunch of resources using
setup.shandthen we configure them using
install.sh. The setup scripts are wrappedin a Docker container so that the environment in which the scripts
execute is well known, e.g. Azure CLI is installed, helm is available,
kubectl is available, etc.
Most of the setup scripts fall-back to ARM templates to create the
resources in Azure. The ARM templates can be modified to configure
parameters such as resource SKUs, number of VMs to spin up for the
cluster, type of VMs to use in the cluster, etc.
The install scripts perform operations such as creating the database
schema, deploying the application to Kubernetes via Helm and setting up
the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego
can configure a TLS certificate for the ingress via Let's Encrypt.
Note that the ingress to the applications running in the Kubernetes
cluster are configured as just one entrypoint so that a single TLS
certificate and DNS entry can be used (Azure public IP resource
currently only support a single DNS label per IP). To solve routing, we
thus need to mount the two sub-applications (OKpy and AutoPY) at
sub-urls of the ingress which requires some minor application changes
which are submitted in a separate pull request:
#1300
The setup scripts can be run either via the "Deploy to Azure" button (in
which case the setup Docker container is run in an Azure Container
Instance) or via a local
docker runcommand. In the latter case, it'sadvised to also include a volume mount
docker run -v $PWD:/app/secretsso that the access keys for the resources created during the deployment
will be made available to the deployer. Note that in order to skip
deploying specific resources (e.g. to re-use existing resources), you
can place a file into the secrets folder that contains the access keys
for your existing resources in which case the setup scripts will skip
deploying that step.