Skip to content

Switch Azure deployment to Kubernetes - #1301

Merged
Sumukh merged 2 commits into
okpy:masterfrom
c-w:bug/c-w/azure-aks
Jul 6, 2018
Merged

Sumukh merged 2 commits into
okpy:masterfrom
c-w:bug/c-w/azure-aks

Conversation

@c-w

@c-w c-w commented Jun 29, 2018

Copy link
Copy Markdown
Contributor

Previously we were hosting the ok-server, ok-worker, autopy-manager and
autopy-worker in AppService for containers. However, we were seeing some
issues with this approach as workers were killed without death
notifications and rq/Redis was getting confused. After this change, we
now deploy the compute (workers and servers) to Azure Kubernetes
service.

In order to make it easier to orchestrate the setup of the various
resources in Azure as well as the Kubernetes deployments, we're using
bash scripts. First, we create a bunch of resources using setup.sh and
then we configure them using install.sh. The setup scripts are wrapped
in a Docker container so that the environment in which the scripts
execute is well known, e.g. Azure CLI is installed, helm is available,
kubectl is available, etc.

Most of the setup scripts fall-back to ARM templates to create the
resources in Azure. The ARM templates can be modified to configure
parameters such as resource SKUs, number of VMs to spin up for the
cluster, type of VMs to use in the cluster, etc.

The install scripts perform operations such as creating the database
schema, deploying the application to Kubernetes via Helm and setting up
the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego
can configure a TLS certificate for the ingress via Let's Encrypt.

Note that the ingress to the applications running in the Kubernetes
cluster are configured as just one entrypoint so that a single TLS
certificate and DNS entry can be used (Azure public IP resource
currently only support a single DNS label per IP). To solve routing, we
thus need to mount the two sub-applications (OKpy and AutoPY) at
sub-urls of the ingress which requires some minor application changes
which are submitted in a separate pull request:
#1300

The setup scripts can be run either via the "Deploy to Azure" button (in
which case the setup Docker container is run in an Azure Container
Instance) or via a local docker run command. In the latter case, it's
advised to also include a volume mount docker run -v $PWD:/app/secrets
so that the access keys for the resources created during the deployment
will be made available to the deployer. Note that in order to skip
deploying specific resources (e.g. to re-use existing resources), you
can place a file into the secrets folder that contains the access keys
for your existing resources in which case the setup scripts will skip
deploying that step.

Sumukh pushed a commit that referenced this pull request Jul 2, 2018
CircleCI version `1.0` is deprecated and will be start rejecting jobs on August 30th. This PR migrates our workflow to `2.0`. This loosely follows the the [1.0 -> 2.0 Migration Guide](https://circleci.com/docs/2.0/migrating-from-1-2/).

This incidentally fixes the pip failures on recent CircleCI runs failing (#1299 #1301 etc) with the following error

`_NamespacePath object has no attribute sort `


Signed-off-by: Colin Schoen <cschoen@berkeley.edu>

@colinschoen colinschoen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to merge this... here is the stamp. I'll let you merge in case you want someone else from Microsoft to review.

@c-w

c-w commented Jul 3, 2018 •

Copy link
Copy Markdown
Contributor Author

@marrobi @martinpeck Would you like to review this or are you happy to merge this as-is? There's an architecture diagram here:

Azure PaaS Architecture

c-w and others added 2 commits July 3, 2018 15:39
Previously we were hosting the ok-server, ok-worker, autopy-manager and
autopy-worker in AppService for containers. However, we were seeing some
issues with this approach as workers were killed without death
notifications and rq/Redis was getting confused. After this change, we
now deploy the compute (workers and servers) to Azure Kubernetes
service.

In order to make it easier to orchestrate the setup of the various
resources in Azure as well as the Kubernetes deployments, we're using
bash scripts. First, we create a bunch of resources using `setup.sh` and
then we configure them using `install.sh`. The setup scripts are wrapped
in a Docker container so that the environment in which the scripts
execute is well known, e.g. Azure CLI is installed, helm is available,
kubectl is available, etc.

Most of the setup scripts fall-back to ARM templates to create the
resources in Azure. The ARM templates can be modified to configure
parameters such as resource SKUs, number of VMs to spin up for the
cluster, type of VMs to use in the cluster, etc.

The install scripts perform operations such as creating the database
schema, deploying the application to Kubernetes via Helm and setting up
the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego
can configure a TLS certificate for the ingress via Let's Encrypt.

Note that the ingress to the applications running in the Kubernetes
cluster are configured as just one entrypoint so that a single TLS
certificate and DNS entry can be used (Azure public IP resource
currently only support a single DNS label per IP). To solve routing, we
thus need to mount the two sub-applications (OKpy and AutoPY) at
sub-urls of the ingress which requires some minor application changes
which are submitted in a separate pull request:
#1300

The setup scripts can be run either via the "Deploy to Azure" button (in
which case the setup Docker container is run in an Azure Container
Instance) or via a local `docker run` command. In the latter case, it's
advised to also include a volume mount `docker run -v $PWD:/app/secrets`
so that the access keys for the resources created during the deployment
will be made available to the deployer. Note that in order to skip
deploying specific resources (e.g. to re-use existing resources), you
can place a file into the secrets folder that contains the access keys
for your existing resources in which case the setup scripts will skip
deploying that step.
@c-w

c-w commented Jul 6, 2018

Copy link
Copy Markdown
Contributor Author

@colinschoen Let's merge this. Thanks!

@Sumukh
Sumukh merged commit e630eb7 into okpy:master Jul 6, 2018
@c-w
c-w deleted the bug/c-w/azure-aks branch July 6, 2018 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants