Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch Azure deployment to Kubernetes #1301

Merged
merged 2 commits into from Jul 6, 2018

Conversation

Projects
None yet
3 participants
@c-w
Copy link
Contributor

commented Jun 29, 2018

Previously we were hosting the ok-server, ok-worker, autopy-manager and
autopy-worker in AppService for containers. However, we were seeing some
issues with this approach as workers were killed without death
notifications and rq/Redis was getting confused. After this change, we
now deploy the compute (workers and servers) to Azure Kubernetes
service.

In order to make it easier to orchestrate the setup of the various
resources in Azure as well as the Kubernetes deployments, we're using
bash scripts. First, we create a bunch of resources using setup.sh and
then we configure them using install.sh. The setup scripts are wrapped
in a Docker container so that the environment in which the scripts
execute is well known, e.g. Azure CLI is installed, helm is available,
kubectl is available, etc.

Most of the setup scripts fall-back to ARM templates to create the
resources in Azure. The ARM templates can be modified to configure
parameters such as resource SKUs, number of VMs to spin up for the
cluster, type of VMs to use in the cluster, etc.

The install scripts perform operations such as creating the database
schema, deploying the application to Kubernetes via Helm and setting up
the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego
can configure a TLS certificate for the ingress via Let's Encrypt.

Note that the ingress to the applications running in the Kubernetes
cluster are configured as just one entrypoint so that a single TLS
certificate and DNS entry can be used (Azure public IP resource
currently only support a single DNS label per IP). To solve routing, we
thus need to mount the two sub-applications (OKpy and AutoPY) at
sub-urls of the ingress which requires some minor application changes
which are submitted in a separate pull request:
#1300

The setup scripts can be run either via the "Deploy to Azure" button (in
which case the setup Docker container is run in an Azure Container
Instance) or via a local docker run command. In the latter case, it's
advised to also include a volume mount docker run -v $PWD:/app/secrets
so that the access keys for the resources created during the deployment
will be made available to the deployer. Note that in order to skip
deploying specific resources (e.g. to re-use existing resources), you
can place a file into the secrets folder that contains the access keys
for your existing resources in which case the setup scripts will skip
deploying that step.

Sumukh added a commit that referenced this pull request Jul 2, 2018

Upgrade to CircleCi version 2 (#1302)
CircleCI version `1.0` is deprecated and will be start rejecting jobs on August 30th. This PR migrates our workflow to `2.0`. This loosely follows the the [1.0 -> 2.0 Migration Guide](https://circleci.com/docs/2.0/migrating-from-1-2/).

This incidentally fixes the pip failures on recent CircleCI runs failing (#1299 #1301 etc) with the following error

`_NamespacePath object has no attribute sort `


Signed-off-by: Colin Schoen <cschoen@berkeley.edu>
@colinschoen
Copy link
Member

left a comment

Happy to merge this... here is the stamp. I'll let you merge in case you want someone else from Microsoft to review.

@c-w

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2018

@marrobi @martinpeck Would you like to review this or are you happy to merge this as-is? There's an architecture diagram here:

Azure PaaS Architecture

c-w and others added some commits Jun 27, 2018

Switch Azure deployment to Kubernetes
Previously we were hosting the ok-server, ok-worker, autopy-manager and
autopy-worker in AppService for containers. However, we were seeing some
issues with this approach as workers were killed without death
notifications and rq/Redis was getting confused. After this change, we
now deploy the compute (workers and servers) to Azure Kubernetes
service.

In order to make it easier to orchestrate the setup of the various
resources in Azure as well as the Kubernetes deployments, we're using
bash scripts. First, we create a bunch of resources using `setup.sh` and
then we configure them using `install.sh`. The setup scripts are wrapped
in a Docker container so that the environment in which the scripts
execute is well known, e.g. Azure CLI is installed, helm is available,
kubectl is available, etc.

Most of the setup scripts fall-back to ARM templates to create the
resources in Azure. The ARM templates can be modified to configure
parameters such as resource SKUs, number of VMs to spin up for the
cluster, type of VMs to use in the cluster, etc.

The install scripts perform operations such as creating the database
schema, deploying the application to Kubernetes via Helm and setting up
the DNS for the Kubernetes ingress via the Azure CLI so that kube-lego
can configure a TLS certificate for the ingress via Let's Encrypt.

Note that the ingress to the applications running in the Kubernetes
cluster are configured as just one entrypoint so that a single TLS
certificate and DNS entry can be used (Azure public IP resource
currently only support a single DNS label per IP). To solve routing, we
thus need to mount the two sub-applications (OKpy and AutoPY) at
sub-urls of the ingress which requires some minor application changes
which are submitted in a separate pull request:
#1300

The setup scripts can be run either via the "Deploy to Azure" button (in
which case the setup Docker container is run in an Azure Container
Instance) or via a local `docker run` command. In the latter case, it's
advised to also include a volume mount `docker run -v $PWD:/app/secrets`
so that the access keys for the resources created during the deployment
will be made available to the deployer. Note that in order to skip
deploying specific resources (e.g. to re-use existing resources), you
can place a file into the secrets folder that contains the access keys
for your existing resources in which case the setup scripts will skip
deploying that step.
@c-w

This comment has been minimized.

Copy link
Contributor Author

commented Jul 6, 2018

@colinschoen Let's merge this. Thanks!

@Sumukh Sumukh merged commit e630eb7 into okpy:master Jul 6, 2018

3 checks passed

Travis CI - Pull Request Build Passed
Details
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@c-w c-w deleted the c-w:bug/c-w/azure-aks branch Jul 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.