CI/CD for Azure deployments

[pwalsh]

Requires #78

WIP

We need to be able to set up a k8s cluster + everything around it (nfs, db, permissions) using terraform so that cco cluster initialize passes correctly.

Acceptance Criteria

• We are able to set up Azure K8 cluster using terraform
• We are able to do helm based deployment on that cluster
• We can access the deployed running CKAN instance from browser

Tasks

• Download and install terraform
• Read about terraform, play around with it, maybe deploy hello world somewhere, somehow
• Try to setup K8 environment manually first
  • Initialize Cluster
  • Deploy CKAN instance
• Try to setup K8 environment with terraform

Walkthrough

Download and install terraform (linux 64bit)

# Install terraform
wget -O terraform.zip https://releases.hashicorp.com/terraform/0.12.18/terraform_0.12.18_linux_amd64.zip &&\
unzip terraform.zip &&\
sudo mv terraform /usr/local/bin/ &&\
terraform

# Intall Azure CLI and login
curl -L https://aka.ms/InstallAzureCli | bash
az login

# Create AZ service Princibal (user with limited perms)
SUBSCRIPTION_ID=your_subscription_id_from_login
az account set --subscription="${SUBSCRIPTION_ID}"
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/${SUBSCRIPTION_ID}"

#!/bin/sh
echo "Setting environment variables for Terraform"
export ARM_SUBSCRIPTION_ID=your_subscription_id
export ARM_CLIENT_ID=your_appId
export ARM_CLIENT_SECRET=your_password
export ARM_TENANT_ID=your_tenant_id
export ARM_ENVIRONMENT=public

Analysis

• Terraform Docs: https://www.terraform.io/
• Getting started with k8 and terraform: https://www.terraform.io/docs/providers/kubernetes/guides/getting-started.html
• Azure - Terraform - K8: https://www.terraform.io/docs/providers/azurerm/r/kubernetes_cluster.html
• Create k8 cluster with on azure with terraform https://docs.microsoft.com/en-us/azure/terraform/terraform-create-k8s-cluster-with-tf-and-aks
• Terraform-Azure Hub - https://docs.microsoft.com/en-us/azure/terraform/

QA

• What are the manual steps?
• What are semi-manual steps?
• What info Do we need to start therefor? Eg superuser credentials from azure or similar...

[zelima]

On every PR/push to the branch, using travis and terraform, we are

• creating azure kubernetes cluster
• creating azuresql server and Database instance
• Initializing CKAN Cloud Cluster
• Deploying CKAN instance with helm based deployment using CCO
• destroying after fail/success

For now, we are not checking instance accessible on localhost. Successful deploy mean CKAN is up and running.

• The first successful build https://travis-ci.org/datopian/ckan-cloud-operator/builds/637377783
• PR Automate CKAN instance deployment on azure k8 cluster with terraform #105

[pwalsh]

@akariv @zelima is this in a state for me to do acceptance testing yet?

[zelima]

Still needs to be merged #105

[zelima]

FIXED