Automating infrastructure deployments for CCO-managed clusters

[pwalsh]

Job story

When I standup a cluster of CKANs managed by CCO, I want to be able to automate the required infrastructure deployment in a platform-agnostic way, so I have a consistent foundation for infrastructure resource management that is code-driven and does not require manual configuration steps.

Context

Currently, we have a range of manual steps that are under- or un-documented, to run CKAN Cloud, and much of these are related to standing up the k8s configuration required to run CCO within. Further, we now have CCO deployments on Azure, AWS and GCP, and each of these deployment targets has requirements for infrastructure configuration that we want to ensure are reproducible in code. A tool like Terraform may provide us with the necessary foundation to write platform-agnostic deployments for CCO infrastructure, at least targeting Azure, AWS, and GCP.

Acceptance criteria

• A CLI and a method of configuration for standing up k8s-managed clusters in which to run CCO
  • Prefer an existing tool like Terraform that is designed for this
  • Documented support for Azure, AWS, GCP
• Default values/configuration for this CLI as part of the CCO repository or else another repository for this specific purpose
• CCO Documentation that walks the user through exactly how this tool automates configuration of the required infrastructure, and describes the various configuration points available

[pwalsh]

Responsibilities of Terraform

In standing up an environment for managing a CKAN cluster, Terraform will have the following responsibilities:

• Standup k8s with default configuration of k8s-specific resources
• Standup external cloud services as appropriate for the provider
  • DB
  • Search
  • Cache
  • Object storage
• Standup and configure a DNS provider (cloud platform native, or cloudflare)
• Configure and deploy a service into the cluster that provides kubectl, cco and jenkins
  • Requires setting up dockerfile for this

In order to do this, the system administrator will have to have created a project (or similar) in advance on the cloud platform, so that Terraform has the appropriate user with permissions/keys to perform all required tasks.

We will ship a default terraform configuration with support for:

• AWS
• Azure
• GCP

And document ways to modify these defaults if needed.

[kimwdavidson]

Esteban to breakdown ticket

• reusable terraform piece
• make it work for GSA environment
• create an image of the management operator

[pwalsh]

@estebanruseler @kimwdavidson

• terraform (this ticket)
• make it work for GSA CI/CD for AWS deployments #78 (and/or, CI/CD for GCP deployments #80 but as we are still targeting AWS, then CI/CD for AWS deployments #78 for now)
• create an image Create (versioned) image for CCO and its direct dependencies #82

[kimwdavidson]

move to backlog

[pwalsh]

@akariv @zelima is this in a state for me to do acceptance testing yet?

[zelima]

We are still working on docs. Should have ready tomorrow