terraform-aws-rke-rancher-master-cluster

Terraform module that creates an RKE cluster, meant to serve as nothing but a highly-available Rancher "master" cluster

Introduction

Purpose

The purpose of this module is to give an easy way to stand up a production-ready Rancher "master" cluster. It is intended to be a "turn-key" module, so it includes (almost) everything needed to have Rancher up and running, including the AWS compute infrastructure, Kubernetes cluster, load balancer, Route53 DNS entry, and the Rancher deployment itself.

High-level design

Resources provisioned

• 3 "node groups" of EC2 instances - gives you the ability to upgrade the AMI of one node group at a time so you can do an in-place upgrade
  • Does not use AutoScalingGroups (yet) - There's a bit of "chicken and egg" problem with the initial standup of a Rancher Server cluster. Worker clusters can use ASGs, but it isn't as easy to dynamically join instances to the master cluster
  • Currently creates Ubuntu nodes with Docker installed since that is what others that have come before have done, but the desire is to switch to CentOS with optional use of Red Hat Enterprise Linux (RHEL) because of its greater support for automated security tools that are commonly used in the federal government.
• A Kubernetes cluster installed on the EC2 instances
  • Uses the Terraform RKE provider
  • Labels all nodes with ["controlplane", "etcd", "worker"] - Remember this cluster should be used as the Rancher master cluster and nothing else
• A Classic Load Balancer (ELB) with listeners on port 80 and port 443 that points to port 80 and 443 of the cluster nodes
• 2 Security Groups
  • The nodes security group is used by the EC2 instances and allows:
    • Any traffic inside its own security group
    • SSH traffic from anywhere
    • K8s API traffic from anywhere
    • Traffic on ports 80 and 443 from the elb security group
  • The elb security group is used by the load balancer and allows:
    • Traffic on ports 80 and 443 from anywhere
• An AWS Key Pair with a new TLS private key
• A Route53 record that configures a dnsName to point at the ELB
• Uses a local-exec to helmfile apply CertManager and Rancher Server

Limitations

1. At the moment, this module cannot be deployed to private subnets. Deploying to private subnets can be added later if desired.

Usage

Prerequisites

1. Terraform v0.13+ - Uses the new way to pull down 3rd party providers.
2. *nix operating system - Windows not supported. If you need to use this on Windows you can run it from a Docker container.
3. Since this module uses a local-exec, the following tools also need to be installed on the machine using this module:
   1. kubectl
   2. helm
   3. helmfile
   4. helm-diff plugin

Instructions

Complete Example

See examples/complete for an example of how to use this module. For your convenience a Taskfile has been provided to be used with go-task.

cd examples/complete
task plan
task apply
task destroy

There are a few parameters that are specific to your AWS account and your domain name you want to use that are not included in the example terraform.tfvars. You should create a override.tfvars file and add the missing parameters to that.

Provider config

This module uses provider aliases, so you have to explicitly pass in provider configurations. Here's a minimum example:

provider "aws" {
  region = var.region
}

provider "random" {}

provider "tls" {}

provider "rke" {
  debug = true
}

provider "rancher2" {
  alias     = "bootstrap"
  api_url   = "https://${var.subdomain_rancher}.${var.hosted_zone}"
  insecure  = false
  bootstrap = true
}

module "rke_rancher_master_cluster" {
  source                          = "git::https://path/to/repo.git?ref=tags/x.y.z"
  additional_tag_map              = {}
  instance_type                   = var.instance_type
  kubernetes_version              = var.kubernetes_version
  name                            = var.name
  namespace                       = var.namespace
  node_group_1_subnet_id          = var.node_group_1_subnet_id
  node_group_2_subnet_id          = var.node_group_2_subnet_id
  node_group_3_subnet_id          = var.node_group_3_subnet_id
  node_volume_size                = var.node_volume_size
  stage                           = var.stage
  vpc_id                          = var.vpc_id
  hosted_zone                     = var.hosted_zone
  subdomain_rancher               = var.subdomain_rancher
  rancher_letsencrypt_email       = var.rancher_letsencrypt_email
  rancher_letsencrypt_environment = var.rancher_letsencrypt_environment
  providers = {
    aws                = aws
    random             = random
    tls                = tls
    rke                = rke
    rancher2.bootstrap = rancher2.bootstrap
  }
}

Logging into Rancher

The module outputs variables rancher_endpoint and rancher_admin_password. The username is admin. The admin password is managed by Terraform, don't change it manually.

Contributing

Contributors to this module should make themselves familiar with this section.

Prerequisites

• Terraform v0.13+
• pre-commit
• Pre-commit hook dependencies
  • nodejs (for the prettier hook)
  • tflint
  • terraform-docs
  • tfsec
• Run pre-commit install in root dir of repo (installs the pre-commit hooks so they run automatically when you try to do a git commit)

Using the ASDF version manager is highly encouraged. The project supports it by using a .tool-versions file to specify the versions of tools used and ensure that all necessary tools are installed.

See this Gist for a quick way to add a set of plugins that will work for this project

Versioning

This module will use SemVer, and will stay on v0.X for the foreseeable future

Requirements

Name	Version
terraform	>= 0.13.0
aws	>= 2.0.0
local	>= 1.0.0
null	>= 2.0.0
rancher2	>= 1.0.0
random	>= 2.0.0
rke	>= 1.0.0
template	>= 2.0.0
tls	>= 2.0.0

Providers

Name	Version
aws	>= 2.0.0
local	>= 1.0.0
null	>= 2.0.0
rancher2.bootstrap	>= 1.0.0
random	>= 2.0.0
rke	>= 1.0.0
tls	>= 2.0.0

Inputs

Name	Description	Type	Default	Required
additional_tag_map	Map of additional tags to apply to every taggable resource. If you don't want any use an empty map - '{}'	map(string)	n/a	yes
description	Short description of what/why this product exists	string	n/a	yes
hosted_zone_domain_name	Domain name of the hosted zone to create records in	string	n/a	yes
hosted_zone_id	ID of Route53 hosted zone to create records in	string	n/a	yes
instance_type	Instance type to use for the cluster nodes	string	n/a	yes
kubernetes_version	Kubernetes version to use. Must be supported by the version of the RKE provider you are using. See https://github.com/rancher/terraform-provider-rke/releases	string	n/a	yes
name	Solution name	string	n/a	yes
namespace	Namespace, which could be your organization name or abbreviation	string	n/a	yes
node_group_1_subnet_id	Subnet to deploy node group 1 to	string	n/a	yes
node_group_2_subnet_id	Subnet to deploy node group 2 to	string	n/a	yes
node_group_3_subnet_id	Subnet to deploy node group 3 to	string	n/a	yes
node_volume_size	Volume size of worker node disk in GB	string	n/a	yes
owner	Email address of owner	string	n/a	yes
rancher_letsencrypt_email	Email address to use for Rancher's LetsEncrypt certificate	string	n/a	yes
rancher_letsencrypt_environment	LetsEncrypt environment to use - Valid options: 'staging', 'production'	string	n/a	yes
repo	Repo URL that is responsible for this resource	string	n/a	yes
stage	Stage, e.g. 'prod', 'staging', 'dev'	string	n/a	yes
subdomain_rancher	Rancher's endpoint will be '{subdomain_rancher}.{hosted_zone_domain_name}'. {subdomain_rancher} can be multi-layered e.g. 'rancher.foo.bar'	string	n/a	yes
vpc_id	ID of the VPC to deploy to	string	n/a	yes

Outputs

Name	Description
cluster_kubeconfig	KUBECONFIG yaml file contents to connect to the cluster. DO NOT USE unless you have no other options. Users should use the KUBECONFIG that Rancher provides to them rather than this.
rancher_admin_password	Password for Rancher 'admin' user
rancher_admin_token	API Token for Rancher 'admin' user
rancher_endpoint	Endpoint of Rancher Server
ssh_private_key	Cluster nodes' private SSH key
ssh_public_key	Cluster nodes' public SSH key