WIP: terraform openstack

contrib/terraform/openstack/README.md

@@ -0,0 +1,137 @@
+# Kubernetes on Openstack with Terraform
+
+Provision a Kubernetes cluster with [Terraform](https://www.terraform.io) on
+Openstack.
+
+## Status
+
+This will install a Kubernetes cluster on an Openstack Cloud. It is tested on a
+OpenStack Cloud provided by [BlueBox](https://www.blueboxcloud.com/) and
+should work on most modern installs of OpenStack that support the basic
+services.
+
+There are some assumptions made to try and ensure it will work on your openstack cluster.
+
+* floating-ips are used for access
+* you already have a suitable OS image in glance
+* you already have both an internal network and a floating-ip pool created
+* you have security-groups enabled
+
+
+## Requirements
+
+- [Install Terraform](https://www.terraform.io/intro/getting-started/install.html)
+
+## Terraform
+
+Terraform will be used to provision all of the OpenStack resources required to
+run Docker Swarm.   It is also used to deploy and provision the software
+requirements.
+
+### Prep
+
+#### OpenStack
+
+Ensure your OpenStack credentials are loaded in environment variables. This is
+how I do it:
+
+```
+$ source ~/.stackrc
+```
+
+You will need two networks before installing, an internal network and 
+an external (floating IP Pool) network. The internet network can be shared as
+we use security groups to provide network segregation. Due to the many
+differences between OpenStack installs the Terraform does not attempt to create
+these for you.
+
+By default Terraform will expect that your networks are called `internal` and
+`external`. You can change this by altering the Terraform variables `network_name` and `floatingip_pool`.
+
+A full list of variables you can change can be found at [variables.tf](variables.tf).
+
+All OpenStack resources will use the Terraform variable `cluster_name` (
+default `example`) in their name to make it easier to track. For example the
+first compute resource will be named `example-kubernetes-1`.
+
+#### Terraform
+
+Ensure your local ssh-agent is running and your ssh key has been added. This
+step is required by the terraform provisioner:
+
+```
+$ eval $(ssh-agent -s)
+$ ssh-add ~/.ssh/id_rsa
+```
+
+
+Ensure that you have your Openstack credentials loaded into Terraform
+environment variables. Likely via a command similar to:
+
+```
+$ echo Setting up Terraform creds && \
+  export TF_VAR_username=${OS_USERNAME} && \
+  export TF_VAR_password=${OS_PASSWORD} && \
+  export TF_VAR_tenant=${OS_TENANT_NAME} && \
+  export TF_VAR_auth_url=${OS_AUTH_URL}
+```
+
+# Provision a Kubernetes Cluster on OpenStack
+
+```
+terraform apply -state=contrib/terraform/openstack/terraform.tfstate contrib/terraform/openstack
+openstack_compute_secgroup_v2.k8s_master: Creating...
+  description: "" => "example - Kubernetes Master"
+  name:        "" => "example-k8s-master"
+  rule.#:      "" => "<computed>"
+...
+...
+Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
+
+The state of your infrastructure has been saved to the path
+below. This state is required to modify and destroy your
+infrastructure, so keep it safe. To inspect the complete state
+use the `terraform show` command.
+
+State path: contrib/terraform/openstack/terraform.tfstate
+```
+
+Make sure you can connect to the hosts:
+
+```
+$ ansible -i contrib/terraform/openstack/hosts -m ping all
+example-k8s_node-1 | SUCCESS => {
+    "changed": false, 
+    "ping": "pong"
+}
+example-etcd-1 | SUCCESS => {
+    "changed": false, 
+    "ping": "pong"
+}
+example-k8s-master-1 | SUCCESS => {
+    "changed": false, 
+    "ping": "pong"
+}
+```
+
+if it fails try to connect manually via SSH ... it could be somthing as simple as a stale host key.
+
+Deploy kubernetes:
+
+```
+$ ansible-playbook --become -i contrib/terraform/openstack/hosts cluster.yml
+```
+
+# clean up:
+
+```
+$ terraform destroy
+Do you really want to destroy?
+  Terraform will delete all your managed infrastructure.
+  There is no undo. Only 'yes' will be accepted to confirm.
+
+  Enter a value: yes
+...
+...
+Apply complete! Resources: 0 added, 0 changed, 12 destroyed.
+```

contrib/terraform/openstack/group_vars/all.yml

@@ -0,0 +1,136 @@
+# Directory where the binaries will be installed
+bin_dir: /usr/local/bin
+
+# Where the binaries will be downloaded.
+# Note: ensure that you've enough disk space (about 1G)
+local_release_dir: "/tmp/releases"
+
+# Uncomment this line for CoreOS only.
+# Directory where python binary is installed
+# ansible_python_interpreter: "/opt/bin/python"
+
+# This is the group that the cert creation scripts chgrp the
+# cert files to. Not really changable...
+kube_cert_group: kube-cert
+
+# Cluster Loglevel configuration
+kube_log_level: 2
+
+# Users to create for basic auth in Kubernetes API via HTTP
+kube_api_pwd: "changeme"
+kube_users:
+  kube:
+    pass: "{{kube_api_pwd}}"
+    role: admin
+  root:
+    pass: "changeme"
+    role: admin
+
+# Kubernetes cluster name, also will be used as DNS domain
+cluster_name: cluster.local
+
+# For some environments, each node has a pubilcally accessible
+# address and an address it should bind services to.  These are
+# really inventory level variables, but described here for consistency.
+#
+# When advertising access, the access_ip will be used, but will defer to
+# ip and then the default ansible ip when unspecified.
+#
+# When binding to restrict access, the ip variable will be used, but will
+# defer to the default ansible ip when unspecified.
+#
+# The ip variable is used for specific address binding, e.g. listen address
+# for etcd.  This is use to help with environments like Vagrant or multi-nic
+# systems where one address should be preferred over another.
+# ip: 10.2.2.2
+#
+# The access_ip variable is used to define how other nodes should access
+# the node.  This is used in flannel to allow other flannel nodes to see
+# this node for example.  The access_ip is really useful AWS and Google
+# environments where the nodes are accessed remotely by the "public" ip,
+# but don't know about that address themselves.
+# access_ip: 1.1.1.1
+
+# Choose network plugin (calico, weave or flannel)
+kube_network_plugin: flannel
+
+# Kubernetes internal network for services, unused block of space.
+kube_service_addresses: 10.233.0.0/18
+
+# internal network. When used, it will assign IP
+# addresses from this range to individual pods.
+# This network must be unused in your network infrastructure!
+kube_pods_subnet: 10.233.64.0/18
+
+# internal network total size (optional). This is the prefix of the
+# entire network. Must be unused in your environment.
+# kube_network_prefix: 18
+
+# internal network node size allocation (optional). This is the size allocated
+# to each node on your network.  With these defaults you should have
+# room for 4096 nodes with 254 pods per node.
+kube_network_node_prefix: 24
+
+# With calico it is possible to distributed routes with border routers of the datacenter.
+peer_with_router: false
+# Warning : enabling router peering will disable calico's default behavior ('node mesh').
+# The subnets of each nodes will be distributed by the datacenter router
+
+# The port the API Server will be listening on.
+kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
+kube_apiserver_port: 443 # (https)
+kube_apiserver_insecure_port: 8080 # (http)
+
+# Internal DNS configuration.
+# Kubernetes can create and mainatain its own DNS server to resolve service names
+# into appropriate IP addresses. It's highly advisable to run such DNS server,
+# as it greatly simplifies configuration of your applications - you can use
+# service names instead of magic environment variables.
+# You still must manually configure all your containers to use this DNS server,
+# Kubernetes won't do this for you (yet).
+
+# Upstream dns servers used by dnsmasq
+upstream_dns_servers:
+  - 8.8.8.8
+  - 8.8.4.4
+#
+# # Use dns server : https://github.com/ansibl8s/k8s-skydns/blob/master/skydns-README.md
+dns_setup: true
+dns_domain: "{{ cluster_name }}"
+#
+# # Ip address of the kubernetes skydns service
+skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
+dns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(2)|ipaddr('address') }}"
+
+# There are some changes specific to the cloud providers
+# for instance we need to encapsulate packets with some network plugins
+# If set the possible values are either 'gce', 'aws' or 'openstack'
+# When openstack is used make sure to source in the openstack credentials
+# like you would do when using nova-client before starting the playbook.
+# cloud_provider:
+
+# For multi masters architecture:
+# kube-proxy doesn't support multiple apiservers for the time being so you'll need to configure your own loadbalancer
+# This domain name will be inserted into the /etc/hosts file of all servers
+# configuration example with haproxy :
+# listen kubernetes-apiserver-https
+#   bind 10.99.0.21:8383
+#    option ssl-hello-chk
+#    mode tcp
+#    timeout client 3h
+#    timeout server 3h
+#    server master1 10.99.0.26:443
+#    server master2 10.99.0.27:443
+#    balance roundrobin
+# apiserver_loadbalancer_domain_name: "lb-apiserver.kubernetes.local"
+
+## Set these proxy values in order to update docker daemon to use proxies
+# http_proxy: ""
+# https_proxy: ""
+# no_proxy: ""
+
+## A string of extra options to pass to the docker daemon.
+## This string should be exactly as you wish it to appear.
+## An obvious use case is allowing insecure-registry access
+## to self hosted registries like so:
+docker_options: "--insecure-registry={{ kube_service_addresses }}"

contrib/terraform/openstack/hosts

@@ -0,0 +1 @@
+../terraform.py

contrib/terraform/openstack/kubespray.tf

@@ -0,0 +1,94 @@
+resource "openstack_networking_floatingip_v2" "k8s_master" {
+    count = "${var.number_of_k8s_masters}"
+    pool = "${var.floatingip_pool}"
+}
+
+resource "openstack_networking_floatingip_v2" "k8s_node" {
+    count = "${var.number_of_k8s_nodes}"
+    pool = "${var.floatingip_pool}"
+}
+
+
+resource "openstack_compute_keypair_v2" "k8s" {
+    name = "kubernetes-${var.cluster_name}"
+    public_key = "${file(var.public_key_path)}"
+}
+
+resource "openstack_compute_secgroup_v2" "k8s_master" {
+    name = "${var.cluster_name}-k8s-master"
+    description = "${var.cluster_name} - Kubernetes Master"
+}
+
+resource "openstack_compute_secgroup_v2" "k8s" {
+    name = "${var.cluster_name}-k8s"
+    description = "${var.cluster_name} - Kubernetes"
+    rule {
+        ip_protocol = "tcp"
+        from_port = "22"
+        to_port = "22"
+        cidr = "0.0.0.0/0"
+    }
+    rule {
+        ip_protocol = "icmp"
+        from_port = "-1"
+        to_port = "-1"
+        cidr = "0.0.0.0/0"
+    }
+    rule {
+        ip_protocol = "tcp"
+        from_port = "1"
+        to_port = "65535"
+        self = true
+    }
+    rule {
+        ip_protocol = "udp"
+        from_port = "1"
+        to_port = "65535"
+        self = true
+    }
+    rule {
+        ip_protocol = "icmp"
+        from_port = "-1"
+        to_port = "-1"
+        self = true
+    }
+}
+
+resource "openstack_compute_instance_v2" "k8s_master" {
+    name = "${var.cluster_name}-k8s-master-${count.index+1}"
+    count = "${var.number_of_k8s_masters}"
+    image_name = "${var.image}"
+    flavor_id = "${var.flavor_k8s_master}"
+    key_pair = "${openstack_compute_keypair_v2.k8s.name}"
+    network {
+        name = "${var.network_name}"
+    }
+    security_groups = [ "${openstack_compute_secgroup_v2.k8s_master.name}",
+                        "${openstack_compute_secgroup_v2.k8s.name}" ]
+    floating_ip = "${element(openstack_networking_floatingip_v2.k8s_master.*.address, count.index)}"
+    metadata = {
+        ssh_user = "${var.ssh_user}"
+        kubespray_groups = "etcd,kube-master,kube-node,k8s-cluster"
+    }
+}
+
+resource "openstack_compute_instance_v2" "k8s_node" {
+    name = "${var.cluster_name}-k8s-node-${count.index+1}"
+    count = "${var.number_of_k8s_nodes}"
+    image_name = "${var.image}"
+    flavor_id = "${var.flavor_k8s_node}"
+    key_pair = "${openstack_compute_keypair_v2.k8s.name}"
+    network {
+        name = "${var.network_name}"
+    }
+    security_groups = ["${openstack_compute_secgroup_v2.k8s.name}" ]
+    floating_ip = "${element(openstack_networking_floatingip_v2.k8s_node.*.address, count.index)}"
+    metadata = {
+        ssh_user = "${var.ssh_user}"
+        kubespray_groups = "kube-node,k8s-cluster"
+    }
+}
+
+#output "msg" {
+#    value = "Your hosts are ready to go!\nYour ssh hosts are: ${join(", ", openstack_networking_floatingip_v2.k8s_master.*.address )}"
+#}