Merge remote-tracking branch 'upstream/release/1.2.0' into 1.2.0-BETA1

* upstream/release/1.2.0: (72 commits) docs: 1.2 upgrade instructions elk: increase default resources (mantl#1569) docs: 1.2 rc1 changelog update (mantl#1581) kafka: create topics and cleanup config (mantl#1525) Add playbook to force consul leader election (mantl#948) updated functional IP descriptions traefik: fixes traefik ui Add memory usage restrictions to Docker containers (mantl#1451) traefik: package version 1.0.0 distrib-dnsmasq upgrade to 0.2.6 use global yum repository instead of targeting an rpm file upgrade to calico 0.19.0 use real vault 0.5.3 package use preliminary vault-0.5.3 package template the repos file Proper mantl version and repo destination Use versioned mantl bintray repositories remove test.mesos.yml. Fixes mantl#1522 Replace CiscoCloud logo with Mantl logo in docs (mantl#1470) kibana: use consistent val for elasticsearch-http ...
waldophotos · Jun 22, 2016 · 2ff84f8 · 2ff84f8
2 parents 7b5846c + c31d34b
commit 2ff84f8
Show file tree

Hide file tree

Showing 108 changed files with 1,455 additions and 512 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,74 @@
 # Changelog
 
+## 1.2 (June 21, 2016)
+
+* Mesos and Marathon improvements
+  * Upgrade Mesos to 0.28.2 and Marathon to 1.1.1 #1524 (https://github.com/CiscoCloud/mantl/issues/1524)
+  * Upgrade Mesos and Marathon #1514 (https://github.com/CiscoCloud/mantl/issues/1514)
+  * rotate only the actual log files for mesos #1529 (https://github.com/CiscoCloud/mantl/issues/1529)
+  * mesos logroate issues  #1511 (https://github.com/CiscoCloud/mantl/issues/1511)
+
+* Packaging improvements
+  * Mantl has a dedicated rpm repository per release #1466 (https://github.com/CiscoCloud/mantl/issues/1466)
+  * Mantl has a dedicated rpm repository per release #1555 (https://github.com/CiscoCloud/mantl/issues/1555)
+
+* Traefik Upgrade
+  * traefik: package version 1.0.0 #1559 (https://github.com/CiscoCloud/mantl/issues/1559)
+  * traefik: fixes traefik ui #1563 (https://github.com/CiscoCloud/mantl/issues/1563)
+
+* Mutli-cloud automated testing improvements
+  * TRAVIS DOCKER Install bash for provisioning #1499 (https://github.com/CiscoCloud/mantl/issues/1499)
+  * dockerfile: upgrade to terraform v0.6.16 #1533 (https://github.com/CiscoCloud/mantl/issues/1533)
+
+* Documentation improvements
+  * readme: revamp supported platform section #1509 (https://github.com/CiscoCloud/mantl/issues/1509)
+  * Replace CiscoCloud logo with Mantl logo in docs #1470 (https://github.com/CiscoCloud/mantl/issues/1470)
+  * updated functional IP descriptions #1567 (https://github.com/CiscoCloud/mantl/issues/1567)
+
+* AWS Provider Improvements
+  * Change default vpc name to mantl #1479 (https://github.com/CiscoCloud/mantl/issues/1479)
+
+* GCE Provider Improvements
+  * gce: short_name and long_name should default to "mantl" #1503 (https://github.com/CiscoCloud/mantl/issues/1503)
+  * gce: long_name and short_name should default to "mantl" #1502 (https://github.com/CiscoCloud/mantl/issues/1502)
+  * gce: default network short_name to mantl #1549 (https://github.com/CiscoCloud/mantl/issues/1549)
+
+* ELK Stack Improvements
+  * Split ELK role into standalone Elasticsearch and Kibana roles #1481 (https://github.com/CiscoCloud/mantl/issues/1481)
+  * inconsistent value for elasticsearch-http when using kibana-mesos #1545 (https://github.com/CiscoCloud/mantl/issues/1545)
+  * Consistent kibana elasticsearch-http value #1550 (https://github.com/CiscoCloud/mantl/issues/1550)
+  * elk: increase default resources #1569 (https://github.com/CiscoCloud/mantl/issues/1569)
+
+* Calico support
+  * Upgrade Calico to latest - 0.19.0 #1523 (https://github.com/CiscoCloud/mantl/issues/1523)
+  * Calico with Mesos and Kubernetes integrated #1521 (https://github.com/CiscoCloud/mantl/issues/1521)
+
+* Vault improvements
+  * Lock down vault configuration file #1494 (https://github.com/CiscoCloud/mantl/issues/1494)
+  * vault: verify server with TLS #1498 (https://github.com/CiscoCloud/mantl/issues/1498)
+  * Vault 0.5.3 #1510 (https://github.com/CiscoCloud/mantl/issues/1510)
+  * Upgrade to Vault 0.5 #1496 (https://github.com/CiscoCloud/mantl/issues/1496)
+
+* Kafka Addon Improvements
+  * kafka: create topics and cleanup config #1525 (https://github.com/CiscoCloud/mantl/issues/1525)
+  * Kafka enhancements #1468 (https://github.com/CiscoCloud/mantl/issues/1468)
+
+* Other fixes and enhancements
+  * reduce code duplication in nginx_proxy.yml #1526 (https://github.com/CiscoCloud/mantl/issues/1526)
+  * readme: add clc to supported platforms #1536 (https://github.com/CiscoCloud/mantl/issues/1536)
+  * Update vagrant to include kubeworkers and refator edge, worker loop #1365 (https://github.com/CiscoCloud/mantl/issues/1365)
+  * Remove mantl rename FAQ #1471 (https://github.com/CiscoCloud/mantl/issues/1471)
+  * List cloud-init-providers in a variable #1508 (https://github.com/CiscoCloud/mantl/issues/1508)
+  * Hardcode consul_dns_domain #1449 (https://github.com/CiscoCloud/mantl/issues/1449)
+  * Fix jupyter example - ws not a valid config for traefik #1547 (https://github.com/CiscoCloud/mantl/issues/1547)
+  * Ensure package removal url is only one line #1539 (https://github.com/CiscoCloud/mantl/issues/1539)
+  * Extraneous file "test.mesos.yml" #1522 (https://github.com/CiscoCloud/mantl/issues/1522)
+  * Add memory usage restrictions to Docker containers #1451 (https://github.com/CiscoCloud/mantl/issues/1451)
+  * Distributive-dnsmasq 0.2.6 #1558 (https://github.com/CiscoCloud/mantl/issues/1558)
+  * remove test.mesos.yml  #1552 (https://github.com/CiscoCloud/mantl/issues/1552)
+  * Add playbook to force consul leader election #948 (https://github.com/CiscoCloud/mantl/issues/948)
+  * Vagrant update to include Kube worker #1542 (https://github.com/CiscoCloud/mantl/issues/1542)
+
 ## 1.1 (May 13, 2016)
 
 ### Features and Improvements

diff --git a/Dockerfile b/Dockerfile
@@ -1,8 +1,7 @@
 FROM alpine:3.3
 
-RUN apk add --no-cache build-base curl git openssh openssl py-pip python python-dev unzip \
+RUN apk add --no-cache bash build-base curl git libffi-dev openssh openssl-dev py-pip python python-dev unzip \
 	&& git clone https://github.com/CiscoCloud/mantl /mantl \
-	&& apk add --no-cache build-base python-dev py-pip \
 	&& pip install -r /mantl/requirements.txt \
 	&& apk del build-base python-dev py-pip
 
@@ -11,7 +10,7 @@ ENV MANTL_CONFIG_DIR /local
 
 VOLUME /root/.ssh
 
-ENV TERRAFORM_VERSION 0.6.12
+ENV TERRAFORM_VERSION 0.6.16
 RUN mkdir -p /tmp/terraform/ && \
     cd /tmp/terraform/ && \
     curl -SLO https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \

diff --git a/README.md b/README.md
@@ -1,5 +1,4 @@
-
-![image](./mantl-logo-1.png)
+![image](./docs/_static/mantl-logo.png)
 
 # Overview
 
@@ -10,7 +9,7 @@
 Mantl is a modern, batteries included platform for rapidly deploying globally
 distributed services
 
-<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc/generate-toc again -->
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-generate-toc again -->
 **Table of Contents**
 
 - [Overview](#overview)
@@ -21,6 +20,7 @@ distributed services
         - [Architecture](#architecture)
         - [Control Nodes](#control-nodes)
         - [Resource Nodes](#resource-nodes)
+        - [Edge Nodes](#edge-nodes)
     - [Getting Started](#getting-started)
         - [Software Requirements](#software-requirements)
         - [Deploying on multiple servers](#deploying-on-multiple-servers)
@@ -29,12 +29,15 @@ distributed services
         - [Mesos Frameworks](#mesos-frameworks)
         - [Security](#security)
         - [Operations](#operations)
-        - [Platform Support](#platform-support)
+        - [Supported Platforms](#supported-platforms)
+            - [Community Supported Platforms](#community-supported-platforms)
     - [Development](#development)
+    - [Getting Support](#getting-support)
     - [License](#license)
 
 <!-- markdown-toc end -->
 
+
 ## Features
 
 ### Core Components
@@ -207,21 +210,27 @@ repo](https://github.com/CiscoCloud/mantl-universe).
 - [ ] Self maintaining system (log rotation, etc)
 - [ ] Self healing system (automatic failed instance replacement, etc)
 
-### Platform Support
-
-- [x] Vagrant (Mac OSX + VirtualBox)
-- [x] Vagrant (Linux + VirtualBox)
+### Supported Platforms
+
+- [x] [Amazon Web Services](https://aws.amazon.com/)
+- [x] [CenturyLinkCloud](https://www.ctl.io)
+- [x] [Cisco Cloud Services](http://www.cisco.com/c/en/us/solutions/cloud/overview.html)
+- [x] [Cisco MetaCloud](http://www.cisco.com/c/en/us/products/cloud-systems-management/metapod/index.html)
+- [x] [DigitalOcean](https://www.digitalocean.com/)
+- [x] [Joyent Triton](https://www.joyent.com/)
+- [x] [Google Compute Engine](https://cloud.google.com/compute/)
+- [x] [OpenStack](http://www.openstack.org/)
+- [x] [Vagrant](https://www.vagrantup.com/) (Linux/OSX + VirtualBox)
+- [ ] [Apache CloudStack](https://cloudstack.apache.org/)
+- [ ] [Cisco Unified Computing System](http://www.cisco.com/c/en/us/products/servers-unified-computing/index.html)
+- [ ] [Microsoft Azure](https://azure.microsoft.com/en-us/?b=16.17)
 - [ ] Vagrant (Windows + VirtualBox)
-- [x] OpenStack
-- [x] Cisco Cloud Services
-- [x] Cisco MetaCloud
-- [ ] Cisco Unified Computing System
-- [x] Amazon Web Services
-- [ ] Microsoft Azure
-- [x] Google Compute Engine
-- [ ] VMware vSphere
-- [ ] Apache CloudStack
-- [x] Digital Ocean
+
+#### Community Supported Platforms
+
+- [x] Bare Metal
+- [x] [IBM Softlayer](http://www.softlayer.com/)
+- [x] [VMware vSphere](http://www.vmware.com/products/vsphere/)
 
 Please see [milestones](https://github.com/CiscoCloud/mantl/milestones) for
 more details on the roadmap.

diff --git a/Vagrantfile b/Vagrantfile
@@ -4,15 +4,19 @@ require "yaml"
 
 config_hash = {
   "worker_count" => 1,
+  "kubeworker_count" => 1,
   "control_count" => 1,
-  "edge_count" => 0,
+  "edge_count" => 1,
+  "kubeworker_ip_start" => "192.168.100.15",
   "worker_ip_start" => "192.168.100.20",
   "control_ip_start" => "192.168.100.10",
   "edge_ip_start" => "192.168.100.25",
+  "kubeworker_memory" => 1024,
   "worker_memory" => 1024,
   "control_memory" => 1024,
   "edge_memory" => 512,
   "worker_cpus" => 1,
+  "kubeworker_cpus" => 1,
   "control_cpus" => 1,
   "edge_cpus" => 1,
   "network" => "private",
@@ -26,77 +30,59 @@ else
  config_hash = config_hash.merge(YAML.load(File.read(config_path)))
 end
 
-Vagrant.require_version ">= 1.8"
-
-Vagrant.configure(2) do |config|
-  # Prefer VirtualBox before VMware Fusion
-  config.vm.provider "virtualbox"
-  config.vm.provider "vmware_fusion"
-  config.vm.box = "CiscoCloud/mantl"
-
-  # Disable shared folder(s) for non-provisioning machines
-  config.vm.synced_folder ".", "/vagrant", disabled: true
-  config.vm.synced_folder ".", "/home/vagrant/sync", disabled: true
+# This method is used to spin up the workers, kubeworkers and edge nodes.
+def spin_up(config_hash:, config:, host_array:, hostvars:, hosts:, role:)
+  (1..config_hash["#{role}_count"]).each do |w|
+    hostname = "#{role}-00#{w}"
+    ip = config_hash["#{role}_ip_start"] + "#{w}"
 
-  # All hostvars will be stored in this hash, progressively as the VMs are made
-  # and configured. These lists will hold hostnames that belong to these groups.
-  hostvars, workers, controls, edges = {}, [], [], []
-  # This variable will be appended to the /etc/hosts file on the provisioner
-  hosts = ""
-
-  (1..config_hash["worker_count"]).each do |w|
-    hostname = "worker-00#{w}"
-    ip = config_hash["worker_ip_start"] + "#{w}"
-
-    config.vm.define hostname do |worker|
+    config.vm.define hostname do |server|
       # Tested with 2 workers w/ 512mb, and a single one w/ 1024mb memory.
-      worker.vm.provider :virtualbox do |vb|
-        vb.customize ["modifyvm", :id, "--cpus", config_hash["worker_cpus"]]
-        vb.customize ["modifyvm", :id, "--memory", config_hash["worker_memory"]]
+      server.vm.provider :virtualbox do |vb|
+        vb.customize ["modifyvm", :id, "--cpus", config_hash["#{role}_cpus"]]
+        vb.customize ["modifyvm", :id, "--memory", config_hash["#{role}_memory"]]
       end
-      worker.vm.hostname = hostname
-      worker.vm.network "#{config_hash['network']}_network", :ip => ip
+      server.vm.hostname = hostname
+      server.vm.network "#{config_hash['network']}_network", :ip => ip
 
       hosts += "#{ip}    #{hostname}\n"
       # Update Ansible variables
-      workers << hostname
-      worker_hostvars = {
+      host_array << hostname
+      server_hostvars = {
         hostname => {
           "ansible_ssh_host" => ip,
           "private_ipv4" => ip,
           "public_ipv4" => ip,
-          "role" => "worker"
+          "role" => role
         }
       }
-      hostvars.merge!(worker_hostvars)
+      hostvars.merge!(server_hostvars)
     end
   end
+end
 
-  (1..config_hash["edge_count"]).each do |e|
-    hostname = "edge-0#{e}"
-    ip = config_hash["edge_ip_start"] + "#{e}"
+Vagrant.require_version ">= 1.8"
 
-    config.vm.define hostname do |edge|
-      edge.vm.provider :virtualbox do |vb|
-        vb.customize ["modifyvm", :id, "--cpus", config_hash["edge_cpus"]]
-        vb.customize ["modifyvm", :id, "--memory", config_hash["edge_memory"]]
-      end
-      edge.vm.hostname = hostname
-      edge.vm.network "#{config_hash['network']}_network", :ip => ip
+Vagrant.configure(2) do |config|
+  # Prefer VirtualBox before VMware Fusion
+  config.vm.provider "virtualbox"
+  config.vm.provider "vmware_fusion"
+  config.vm.box = "CiscoCloud/mantl"
 
-      hosts += "#{ip}    #{hostname}\n"
-      edges << hostname
-      edge_hostvars = {
-        hostname => {
-          "ansible_ssh_host" => ip,
-          "private_ipv4" => ip,
-          "public_ipv4" => ip,
-          "role" => "edge"
-        }
-      }
-      hostvars.merge!(edge_hostvars)
-    end
-  end
+  # Disable shared folder(s) for non-provisioning machines
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder ".", "/home/vagrant/sync", disabled: true
+
+  # All hostvars will be stored in this hash, progressively as the VMs are made
+  # and configured. These lists will hold hostnames that belong to these groups.
+  hostvars, workers, controls, edges, kubeworkers = {}, [], [], [], []
+  # This variable will be appended to the /etc/hosts file on the provisioner
+  hosts = ""
+
+  # Spin up all the worker and edge nodes based on your config or default config.
+  spin_up(config_hash: config_hash, config: config, hostvars: hostvars, host_array: workers, hosts: hosts, role: 'worker')
+  spin_up(config_hash: config_hash, config: config, hostvars: hostvars, host_array: kubeworkers, hosts: hosts, role: 'kubeworker')
+  spin_up(config_hash: config_hash, config: config, hostvars: hostvars, host_array: edges, hosts: hosts, role: 'edge')
 
   (1..config_hash["control_count"]).each do |c|
     hostname = "control-0#{c}"
@@ -141,8 +127,10 @@ Vagrant.configure(2) do |config|
           "role=worker" => workers,
           "role=worker:vars" => { "consul_is_server" => false },
           "role=edge" => edges,
+          "role=kubeworker" => kubeworkers,
+          "role=kubeworker:vars" => { "consul_is_server" => false },
           "role=edge:vars" => { "consul_is_server" => false },
-          "dc=vagrantdc" => workers + controls + edges,
+          "dc=vagrantdc" => workers + kubeworkers + controls + edges,
           "dc=vagrantdc:vars" => {
             # Ansible 2.0 depreciates the 'ssh' in these vars
             "ansible_ssh_user" => "vagrant",

diff --git a/addons/elasticsearch.yml b/addons/elasticsearch.yml
@@ -0,0 +1,10 @@
+# Elasticsearch. Run this playbook after running the core playbook (sample.yml)
+
+# CHECK SECURITY - when customizing you should leave this in. If you take it out
+# and forget to specify security.yml, security could be turned off on components
+# in your cluster!
+- include: "{{ playbook_dir }}/../playbooks/check-requirements.yml"
+
+- hosts: all
+  roles:
+    - elasticsearch
diff --git a/addons/elk-uninstall.yml b/addons/elk-uninstall.yml
diff --git a/addons/elk.yml b/addons/elk.yml
@@ -7,9 +7,11 @@
 - include: "{{ playbook_dir }}/../playbooks/check-requirements.yml"
 
 - hosts: all
-  roles:
-    - logstash
-
-- hosts: role=control
+  vars:
+    elk_uninstall: false
+    kibana_uninstall: "{{ elk_uninstall|bool }}"
+    elasticsearch_uninstall: "{{ elk_uninstall|bool }}"
+    elasticsearch_remove_data: false
+    kibana_logstash_config: true
   roles:
     - elk