Merge pull request #3 from xlab-si/feature/chef-bootstrap

Add Chef based bootstrap resources

bootstrap/README.md

@@ -0,0 +1,137 @@
+Deploying DMon
+==================
+
+This document describes two alternative ways of deploying DMON:
+
+* [Using Vagrant](#vagrant-deployment)
+* [Using Chef](#chef-deployment)
+* [Using Cloudify](#cloudify-deployment)
+
+
+Vagrant deployment
+------------------
+
+This is the easiest way of getting DMon up and running. Make sure you have
+Vagrant and VirtualBox installed and then execute
+`vagrant up --provider virtualbox`. This command will create new virtual
+machine and install DMon onto it.
+
+When the process is done, visit `localhost:5001` to get to the DMon's rest
+service. Another end point that is exposed is `localhost:5601` that points to
+Kibana's user interface.
+
+
+Chef deployment
+---------------
+
+In a dedicated Ubuntu 14.04 host, first install the
+[Chef client](https://downloads.chef.io/chef#ubuntu), e.g.:
+
+```bash
+$ wget https://packages.chef.io/files/stable/chef/12.18.31/ubuntu/14.04/chef_12.18.31-1_amd64.deb
+$ sudo dpkg -i chef_12.18.31-1_amd64.deb
+```
+
+Then obtain this cookbook repository:
+
+```bash
+$ git clone https://github.com/dice-project/DICE-Chef-Repository.git
+$ cd DICE-Chef-Repository
+4 git checkout develop
+```
+
+Before we run the installation, we need to provide the configuration of the
+DMon to be bootstrapped. We name the configuration file as `dmon.json` and
+populate it with the following contents:
+
+```json
+{
+  "java": {
+    "jdk_version": "8",
+    "install_flavor": "openjdk"
+  },
+  "cloudify": {
+    "node_id": "dmon-node",
+    "deployment_id": "dmon-deploy-id"
+  },
+  "dmon": {
+    "openssl_conf": "[req]\ndistinguished_name = req_distinguished_name\nx509_extensions = v3_req\nprompt = no\n[req_distinguished_name]\nC = SL\nST = Slovenia\nL =  Ljubljana\nO = Xlab\nCN = *\n[v3_req]\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nbasicConstraints = CA:TRUE\nsubjectAltName = IP:0.0.0.0\n[v3_ca]\nkeyUsage = digitalSignature, keyEncipherment\nsubjectAltName = IP:0.0.0.0\n"
+  }
+}
+```
+
+Then use Chef client in its zero mode to execute the recipes:
+
+```bash
+$ sudo chef-client -z \
+  -j dmon.json \
+  -o recipe[dice_common::host],recipe[apt::default],recipe[java::default],recipe[dmon::default],recipe[dmon::elasticsearch],recipe[dmon::kibana],recipe[dmon::logstash]
+```
+
+
+Cloudify deployment
+-------------------
+
+This process will create a new node in the target platform (FCO or OpenStack)
+and install the whole DMon stack on top of it. It requires a Cloudify Manager
+to be installed at the `CFY_MANAGER_HOST` address.
+
+### Preparing environment
+
+At the workstation node (i.e., our laptop, desktop PC where we install from),
+we need to have the Cloudify Manager CLI installed. The following steps
+are based on the [official documentation][CloudifyManagerBootstrap]:
+
+For Redhat related GNU/Linux distributions, following packages need to be
+installed: `python-virtualenv` and `python-devel`. Adjust properly for
+Ubuntu and the like.
+
+Now create new folder, create new python virtual environment and install
+`cloudify` package.
+
+    $ mkdir -p ~/dice && cd ~/dice
+    $ virtualenv venv
+    $ . venv/bin/activate
+    $ pip install cloudify==3.4.0
+
+Next we change to the directory containing the deployment blueprint and
+connect the Cloudify CLI client to the Cloudify Manager. Note that
+for the secured Cloudify Manager, we need to set the credentials in the
+environment variables `CLOUDIFY_USERNAME` and `CLOUDIFY_PASSWORD`.
+
+    $ cd ~/IeAT-DICE-Repository/bootstrap
+    $ export CLOUDIFY_USERNAME=admin
+    $ export CLOUDIFY_PASSWORD='OurCfyMngPassword'
+    $ cfy -t $CFY_MANAGER_HOST
+
+[CloudifyManagerBootstrap]:http://docs.getcloudify.org/3.4.0/manager/bootstrapping/
+
+### Preparing inputs
+
+The blueprint deployment needs a few parameters to be specified at this point.
+Use an `inputs-$PLATFORM.example.yaml` for your platform as a template to fill
+in, e.g., for the OpenStack:
+
+    $ cp inputs-openstack.example.yaml inputs-openstack.yaml
+
+Use a text editor to replace the values set in the inputs template with the
+values that will apply to your deploy. To do this, follow the comments in the
+`inputs-openstack.yaml` file.
+
+### Executing deployment
+
+To run the deployment of the DMon blueprint, use convenience scripts (which, in
+turn, call `cfy`):
+
+    $ ./up.sh openstack dmon-main
+
+Here, `openstack` is the target platform, and the script will use this name to
+choose the blueprint file (`openstack.yaml`) and the inputs file
+(`inputs-openstack.yaml`). The `dmon-main` string names the deployment in the
+Cloudify Manager.
+
+### Removing deployment
+
+The DMon deployment can be uninstalled using the following call:
+
+    $ ./dw.sh dmon-main

bootstrap/Vagrantfile

@@ -0,0 +1,15 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu/trusty64"
+  config.vm.network "forwarded_port", guest: 5001, host: 5001
+  config.vm.network "forwarded_port", guest: 5601, host: 5601
+
+  config.vm.provider "virtualbox" do |vb|
+    vb.memory = 4096
+    vb.cpus = 2
+  end
+
+  config.vm.provision "shell", path: "provision.sh"
+end

bootstrap/common/dmon.yaml

@@ -0,0 +1,100 @@
+inputs:
+
+  cluster_name:
+    description: >
+      The name of the monitoring cluster, used in elacticsearch and
+      logstash.
+    default: diceMonitoringCluster
+
+  # optional: provide either openssl_conf, or both lsf_cert and lsf_key
+  openssl_conf:
+    description: >
+      OpenSSL configuration contents for an auto-generated self-signed
+      certificate to be used in logstash.
+    default: {}
+
+  # optional: provide either openssl_conf, or both lsf_cert and lsf_key
+  lsf_cert:
+    description: Contents of the certificate to be used in logstash.
+    default: {}
+
+  # optional: provide either openssl_conf, or both lsf_cert and lsf_key
+  lsf_key:
+    description: Contents of the private key to be used in logstash.
+    default: {}
+
+
+node_types:
+
+  dice.components.dmon.Core:
+    derived_from: dice.chef.SoftwareComponent
+    properties:
+      create_runlist:
+        default:
+          - recipe[dice_common::host]
+          - recipe[apt::default]
+          - recipe[java::default]
+          - recipe[dmon::default]
+          - recipe[dmon::elasticsearch]
+          - recipe[dmon::kibana]
+          - recipe[dmon::logstash]
+      chef_attributes:
+        default:
+          java:
+            jdk_version: "8"
+            install_flavor: openjdk
+          dmon:
+            lsf_cert:           { get_input: lsf_cert       }
+            lsf_key:            { get_input: lsf_key        }
+            openssl_conf:       { get_input: openssl_conf   }
+            es: { cluster_name: { get_input: cluster_name   } }
+            kb: { cluster_name: { get_input: cluster_name   } }
+            ls: { cluster_name: { get_input: cluster_name   } }
+
+  dice.firewall_rules.dmon.Core:
+    derived_from: dice.firewall_rules.Base
+    properties:
+      rules:
+        default:
+          - remote_ip_prefix: 0.0.0.0/0
+            port_range_min: 5000
+            port_range_max: 5002
+          - remote_ip_prefix: 0.0.0.0/0
+            port: 5601
+          - remote_ip_prefix: 0.0.0.0/0
+            port: 25826
+            protocol: udp
+
+
+node_templates:
+
+  dmon_ip:
+    type: dice.VirtualIP
+
+  dmon_firewall:
+    type: dice.firewall_rules.dmon.Core
+
+  dmon_vm:
+    type: dice.hosts.Large
+    relationships:
+      - type: dice.relationships.ProtectedBy
+        target: dmon_firewall
+      - type: dice.relationships.IPAvailableFrom
+        target: dmon_ip
+
+  dmon_service:
+    type: dice.components.dmon.Core
+    relationships:
+      - type: dice.relationships.ContainedIn
+        target: dmon_vm
+
+
+outputs:
+
+  kibana_url:
+    description: Address of the Kibana web interface
+    value:
+      concat:
+       - "http://"
+       - { get_attribute: [ dmon_ip, floating_ip_address ] }
+       - ":5601"

bootstrap/dmon.json

@@ -0,0 +1,13 @@
+{
+  "java": {
+    "jdk_version": "8",
+    "install_flavor": "openjdk"
+  },
+  "cloudify": {
+    "node_id": "dmon-node",
+    "deployment_id": "dmon-deploy-id"
+  },
+  "dmon": {
+    "openssl_conf": "[req]\ndistinguished_name = req_distinguished_name\nx509_extensions = v3_req\nprompt = no\n[req_distinguished_name]\nC = SL\nST = Slovenia\nL =  Ljubljana\nO = Xlab\nCN = *\n[v3_req]\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nbasicConstraints = CA:TRUE\nsubjectAltName = IP:0.0.0.0\n[v3_ca]\nkeyUsage = digitalSignature, keyEncipherment\nsubjectAltName = IP:0.0.0.0\n"
+  }
+}

bootstrap/dw.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+DEPLOY_NAME=${1:-dmon}
+
+for EXEC_ID in $(cfy executions list -d $DEPLOY_NAME | grep started | awk '{print $2}')
+do
+	cfy executions cancel --execution-id $EXEC_ID
+
+	STATUS=$(cfy executions get -e $EXEC_ID | grep "| *$EXEC_ID" | awk '{print $6}')
+	while [ "$STATUS" != "cancelled" ]
+	do
+		sleep 3
+		STATUS=$(cfy executions get -e $EXEC_ID | grep "| *$EXEC_ID" | awk '{print $6}')
+	done
+done
+
+cfy executions start -d $DEPLOY_NAME -w uninstall
+cfy deployments delete -d $DEPLOY_NAME
+cfy blueprints delete -b $DEPLOY_NAME

bootstrap/fco.yaml

@@ -0,0 +1,6 @@
+# Installation blueprint for DICE Dmon
+tosca_definitions_version: cloudify_dsl_1_3
+
+imports:
+  - http://dice-project.github.io/DICE-Deployment-Cloudify/spec/fco/develop/plugin.yaml
+  - common/dmon.yaml

bootstrap/inputs-fco.example.yaml

@@ -0,0 +1,60 @@
+# FCO settings
+
+username: REPLACE_ME-UUID
+password: REPLACE_ME
+# UUID of the key that should be used by plugin.
+agent_key: 21e90e22-31c6-3d64-8590-af03dea25392
+# FCO customer (UUID).
+customer: e50bfd1b-253a-3290-85ff-95e218398b7e
+# FCO network (UUID).
+network: 050cb5ee-a8fd-3f33-8d83-b601460018c8
+# FCO VDC UUID
+vdc: 9799fe42-02ef-3929-88d4-c993a02cbe1d
+service_url: https://cp.diceproject.flexiant.net
+
+# DMON uses this to name the cluster
+cluster_name: diceMonitoringCluster
+# Change the C, ST, L, O in the following template.
+openssl_conf: |
+  [req]
+  distinguished_name = req_distinguished_name
+  x509_extensions = v3_req
+  prompt = no
+  [req_distinguished_name]
+  C = SL
+  ST = Slovenia
+  L =  Ljubljana
+  O = YourOrgLtd
+  CN = *
+  [v3_req]
+  subjectKeyIdentifier = hash
+  authorityKeyIdentifier = keyid,issuer
+  basicConstraints = CA:TRUE
+  subjectAltName = IP:0.0.0.0
+  [v3_ca]
+  keyUsage = digitalSignature, keyEncipherment
+  subjectAltName = IP:0.0.0.0
+# Alternatively, set lsf_cert and lsf_key to pre-existing certicate 
+# and key
+lsf_cert: {}
+lsf_key: {}
+
+# Agent user
+ubuntu_agent_user: ubuntu
+
+# VM characteristics
+ubuntu_image_id: 322f6b64-e341-3939-8b80-93d110db503f
+large_disk: "50 GB Storage Disk"
+large_server_type: "4 GB / 2 CPU"
+
+# the following inputs are not used, so dummy names are ok
+centos_agent_user: DONT_CARE
+centos_image_id: DONT_CARE
+dns_server: DONT_CARE
+medium_disk: DONT_CARE
+medium_image_id: DONT_CARE
+medium_server_type: DONT_CARE
+small_disk: DONT_CARE
+small_image_id: DONT_CARE
+small_server_type: DONT_CARE
+

bootstrap/inputs-openstack.example.yaml

@@ -0,0 +1,40 @@
+# DMON uses this to name the cluster
+cluster_name: diceMonitoringCluster
+# Change the C, ST, L, O in the following template.
+openssl_conf: |
+  [req]
+  distinguished_name = req_distinguished_name
+  x509_extensions = v3_req
+  prompt = no
+  [req_distinguished_name]
+  C = SL
+  ST = Slovenia
+  L =  Ljubljana
+  O = YourOrgLtd
+  CN = *
+  [v3_req]
+  subjectKeyIdentifier = hash
+  authorityKeyIdentifier = keyid,issuer
+  basicConstraints = CA:TRUE
+  subjectAltName = IP:0.0.0.0
+  [v3_ca]
+  keyUsage = digitalSignature, keyEncipherment
+  subjectAltName = IP:0.0.0.0
+# Alternatively, set lsf_cert and lsf_key to pre-existing certicate 
+# and key
+lsf_cert: {}
+lsf_key: {}
+
+# Agent user
+ubuntu_agent_user: ubuntu
+
+# VM characteristics
+ubuntu_image_id: ca290f2d-5163-483b-9dd5-fafe21517c0a
+large_flavor_id: 93e4960e-9b6d-454f-b422-0d50121b01c6
+
+# the following inputs are not used, so dummy names are ok
+centos_agent_user: DONT_CARE
+centos_image_id: DONT_CARE
+dns_server: DONT_CARE
+medium_flavor_id: DONT_CARE
+small_flavor_id: DONT_CARE