Merge branch 'develop'

user_documentation/prepare.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# To compile the documentation, one needs the packages to be installed.
+# To prepare compilation do the following:
+
+set -ex
+
+PDIR=env/doc
+
+rm -rf "$PDIR"
+
+# Create a virtual environment:
+virtualenv "$PDIR"
+
+# Activate it (this step is ALWAYS required before compiling)
+source "$PDIR"/bin/activate
+
+pip install --upgrade pip
+
+# Install the locally checked out packages:
+pip install -r requirements.txt 
+
+set +ex
+
+echo "Virtualenv created. To activate, use this command:"
+echo "source '$PDIR/bin/activate'"

user_documentation/rst/application_description.rst

@@ -1,3 +1,5 @@
+.. _applicationdescription:
+
 Application description
 ***********************
 
@@ -18,7 +20,7 @@ description:
    tosca_definitions_version: tosca_simple_yaml_1_0
 
    imports:
-     - https://raw.githubusercontent.com/micado-scale/tosca/v0.5.0/micado_types.yaml
+     - https://raw.githubusercontent.com/micado-scale/tosca/v0.6.0/micado_types.yaml
 
    repositories:
      docker_hub: https://hub.docker.com/
@@ -210,7 +212,7 @@ To instantiate MiCADO workers on a cloud through EC2 interface, please use the t
          properties:
            cloud:
              interface_cloud: ec2
-             endpoint_cloud: ADD_YOUR_ENDPOINT (e.g ec2.eu-west-1.amazonaws.com )
+             endpoint_cloud: ADD_YOUR_ENDPOINT (e.g https://ec2.eu-west-1.amazonaws.com )
          capabilities:
            host:
              properties:

user_documentation/rst/conf.py

@@ -39,6 +39,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+'sphinx.ext.intersphinx'
 ]
 
 # Add any paths that contain templates here, relative to this directory.

user_documentation/rst/dashboard.rst

@@ -1,7 +1,9 @@
+.. _dashboard:
+
 Dashboard
 *********
 
-MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``http://IP:4000``.
+MiCADO has a simple dashboard that collects web-based user interfaces into a single view. To access the Dashboard, visit ``https://[IP]:[port]``.
 
 The following webpages are currently exposed:
 

user_documentation/rst/deployment.rst

@@ -1,19 +1,33 @@
 Deployment
 **********
 
-As stated in the above section, to use MiCADO, you need to deploy the MiCADO services on a (separate) virtual machine, called MiCADO master. We recommend doing the installation remotely i.e. to download the Ansible playbook on your local machine and run the deployment on an empty virtual machine dedicated for this purpose on your preferred cloud.
+To deploy MiCADO you need a (separate) virtual machine, called MiCADO master. There are two ways of deployment:
+
+* remote: download the Ansible playbook on your local machine, configure the MiCADO master as target machine and run the playbook to perform the deployment remotely.
+* local: login to the MiCADO master, download the Ansible playbook, configure the localhost as target machine and run the playbook to perform the deployment locally.
+
+We recommend to perform the installation remotely as all your configuration files are preserved on your machine, i.e. it is easier to repeat the deployment if needed.
 
 Prerequisites
 =============
 
-Git & Ansible 2.4 or greater are needed on your (local) machine to run the Ansible playbook.
+For the MiCADO master:
+
+* Ubuntu 16.04
+
+For the host where the Ansible playbook is executed (differs depending on local or remote):
+
+* Ansible 2.4 or greater
+* Git
 
-**The version of Ansible in the Ubuntu 16.04 APT repository is outdated and insufficient**
 
 Ansible
 -------
 
-Install Ansible on Ubuntu 16.04.
+Note: Ansible in the Ubuntu 16.04 APT repository is outdated and insufficient (at the time of writing this document)
+
+To install Ansible on Ubuntu 16.04, use these commands:
+
 ::
 
    sudo apt-get update
@@ -22,14 +36,23 @@ Install Ansible on Ubuntu 16.04.
    sudo apt-get update
    sudo apt-get install ansible
 
-To install Ansible on other operation system follow the `official
-installation
-guide <#https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html>`__.
+To install Ansible on other operation system follow the `official installation guide <https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html>`__.
+
+Git
+---
+
+To install Git on Ubuntu, use this command:
+
+::
+
+   sudo apt-get install git-all
+
+To install Git on other operating system follow the `official installation guide <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`__.
 
 Installation
 ============
 
-Perform the following steps on your local machine.
+Perform the following steps either on your local machine or on MiCADO master depending on the installation method.
 
 Step 1: Download the ansible playbook.
 --------------------------------------
@@ -40,12 +63,12 @@ Currently, MiCADO v5 version is available.
 
    git clone https://github.com/micado-scale/ansible-micado.git ansible-micado
    cd ansible-micado
-   git checkout master
+   git checkout v0.6.0
 
 Step 2: Specify credential for instantiating MiCADO workers.
 ------------------------------------------------------------
 
-MiCADO master will use this credential to start/stop VM instances (MiCADO workers) to realize scaling. Credentials here should belong to the same cloud as where MiCADO master is running. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the credential in a file, called credentials.yml. Please, do not modify the structure of the template!
+MiCADO master will use this credential to start/stop VM instances (MiCADO workers) to host the application and to realize scaling. Credentials here should belong to the same cloud as where MiCADO master is running. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the credential in a file, called credentials.yml. Please, do not modify the structure of the template!
 
 ::
 
@@ -54,8 +77,41 @@ MiCADO master will use this credential to start/stop VM instances (MiCADO worker
 
 Edit credentials.yml to add cloud credentials. You will find predefined sections in the template for each cloud interface type MiCADO supports. Fill only the section belonging to your target cloud.
 
-Step 3: (Optional) Specify details of your private Docker repository.
----------------------------------------------------------------------
+Optionally you can use the `Ansible Vault <https://docs.ansible.com/ansible/2.4/vault.html>`_ mechanism to keep the credential data in an encrypted format. To achieve this, create the above file using Vault with the command
+
+::
+
+    ansible-vault create credentials.yml
+
+
+This will launch *vi* or the editor defined in the ``$EDITOR`` environment variable to make changes to the file. If you wish to make any changes to the previously encrypted file, you can use the command
+
+::
+
+    ansible-vault edit credentials.yml
+
+Step 3a: (Optional) Specify security settings and credentials.
+--------------------------------------------------------------
+
+MiCADO master will use these security-related settings and credentials during provisioning.
+
+::
+
+   cp sample-security-cred.yml security-cred.yml
+   vi security-cred.yml
+
+Specify the provisioning method for the x509 keypair used for TLS encryption of the management interface in the ``tls`` subtree:
+
+* The 'self-signed' option generates a new keypair with the specified hostname as subject (or 'micado-master' if omitted).
+* The 'user-supplied' option lets the user add the keypair as plain multiline strings (in unencrypted format) in the ansible_user_data.yml file under the 'cert' and 'key' subkeys respectively.
+
+Specify the default username and password for the administrative we user in the the ``authentication`` subtree.
+
+Optionally you may use the Ansible Vault mechanism as described in Step 2 to protect the confidentiality and integrity of this file as well.
+
+
+Step 3b: (Optional) Specify details of your private Docker repository.
+----------------------------------------------------------------------
 
 Set the Docker login credentials of your private Docker registries in which your personal containers are stored. We recommend making a copy of our predefined template and edit it. The ansible playbook expects the docker registry details in a file, called docker-cred.yml. Please, do not modify the structure of the template!
 
@@ -66,16 +122,22 @@ Set the Docker login credentials of your private Docker registries in which your
 
 Edit docker-cred.yml and add username, password, and repository url. To login to the default docker_hub, leave DOCKER_REPO as is (a blank string).
 
+Optionally you may use the Ansible Vault mechanism as described in Step 2 to protect the confidentiality and integrity of this file as well.
+
 Step 4: Launch an empty cloud VM instance for MiCADO master.
 ------------------------------------------------------------
 
 This new VM will host the MiCADO master core services. Use any of aws, ec2, nova, etc command-line tools or web interface of your target cloud to launch a new VM. We recommend a VM with 2 cores, 4GB RAM, 20GB disk. Make sure you can ssh to it (password-free i.e. ssh public key is deployed) and your user is able to sudo (to install MiCADO as root). Store its IP address which will be referred as ``IP`` in the following steps. The following ports should be open on the virtual machine:
 
 ::
 
-   TCP: 22,2377,3000,4000,5000,5050,7946,8080,8300,8301,8302,8500,8600,9090,9093,12345
+   TCP: 22,2377,7946,8300,8301,8302,8500,8600,[web_listening_port]
    UDP: 4789,7946,8301,8302,8600
 
+**NOTE:** ``web_listening_port`` is defined in the ansible inventory file called ``hosts`` and defaults to ``443``
+
+**NOTE:** MiCADO master has built-in firewall, therefore you can leave all ports open at cloud level.
+
 Step 5: Customize the inventory file for the MiCADO master.
 -----------------------------------------------------------
 
@@ -86,7 +148,18 @@ We recommend making a copy of our predefined template and edit it. Use the templ
    cp sample-hosts hosts
    vi hosts
 
-Edit the ``hosts`` file to set ansible variables for MiCADO master machine. Update the following parameters: ansible_host=\ *IP*, ansible_connection=\ *ssh* and ansible_user=\ *YOUR SUDOER ACCOUNT*. Please, revise the other parameters as well, however in most cases the default values are correct.
+Edit the ``hosts`` file to set ansible variables for MiCADO master machine. Update the following parameters:
+
+* **ansible_host**: specifies the publicly reachable ip address of MiCADO master. Set the public or floating ip of the master regardless the deployment method is remote or local. The ip specified here is used by the Dashboard for webpage redirection as well
+* **ansible_connection**: specifies how the target host can be reached. Use "ssh" for remote or "local" for local installation. In case of remote installation, make sure you can authenticate yourself against MiCADO master. We recommend to deploy your public ssh key on MiCADO master before starting the deployment
+* **ansible_user**: specifies the name of your sudoer account, defaults to "ubuntu"
+* **ansible_become**: specifies if account change is needed to become root, defaults to "True"
+* **ansible_become_method**: specifies which command to use to become superuser, defaults to "sudo"
+* **ansible_python_interpreter**: specifies the interpreter to be used for ansible on the target host, defaults to "/usr/bin/python3"
+* **docker_cred_path**: sets the path of file storing the credentials for private docker registries, defaults to "./docker-cred.yml"
+* **web_listening_port**: specifies the listening port of the management interface including the MiCADO dashboard and the REST interface, defaults to the default HTTPS port (443/TCP)
+
+Please, revise all the parameters, however in most cases the default values are correct.
 
 Step 6: Start the installation of MiCADO master.
 ------------------------------------------------
@@ -95,21 +168,31 @@ Step 6: Start the installation of MiCADO master.
 
    ansible-playbook -i hosts micado-master.yml
 
-Health checking
-===============
+If you have used Vault to encrypt your credentials, you have to add the path to your vault credentials to the command line as described in the `Ansible Vault documentation <https://docs.ansible.com/ansible/2.4/vault.html#providing-vault-passwords>`_ or provide it via command line using the command
+::
 
-At the end of the deployment, core MiCADO services will be running on the MiCADO master machine. Here are the commands to test the operation of some of the core MiCADO services:
+    ansible-playbook -i hosts micado-master.yml --ask-vault-pass
 
-*  Occopus:
-::
+After deployment
+================
 
-    curl -s -X GET http://IP:5000/infrastructures/
-*  Prometheus:
-::
+Once the deployment has successfully finished, you can proceed with
 
-    curl -s http://IP:9090/api/v1/status/config | jq '.status'
+* visiting the :ref:`dashboard`
+* using the :ref:`restapi`
+* playing with the :ref:`tutorials`
+* creating your :ref:`applicationdescription`
 
 Check the logs
 ==============
 
-Alternatively, you can SSH into MiCADO master and check the logs at any point after MiCADO is succesfully deployed. All logs are kept under ``/var/log/micado`` and are organised by component. Scaling decisions, for example, can be inspected under ``/var/log/micado/policykeeper``
+You can SSH into MiCADO master and check the logs at any point after MiCADO is succesfully deployed. All logs are kept under ``/var/log/micado`` and are organised by components. Scaling decisions, for example, can be inspected under ``/var/log/micado/policykeeper``
+
+Accessing user service
+======================
+
+In case your application contains container exposing a service, you have two alternatives to access its endpoint:
+
+* via MiCADO master: open up your service port number on the MiCADO master's internal firewall before deployment. To do that, extend the firewall configuration by editing the file(s) located at in the ``roles/micado-master/templates/iptables`` directory. Make sure you open up the cloud firewall as well for the MiCADO master!
+
+* via MiCADO worker: query the ip address of the worker nodes. You can do that through the Dashboard of MiCADO, the Dashboard of your cloud or the REST API of MiCADO. Make sure the port of your service is open up by the cloud firewall for the MiCADO workers!

user_documentation/rst/release_notes.rst

@@ -1,11 +1,50 @@
 Release Notes
 *************
 
+**v0.6.0 (10 Sept 2018)**
+
+- introduce documentation repository and host its content at http://micado-scale.readthedocs.io
+- improve MiCADO master containers restart policy
+- fix MTU issue in relation to Docker
+- fix Occopus restart issue
+- fix health-checking for Cloudbroker-AWS platform
+- update host naming convention for worker and master nodes
+- make wait-update task idempotent in ansible playbook
+- fix issue with worker node deployment in EC2 clouds
+- fix issue with user-defined Docker networks in OpenStack clouds
+- make Submitter response message structure uniform
+- add 'nodes' and 'services' query methods to REST API 
+- improve 'stressng' and 'cqueue' test helper scripts
+- add more compose properties to custom TOSCA definition
+- fix floating ip issues in the Dashboard component
+- add new links to Dashboard to reflect the changes introduced by reverse proxying
+- fix Dashboard to generate links based on the contents of the Host header to find the frontend URL automatically
+- make consul security encryption based on generated random key instead of static key
+- add reverse proxy, TLS encryption and application-level firewalling capabilities to the web interfaces exposed by the MiCADO master node
+- add packet filtering for closing down non-public ports
+- add systemd unit for MiCADO services
+- update the ansible playbook to use the built-in service module for installing and handling MiCADO services
+- update the documentation to reflect the changes after the introduction of reverse proxying
+- add support for form-based authentication of exposed web services
+- add COLA-themed login page
+- add the Credential Manager component to store and handle web service users and passwords securely
+- add support for provisioning a user to the Credential Manager via Ansible
+- add support for user and admin roles in the Credential Manager
+- add support for authorization of the web services based on user role
+- add documentation about the Ansible Vault mechanism to protect sensitive deployment details
+- add support for HTTP basic authentication for APIs
+- add support for making the web interface's listening port configurable
+- update the documentation of API calls in terms of authentication, encryption and reverse proxying
+- add micadoctl tool for user and service management
+- add HTTP method filter to firewall in order to control requests directed to containers
+- add support for IPv6 exposure of services
+- add IPv6 packet filtering
+
 **v0.5.0 (12 July 2018)**
 
-* Introduce supporting TOSCA
-* Introduce supporting user-defined scaling policy
-* Dashboard added with Docker Visualizer, Grafana, Prometheus
-* Deployment with Ansible playbook
-* Support private docker registry
-* Improve persistence of MiCADO master services
+- introduce supporting TOSCA
+- introduce supporting user-defined scaling policy
+- dashboard added with Docker Visualizer, Grafana, Prometheus
+- deployment with Ansible playbook
+- support private docker registry
+- improve persistence of MiCADO master services