Skip to content

Commit

Permalink
update documentation, related to additional services
Browse files Browse the repository at this point in the history 
- mail service attach
- proxy service attach
- prometheus service attach
- cloud providers settings

Pay attention, than there is no Nginx-service default configuration, it depends on
#246
  • Loading branch information
@naumvd95
naumvd95 committed Mar 27, 2018
1 parent f7a6c6a commit 4edcc8b
Show file tree
Hide file tree
Showing 2 changed files with 222 additions and 14 deletions.
1 change: 0 additions & 1 deletion RATIONALE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,3 @@ KQueen supplies the backend API for provider-agnostic cluster management. It ena
* **Update** - install newer version of Kubernetes
* **Autoscale** - watch Kubernetes scheduler or pods and start new minions when all existing minions are fully utilized
* **Manage addons** - enable or disable cluster addons

235 changes: 222 additions & 13 deletions docs/kqueen.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -152,26 +152,235 @@ in the configuration file, set the environment variable matching the KQUEEN_<con
- 127.0.0.0/8
- Addresses allowed to access metrics endpoint without token

Before you provision a Kubernetes cluster, you may need to deploy and configure the following external services:

Backup
-------
* Mail server, to enable members management. For example, to enable a user to invite other users by email. You can use the KQueen predefined mail service or run a new one.

The only one statefull component of Kqueen is the etcd and users should follow https://coreos.com/etcd/docs/latest/v2/admin_guide.html#disaster-recovery. We are using `v2` etcd keys so example backup workflow can be:
* Nginx proxy server, to configure ssl support, domain naming and define certificates.

::
* Prometheus server, to extend the monitoring in KQueen. You can either use a predefined Prometheus server, defined in the docker-compose.production.yml file or use an external one. For an external one, you must include the rules from kqueen/prod/prometheus to the existing Prometheus service.

# Backup etcd to directory /root/backup/ (etcd data stored in /var/lib/etcd/default)
etcdctl backup --data-dir /var/lib/etcd/default --backup-dir /root/backup/

Recovery
To set up the nginx server
--------------------------

1. Configure variables in '.env' file from Nginx section:

.. code-block:: bash
# Domain name for service. Should be equal with name in generated ssl-certificate
NGINX_VHOSTNAME=demo.kqueen.net
# Directory path for certificates in container.Finally it look like $NGINX_SSL_CERTIFICATE_DIR/$NGINX_VHOSTNAME
NGINX_SSL_CERTIFICATE_DIR=/mnt/letsencrypt
2. Open the docker-compose.production.yml file for editing.
3. Check proxy service configuration in docker-compose.production.yml. Pay attention on following variables:

1. ``VHOSTNAME`` - domain name for Kqueen service. Should be equal with domain name in generated certificates. Default: Using variable from .env file, named NGINX_VHOSTNAME
2. ``SSL_CERTIFICATE_DIR`` - Mapped directory for certificates forwarding into docker container. Default: Using variable from .env file, named NGINX_SSL_CERTIFICATE_DIR/NGINX_VHOSTNAME
3. ``SSL_CERTIFICATE_PATH`` - Path for cert+key certificate. Default: ``$SSL_CERTIFICATE_DIR/fullchain.cer``
4. ``SSL_CERTIFICATE_KEY_PATH`` - Path for certificate key. Default: ``$SSL_CERTIFICATE_DIR/$VHOSTNAME.key``
5. ``SSL_TRUSTED_CERTIFICATE_PATH`` - Path for certificate only. Default: ``$SSL_CERTIFICATE_DIR/ca.cer``

.. note::

Check that local certificates naming equal to defined in variables.

4. Map volumes with certificates. Pay attention that destination path should be equal with SSL_CERTIFICATE_DIR. Example:

.. code-block:: yaml
volumes:
- /your/local/cert/storage/kqueen/certs/:${NGINX_SSL_CERTIFICATE_DIR}/${NGINX_VHOSTNAME}:ro
5. Build proxy service image:

.. code-block:: bash
docker-compose -f docker-compose.production.yml build --no-cache
6. Re-run production services:

.. code-block:: bash
docker-compose -f docker-compose.yml -f docker-compose.production.yml up --force-recreate
To set up the mail server
-------------------------

1. Open the docker-compose.production.yml file for editing.
2. Define the mail service. For example:

.. code-block:: yaml
mail:
image: modularitycontainers/postfix
volumes:
- /var/spool/postfix:/var/spool/postfix
- /var/mail:/var/spool/mail
environment:
MYHOSTNAME: 'mail'
3. Configure the following variables in the KQueen UI service section:

.. code-block:: yaml
KQUEENUI_MAIL_SERVER: mail
KQUEENUI_MAIL_PORT: 10025
Once done, you should be able to invite members through email notifications. The email will contain an activation link for the KQueen service with a possibility to set a password. Users with
superadmin rights can also manage member roles.


.. note::

Volume-mapping for mail containers is an additional feature that enables storing the mailing history and forwards an additional postfix mail configuration. You must properly configure it on the local machine. Otherwise, you can run the mail server without volume-mapping.

To set up metrics collecting
----------------------------

1. Open the docker-compose.production.yml file for editing.
2. Configure the Prometheus service IP address, port, and volumes. For example:

.. code-block:: yaml
prometheus:
image: prom/prometheus
restart: always
ports:
- 127.0.0.1:9090:9090
volumes:
- ./prod/prometheus/:/etc/prometheus/:Z
- /mnt/storage/kqueen/prometheus/:/prometheus/
links:
- api
- etcd
3. Define the Prometheus scraper IP address in the KQueen API service section:

.. code-block:: yaml
KQUEEN_PROMETHEUS_WHITELIST: '172.16.238.0/24'
There two ways to obtain metrics:

::
* Kqueen API:

# Move data to etcd directory
mv -v /root/backup/* /var/lib/etcd/default/
.. code-block:: bash
TOKEN=$(curl -s -H "Content-Type: application/json" --data '{"username":"admin","password":"default"}' -X POST <<kqueen_api_host>>:5000/api/v1/auth | jq -r '.access_token'); echo $TOKEN
curl -H "Authorization: Bearer $TOKEN" <<kqueen_api_host>>:5000/metrics/
* Prometheus API:
1. Add the scraper IP address to PROMETHEUS_WHITELIST configuration.
2. Run the following command:
.. code-block:: bash
curl <<prometheus_host>>:<<prometheus_port>>/metrics
All application metrics are exported to the **/metrics** API endpoint. Any external Prometheus instance can then scrape this metric.
Provision a Kubernetes cluster
------------------------------
You can provision a Kubernetes cluster using various community of engines, such as Google Kubernetes engine or Azure Kubernetes Service.
To provision a Kubernetes cluster using the Google Kubernetes Engine:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Login in the Google Kubernetes Engine (https://console.cloud.google.com).
2. Select your Project.
3. Navigate to the ``API’s & Services`` -> ``Credentials`` tab and click ``Create credentials``.
4. From ``Service Account key``, select your service account.
5. Select Json as the key format.
6. Download the Json snippet.
7. Log in to the KQueen web UI.
8. From the ``Create Provisioner`` tab, select ``Google Kubernetes Engine``.
9. Specify your project ID (``Project info`` tab on the main page of the GCE Dashboard https://console.cloud.google.com).
10. Insert the downloaded Json snippet that contains the service account key and submit the provisioner creation.
11. Click ``Deploy Cluster``.
12. Select the defined GCE provisioner.
13. Specify the cluster requirements.
14. Click ``Submit``.
15. To track the cluster status, navigate to the KQueen main dashboard.
To provision a Kubernetes cluster using the Azure Kubernetes Service:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Log in to https://portal.azure.com.
2. Create an Azure Active Directory Application as described in the official Microsoft `Documentation <https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#create-an-azure-active-directory-application>`_.
3. Copy the Application ID, Application Secret, Tenant ID (Directory ID), and Subscription ID to use in step 8.
4. Set the ``Owner`` role to your Application in the Subscription settings to enable the creation of Kubernetes clusters. For details, see the latest steps in the Microsoft official documentation described in step 2. Manually save the Application secret because you are not able to retrieve the secret later. You provide the secret value with the application ID to log in as the application. Store the key value where your application can retrieve it.
5. Navigate to the ``Resource groups`` tab and create a resource group. Copy the ‘Resource group name’ to use in step 8.
6. From the to ``Resource groups`` -> your_group -> Access Control (IAM) tab, verify that the Application has the ``Owner`` role in Resource group.
7. Log in to the KQueen web UI.
8. From the ``Create provisioner`` tab, select the AKS engine and set the following:
1. Set the ``Client ID`` as Application ID from p.3.
2. Set the ``Resource group name`` as Resource group name from p.4
3. Set the ``Secret`` as Application Secret from p.3
4. Set the ``Subscription ID`` as Subscription ID from p.3
5. Set the ``Tenant ID`` as Tenant(Directory) ID from p.3
9. In the KQueen web UI, click ``Deploy Cluster``.
10. Select the AKS provisioner.
11. Specify the cluster requirements.
12. Specify the public SSH key to connect to AKS VM’s. For ssh access into created VM’s, need to assign the public IP address to the VM (follow example `guide <https://gist.github.com/naumvd95/576d6e48200597ca89b26de15e8d3675>`_). Then it is possible to ``ssh azureuser@<<public_ip>> -i .ssh/your_defined_id_rsa``
13. Click ``Submit``.
14. To track the cluster status, navigate to the KQueen main dashboard.
.. note::
The Admin Console in the Azure portal is supported only in Internet Explorer and Microsoft Edge and may fail to operate in other browsers due to Microsoft `issues <https://microsoftintune.uservoice.com/forums/291681-ideas/suggestions/18602776-admin-console-support-on-mac-osx>`_.
**Pay attention** that AKS created separate resource during kubernetes cluster creating (it used defined Resource Group as prefix ). It may affect your billing. For ex.:
.. code-block:: text
Your Resource Group : Kqueen
Additional cluster-generated Resource Group: MC_Kqueen_44a37a65-1dff-4ef8-97ca-87fa3b8aee62_eastus
Related `Issues <https://github.com/Azure/AKS/issues/3>`_.
Related `Documentation <https://docs.microsoft.com/en-us/azure/aks/faq#why-are-two-resource-groups-created-with-aks>`_.
To manually add an existing Kubernetes cluster:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Log in to the KQueen web UI.
2. In the ``Create Cluster`` tab, define a valid Kubernetes configuration file.
As a result, the Kubernetes cluster will be attached in a read-only mode.
ETCD Backup
-----------
Etcd is the only stateful component of KQueen. To recover etcd in case of a failure, follow the procedure described in https://coreos.com/etcd/docs/latest/v2/admin_guide.html#disaster-recovery.
.. note::
The ``v2`` etcd keys are used in deployment.
Example of a backup and recovery workflow:
.. code-block:: bash
# Backup etcd to directory /root/backup/ (etcd data stored in /var/lib/etcd/default)
etcdctl backup --data-dir /var/lib/etcd/default --backup-dir /root/backup/
Recovery
# Start new etcd with these two extra parameters (among the other)
# for example: etcd --force-new-cluster
.. code-block:: bash
# Move data to etcd directory
mv -v /root/backup/* /var/lib/etcd/default/
kqueen
# Start new etcd with these two extra parameters (among the other)
# for example: etcd --force-new-cluster

0 comments on commit 4edcc8b

Please sign in to comment.