Section 6 (#10)

* initial toc

* fixed index for readthedocs

* added mkdocs for forcing the toc structure

* fixed indentation

* fixed node operation error in mkdocs compilation

* restructured the repository

* added missing index page in mkdocs.yml

* fixed linter issues

* enabled readthedocs build in the ci process

* fixing few linting issues and applying comments

* added new rules to the md linting added python 2.7 to the ci-build

* include section 1

* minor fixes

* fix code print

* try to fix index

* fix index

fix index

* fix levels and code

* fix code

* increase indent

* fixes

* fix syntax

* disable md41

* add content for section-2

* index

* initial commit

* finalised section 3

* linting

* content for section 4

* content for section 5

* section 6

* Fixes based on PR feedback

* Fixes for Travis errors

* Fixes for Travis errors

* Fixes based on pull request feedback

* Fixes based on pull request feedback

.markdownlintrc

@@ -2,7 +2,7 @@
   "default": true,
   "MD003": { "style": "atx" },
   "MD007": { "indent": 4 },
-  "MD013": { 
+  "MD013": {
     "code_blocks": false,
     "tables": false
   },
@@ -18,4 +18,4 @@
   "MD041": false,
   "MD042": false,
   "MD013": false
-}
+}

docs/2.policies/1.introduction.md

@@ -0,0 +1,4 @@
+# Policies
+
+The following section describes the Fiware Lab policies regarding privacy, the
+use of cookies, and the terms and conditions of usage.

docs/3.process/1.introduction.md

@@ -0,0 +1,4 @@
+# The Process of Joining Fiware Lab
+
+The following section will go into details to describe how to join Fiware Lab,
+from setting up a node to installing, configuring, upgrading and maintaining it.

docs/6.coordination/1.introduction.md

@@ -0,0 +1,11 @@
+# Coordination
+
+This section offers an overview of:
+
+-   FIWARE Lab coordination and support procedures (including community
+    accounts management, adopted tools, and Help Desk procedures);
+
+-   statistics related to overall resources and usage of FIWARE Lab;
+
+-   a log of General Events (e.g. failures, maintenance) pertaining to
+    the overall FIWARE Lab.

docs/6.coordination/2.procedures.md

@@ -0,0 +1,222 @@
+## Coordination and support procedures
+
+### Coordination Approach
+
+FIWARE Lab activities are managed by bi-weekly meetings on Tuesday 15:00 CET,
+all FIWARE Lab nodes administrators are invited to participate on it and share
+their status and discuss different topics relating to the activities carried out
+within the FIWARE Lab.
+
+The main topics covered within the meeting are:
+
+-   Status of the node
+
+    Each Node Administrator, provides a snapshot of the current status of
+    its node. He/She provides information about the presence within the
+    [Infographic page](http://infographic.lab.fiware.org), about the
+    [Sanity Checks status](http://status.lab.fiware.org), as well as
+    useful updates during specific activities like hardware maintenance,
+    users migration or OpenStack upgrade version.
+
+-   Issues occurred during the previous week
+
+    Each Node Administrator, discusses issues encountered during the past
+    week. This is a crucial point in order to identify and solve possible
+    weaknesses or bugs within the FIWARE Lab architecture (e.g. common
+    connectivity errors toward the centralized keystone) as well as
+    possible weakness in the FIWARE Lab documentation.
+
+-   Instructions from the FIWARE Technical Steering Committee
+
+    This task is to inform all Nodes Administrators about decisions taken
+    by the FIWARE Technical Steering Committee and to design a roadmap of the
+    future activities.
+
+-   Help Desk pending requests
+
+    All Nodes Administrators are asked to verify all pending (not closed)
+    Help Desk requests, and it is discussed how to resolve them as soon as
+    possible in order to meet agreed SLAs.
+
+-   Share suggestions
+
+    This task is to share suggestions among all Node Administrators and
+    FIWARE Lab technical experts. It is an important aspect of the meeting
+    because it allows those who have found a solution or workaround for a
+    specific problem to share that experience among the community to
+    facilitate the expansion and stability of FIWARE Lab.
+
+-   Topics of the day
+
+    This is an open window within the meeting to discuss about topics
+    (even off-topics) not covered during a standard meeting.
+
+Beside all points above, a specific mailing list
+(fiware-lab-federation-nodes@lists.fiware.org) is used by FIWARE Lab node
+administrators and FIWARE Lab experts to exchange each other
+doubts, information, tips and any kind of communication useful to the
+growth and stability of FIWARE Lab.
+
+### Community Account Requests
+
+A Community User is allowed to experiment with FIWARE technology for a
+period of more than 9 months. Typical examples are SMEs/start-ups under
+the FIWARE Accelerator Programme.
+
+Trial Users can always apply to for upgrading their accounts to become
+Community Users. This is granted to everybody if it is understood that
+the application they aim to developing, is considered a relevant
+reference example for the development of the FIWARE Community.
+
+In order to apply to become a Community User it is necessary to compile
+an application form accessible through the main page of the FIWARE Lab
+portal [FIWARE Lab portal](https://account.lab.fiware.org) - click
+the *Request Community Account Upgrade* - button.
+
+![FIWARE Lab request community account upgrade](image19.png)
+
+This will open a window in which information about application is
+requested from the user, this information will be used to understand
+exactly what is the planned use of the resources, why they are required
+and in which FIWARE Lab node you plan to use those resources.
+
+![FIWARE Lab community account request](image20.png)
+
+It will create a ticket inside the FIWARE Lab Upgrade Account in Jira to
+be response by the L1 Support team and assigned accordingly to the
+corresponding FIWARE Lab administrator node in order to resolve it.
+
+![FIWARE Lab Upgrade Account (FLUA) Jira project](./media/image21.png)
+
+From here, the FIWARE Lab node administrators will take those tickets
+and apply the corresponding activities described in
+[Account Management](/docs/5.management/1.account.md) to provide the
+corresponding resources to the user.
+
+### Help Desk Support
+
+The Help Desk activities are daily part of FIWARE Lab operations. The
+Help Desk is the support that FIWARE Lab experts, Nodes Administrators
+and GEs Owners, give to external and internal users. The Help Desk
+activities are structured in 2 main Level of support in order to
+guarantee that agreed SLAs are achieved.
+
+The first level of support is comprised of a team which is in charge of
+managing all FIWARE Lab incoming tickets. This team is also responsible
+for categorizing all incoming tickets in order to guarantee that proper BA
+process can be executed to extract relevant information about the use of the
+different FIWARE GEs.
+
+In the following, we describe in more details the procedures for Level 1
+support.
+
+The Level 1 Help Desk team is organized in 8x5 (Monday to Friday) scheduled
+shifts from 08:00 to 17:00. The team is composed by people from the
+FIWARE Foundation. It provides support for general issues that can be easily
+solved by pointing to the FAQ, Stack Overflow, groups or other
+documentation. Moreover, its responsibility is to filter, categorise and
+forward to Level 2 support (Node Owners or GE Owners) all those tickets
+that it is not able to answer. It is also responsible for managing
+Community Account requests.
+
+At the end of each day, there should be no tickets remaining in status
+“unassigned” within the FIWARE Lab Help Desk queue:
+[*http://backlog.fiware.org/lab/helpdesk*](http://backlog.fiware.org/lab/helpdesk).
+
+#### Level1 Support Requests Management
+
+The activity that each member of the L1 Support Team should be assigned
+the corresponding tickets to the proper person, assigning the
+corresponding Component and for statistical reasons assign the
+corresponding HD-Enabler or HD-Node. They are Jira issues
+attributes that need to be assigned manually when a new ticket is
+received.
+
+![HD-Enabler and HD-Node attributes in issues](image22.png)
+
+For the assignee of the JIRA ticket, we have to differentiate between
+Generic Enablers owners and FIWARE Lab administrators’ nodes. For the
+first one, the list of owners can be obtained from the following table:
+
+[*FIWARE GE owners*](https://docs.google.com/spreadsheets/d/1X9iLY9Znd3Rh-GM4qffGEHRNKqKcHZqdm-dP0rq8OVo/edit#gid=694707434)
+
+It is a working document of FIWARE and it is available for the FIWARE
+Lab administrators. If a FIWARE Lab node administrators want to get
+access to it, they can request access to the owner of the file (fernando
+dot lopez at fiware dot org). Regarding the list of FIWARE Lab node
+administrators, the list can be obtained in the following file:
+
+[*FIWARE Lab node administrators*](https://docs.google.com/spreadsheets/d/1X9iLY9Znd3Rh-GM4qffGEHRNKqKcHZqdm-dP0rq8OVo/edit#gid=744561338)
+
+Regarding the components, usually they are automatically selected by the
+different tools that are behind JIRA, but in case that we found a JIRA
+ticket without the component the available values are the following:
+
+
+
+| **Components** | **Description** |
+| -- | -- |
+| FIWARE-COLLABORATION-REQ | Issues related to the request associated to some type of collaboration with FIWARE in terms of participation in some events or in terms of improvements of some FIWARE GE |
+| FIWARE-FEEDBACK | General issues related to feedback recover from the users. |
+| FIWARE-GENERAL-HELP | Issues in general not classified in the other components. |
+| FIWARE-LAB-HELP | Issues related to some of the FIWARE Lab nodes |
+| FIWARE-MUNDUS-REQ | Issues regarding the activities of FIWARE associated to the extension of FIWARE beyond Europe (FI-GLOBAL project). |
+| FIWARE-OPEN-DATA-REQ | (Deprecated) Issues related to the management of Open Data inside CKAN tool or related to the possibility to contribute with Open Data inside FIWARE ecosystem. |
+| FIWARE-OPS-HELP | Issues related to the use of the different FIWARE Ops tools.
+| FIWARE-SMART-CITIES-REQ | (Deprecated) Issues related to the collaboration in the SmartCities |
+| FIWARE-SPEAKERS-REQ | Issues related to the request to get some FIWARE expert to provide some presentation or speech in some events or just in some Summer School or so on. |
+| FIWARE-TECH-HELP | Issues related to some of the FIWARE GE. |
+| FIWARE-TRAINING-REQ | (Deprecated) Issues related to the request of training for some of the accelerator program. Currently, it is not needed due to there is no request for training from accelerator programs. |
+
+In case of HD-Enabler, it makes reference to the
+corresponding Enabler to which the corresponding JIRA ticket has to be assigned
+to, the way it was described previously. It is something that have to be
+selected once we edit the corresponding JIRA issue.
+
+![Jira Help Desk issue](image23.png)
+
+Last but not least, HD-Node makes reference to the corresponding FIWARE
+Lab node in which this issue should be resolved. Usually, the issues are
+related to FIWARE GEis or FIWARE Lab nodes. The first one is associated
+to the email <fiware-tech-help@lists.fiware.org> and the second is
+associated to <fiware-lab-help@lists.fiware.org>. This means:
+
+-   FIWARE GEs have associated the attribute of the HD-Enabler in order
+    to identity which one is it.
+
+-   FIWARE Lab nodes has associated the attribute HD-Node, in order to
+    know to which FIWARE Lab node is assigned this issue or Jira ticket.
+
+-   Due to these tickets come from different source it is needed to
+    complete only HD-Enabler in case of tickets associated to FIWARE GEs or
+    HD-Node is case of tickets associated to FIWARE Lab nodes.
+
+How it was mentioned before, the reason of it, is just to allow further
+analytical analysis of the tickets that we receive in the different
+channels.
+
+#### Community Account Requests Management
+
+The Level 1 team is also responsible for the process of approval of the
+Community Account requests.
+
+Each time a FIWARE Lab user asks for a Community Account, a “FLUA”
+ticket is generated and the Level 1 team is responsible for verifying
+the eligibility of users and approve the requests of the user as
+trusted.
+
+This is done by examining the scope of the account request, e.g. the
+project for which it was requested and how it adopts FIWARE
+technologies. To be eligible, a request should: i) make relevant use of
+key FIWARE technologies (i.e. not just using the Lab as mean to host
+other technologies); ii) have an experimentation or educational purpose
+(i.e. commercial services cannot be hosted on FIWARE Lab).
+
+Only if an account is eligible, the user is upgraded to Community and the
+resources assigned accordingly.
+
+Based on the preferences of the user and resource available, the Level 1
+team assigns the approved accounts to a Node. The Level 1 team will keep
+open the initial ticket until the node completes the assignment and
+eventually follows up with the assigned node (and the user) to ensure
+that the procedure completes correctly.

docs/6.coordination/3.statistics.md

@@ -0,0 +1,125 @@
+## FIWARE Lab Statistics
+
+### Available Resources
+
+#### Physical resources {#physical-resources .ListParagraph}
+
+According to OpenStack documentation the default overcommit for RAM is
+1.5:1, the default overcommit for CPU is 16:1 and disk space for the
+Virtual Machines should not be overcommitted (please, notice that
+Volumes provisioned by Cinder is not physically where the Virtual
+machines are) -- That is shown in this page of the documentation:
+
+[*Overcommitting CPU and RAM*](https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html)
+
+The available physical resources can be easily calculated when the
+hardware is bought and installed. However, we could ask nova about the
+resources (available and used) with a small and simple script like this
+one:
+
+```
+( nova --timeout 15 --insecure list > /dev/null || exit 1\
+  h=$(($(nova --timeout 60 --insecure list --all-tenants | wc -l) - 4))
+  echo hosts $h
+  for a in `nova --timeout 60 --insecure host-list | awk '/compute/ {print $2}'`; do
+    nova --timeout 60 --insecure host-describe $a | awk '
+    /total/ {printf ("cpu %s\nmem %s\ndisk %s\n",$6,$8,$10)}\
+    /used_now/ {printf ("used_cpu %s\nused_mem %s\nused_disk%s\n",$6,$8,$10)}\
+    ' &\
+done ) | awk '{sum[$1] += $2 } ; END {for (a in sum) {print a,sum[a]}}'
+```
+
+The output for this script is something like this:
+
+```
+mem 6767440
+used_mem 7213056
+disk 512859
+used_disk 69913
+hosts 1864
+cpu 1632
+used_cpu 3509
+```
+
+Where mem, disk and cpu are the physical resources installed in all the
+compute nodes, and `used_mem`, `used_disk` and `used_cpu` are the virtual
+resources used. The other field, hosts, is the number of VMs deployed in
+the node.
+
+As a caveat, the command `nova host-describe` will perform something
+like a Unix `df` command. So, the returned value for disk is the free
+space in `/var/lib/nova/instances`. If a NFS share is mounted in several
+compute nodes (this would make easier some administrative tasks), the
+nova host-describe is going to return the free space in the NFS Share
+multiplied for the number of hosts where it is mounted as well as the
+`used_disk`. So, the disk information is wrong.
+
+#### Floating IPs
+
+The default external network providing floating IPs, how it was
+described in the previous section is named *public-ext-net-01*.
+
+```
+neutron net-show public-ext-net-01
+```
+
+We will see that the subnet associated with that network has ID
+`4430b64a-85d8-4933-ae79-9a76ff1e2aa9`, therefore, querying the subnet:
+
+```
+neutron subnet-show 4430b64a-85d8-4933-ae79-9a76ff1e2aa9
+
+…
+| allocation_pools | {"start": "130.206.112.16", "end": "130.206.127.254"} |
+| cidr | 130.206.112.0/20 |
+```
+
+We have a /20 CIDR which is 2^(32-20)=4096 IPs, however, the 1st one is
+130.206.112.16 (16 IPs not in the pool from 130.206.112.0 to
+130.206.112.15) and the last IP in the pool is 130.206.127.254 (1 IP not
+in the pool at the end) - So There are 17 IPs not in the pool, this
+means 4096-17=4079 floating IPs. To know how many floating IPs there are
+in use, we can check it with Neutron as administrator:
+
+```
+neutron floatingip-list | grep “ 130.206” | wc -l
+```
+
+### Get summary statistics
+
+In order to get the summary statistics on one node during a period of
+time, you can execute the command:
+
+```
+$ openstack usage list
+```
+
+It will return the list of resources consumed for each tenant/project as
+shown below:
+
+```
+Usage from 2013-06-25 to 2013-07-24:
+
+  Project   Servers   RAM MB-Hours   CPU Hours   Disk GB-Hours
+  --------- --------- -------------- ----------- ---------------
+  demo      1         344064.44      672.00      0.00
+  stack     3         671626.76      327.94      6558.86
+```
+
+### Hosted Users
+
+In order to get the Users, there are several ways of doing this, the way
+we usually do is using a script made in Python which retrieves all the
+information about users stored in Keystone (groups, users, roles,
+role_assignments, etc.) and produces a big json.
+
+Using the Command Line tool *jq*, we query this json file:
+
+```
+jq -r '.role_assignments[] |
+       select (.role.id == "<role_id basic, trial or community") |
+       .user.id' <fichero_json_usuarios> | wc -l
+```
+
+It is worth noting that this information is also visible at
+[*http://infographic.lab.fiware.org*](http://infographic.lab.fiware.org)