Merge pull request #184 from earthgecko/py3

py3 docs changes and additions

docs/alerts.rst

@@ -0,0 +1,196 @@
+======
+Alerts
+======
+
+Alerts in Skyline are configured via two sets of settings.  There are the
+Analyzer (Mirage and Ionosphere) related settings and the Boundary related
+alert settings.  This is due to the two classes of alerts being different,
+with Analzyer, Mirage and Ionosphere alerts being related to anomalies and
+Boundary alerts being related to breaches of the static and dynamic thresholds
+defined for Boundary metrics.
+
+Required smtp alerter for Analyzer and Mirage metrics
+=====================================================
+
+Skyline was initially driven by SMTP alerts and as such the analysis pipeline is
+tightly coupled to SMTP alerts.  Many of the resources required for further
+analysis and for other alerters are created during the SMTP alert process.
+Further to this, the full spectrum of analysis is only done on metrics that have
+a SMTP alert tuple configured for the namespace in :mod:`settings.ALERTS`.  This
+is because the more extensive analysis is only done on the metrics that you care
+about, your KPI metrics.  Doing the extended analysis beyond the 3sigma related
+analysis of metrics that have no alerts configured for them makes no sense as no
+one wants to know.  Therefore only metrics in namespaces that are defined with a
+stmp alerter in :mod:`settings.ALERTS` get analysed by Mirage and Ionosphere.
+It is via the smtp alert tuple that metrics get configured to be Mirage metrics
+by declaring the SECOND_ORDER_RESOLUTION_HOURS in the tuple.
+
+However if you do not want to be SMTP alerted, you can set the
+:mod:`settings.SMTP_OPTS` to `'no_email'` as shown in an example below, but you
+must still declare the namespace with a SMTP alert tuple in
+:mod:`settings.ALERTS`
+
+The following example, we want to alert via Slack, your :mod:`settings.ALERTS`
+and :mod:`settings.SMTP_OPTS` would need to look like this.
+
+.. code-block:: python
+
+  ALERTS = (
+    ('carbon', 'smtp', 3600),  # Analyzer metric, only analysed at FULL_DURATION
+    ('skyline', 'smtp', 3600, 168),  # Mirage enabled as 168 is passed as SECOND_ORDER_RESOLUTION_HOURS
+    ('stats', 'smtp', 1800, 168),  # Mirage enabled as 168 is passed as SECOND_ORDER_RESOLUTION_HOURS
+    # smtp alert tuples MUST be declared first
+    ('carbon', 'slack', 3600),
+    ('skyline', 'slack', 3600),
+    ('stats', 'slack', 1800),
+  )
+  SMTP_OPTS = {
+      # This specifies the sender of email alerts.
+      'sender': 'no_email',
+      # recipients is a dictionary mapping metric names
+      # (exactly matching those listed in ALERTS) to an array of e-mail addresses
+      'recipients': {
+          'stats': ['no_email'],
+          'skyline': ['no_email'],
+          'carbon': ['no_email'],
+      },
+      # This is the default recipient which acts as a catchall for alert tuples
+      # that do not have a matching namespace defined in recipients
+      'default_recipient': ['no_email'],
+      'embed-images': True,
+  }
+
+Alert settings
+==============
+
+For each 3rd party alert service e.g. Slack, PagerDuty, http_alerters, there is
+a setting to enable the specific alerter which must be set to `True` to enable
+the alerter:
+
+- :mod:`settings.SYSLOG_ENABLED`
+- :mod:`settings.PAGERDUTY_ENABLED`
+- :mod:`settings.SLACK_ENABLED`
+- :mod:`settings.HTTP_ALERTERS_ENABLED`
+
+Analyzer, Mirage and Ionosphere related alert settings (anomaly detection) are:
+
+- :mod:`settings.ENABLE_ALERTS` - must be set to `True` to enable alerting
+- :mod:`settings.ENABLE_FULL_DURATION_ALERTS` - must be set to `True` to enable
+  Analyzer to alert
+- :mod:`settings.ALERTS` - must be defined to enable alerts via Analyzer and
+  Mirage
+- :mod:`settings.SMTP_OPTS` - must be defined
+- :mod:`settings.SLACK_OPTS` - must be defined if you want to alert via Slack
+- :mod:`settings.PAGERDUTY_OPTS` - must be defined if you want to alert via
+  Pagerduty
+- :mod:`settings.SYSLOG_OPTS` - can be used to change syslog settings
+- :mod:`settings.HTTP_ALERTERS_OPTS` - must be defined if you want to push
+  alerts to a http endpoint
+- :mod:`settings.MIRAGE_ENABLE_ALERTS` - must be set to `True` to enable alerts
+  from Mirage
+
+Boundary related alert settings (static and dynamic thresholds) are:
+
+- :mod:`settings.BOUNDARY_ENABLE_ALERTS` - must be set to `True` to enable
+  alerting
+- :mod:`settings.BOUNDARY_METRICS` - must be defined to enable checks and alerts
+  for Boundary
+- :mod:`settings.BOUNDARY_ALERTER_OPTS` - can be used to change the Boundary
+  alert rate limits
+- :mod:`settings.BOUNDARY_SMTP_OPTS` - must be defined if you want to send SMTP
+  alerts
+- :mod:`settings.BOUNDARY_PAGERDUTY_OPTS` - must be defined if you want to alert
+  via Pagerduty
+- :mod:`settings.BOUNDARY_SLACK_OPTS` - must be defined if you want to alert via
+  Slack
+- :mod:`settings.BOUNDARY_HTTP_ALERTERS_OPTS` - must be defined if you want to
+  push alerts to a http endpoint
+
+http_alerter alerts
+===================
+
+The http_alerter alert type, enables Skyline to send alerts to a HTTP endpoint.
+http_alerters are configured under the following settings.
+
+In :mod:`settings.ALERTS` and :mod:`settings.BOUNDARY_METRICS` you defined the
+namespace and http_alerter and expiry (how long to not send an alert after an
+alert has been sent).
+
+.. code-block:: python
+
+  # For Analzyer, Mirage and Ionosphere
+  ALERTS = (
+      ...
+      ('stats', 'http_alerter-external_endpoint', 30),
+  )
+
+  # For Boundary
+  BOUNDARY_METRICS = (
+      # ('metric', 'algorithm', EXPIRATION_TIME, MIN_AVERAGE, MIN_AVERAGE_SECONDS, TRIGGER_VALUE, ALERT_THRESHOLD, 'ALERT_VIAS'),
+      ('stats.rpm.total', 'detect_drop_off_cliff', 1800, 50, 3600, 0, 2, 'slack|pagerduty|http_alerter_external_endpoint'),
+      ('stats.rpm.total', 'less_than', 3600, 0, 0, 15, 2, 'http_alerter_external_endpoint'),
+      ('stats.rpm.total', 'greater_than', 3600, 0, 0, 10000, 1, 'http_alerter_external_endpoint'),
+  )
+
+Each http_alerter can be defined in a common :mod:`settings.HTTP_ALERTERS_OPTS`
+which all the apps refer to.
+
+.. code-block:: python
+
+  HTTP_ALERTERS_OPTS = {
+      'http_alerter-external_endpoint': {
+          'enabled': True,
+          'endpoint': 'https://http-alerter.example.org/alert_reciever',
+          'token': None
+      },
+      'http_alerter-otherapp': {
+          'enabled': False,
+          'endpoint': 'https://other-http-alerter.example.org/alerts',
+          'token': '1234567890abcdefg'
+      },
+  }
+
+
+The http_alerter will post an alert json object to an HTTP endpoint.  Here is
+an example of the alert json POST data:
+
+.. code-block:: json
+
+  {
+    "status": {},
+    "data": {
+      "alert": {
+        "timestamp": "1579638755",
+        "metric": "stats.sites.graphite_access_log.httpd.rpm.total",
+        "value": "75.0",
+        "expiry": "30",
+        "source": "ionosphere",
+        "token": "None",
+        "full_duration": "604800"
+      }
+    }
+  }
+
+Failures
+--------
+
+If the HTTP endpoint does not respond with a 200, the alert item will be added
+to the Redis set (queue) for that alert to be resent.  Each relevant app has a
+Redis set (queue):
+- analyzer.http_alerter.queue
+- mirage.http_alerter.queue
+- ionosphere.http_alerter.queue
+- boundary.http_alerter.queue
+
+Mirage sends the initial HTTP alert for Mirage and Mirage/Ionosphere metrics, if
+an alert fails to be sent it is added to the resend queue.
+
+Analyzer handles *all* the resend queues and resends for Analyzer, Mirage and
+Ionosphere.  There are checks to determine whether the endpoint is healthy to
+prevent Analyzer from repeatedly attempting to resend all alerts to a
+http_alerter endpoint that is down.
+
+Boundary handles the boundary resend queue and resends for boundary alerts.
+Boundary uses the same checks to determine whether the endpoint is healthy to
+prevent Boundary from attempting to repeatedly resend all alerts to a
+http_alerter endpoint that is down.

docs/running-in-python-virtualenv-py2.rst

@@ -0,0 +1,130 @@
+==========================================
+Running Skyline in a Python 2.7 virtualenv
+==========================================
+
+.. note:: This is no longer supported with the EOL of Python 2 but this documentation
+  remains for anyone still using py2 until it does not work any longer.
+
+Running Skyline in a Python virtualenv is the recommended way to run
+Skyline. This allows for Skyline apps and all dependencies to be
+isolated from the system Python version and packages and allows Skyline
+be be run with a specific version of Python and as importantly specific
+versions of the dependencies packages.
+
+The possible overhead of adding Python virtualenv functionality to any
+configuration management is probably worth the effort in the long run.
+
+``sudo``
+~~~~~~~~
+
+Use ``sudo`` appropriately for your environment wherever necessary.
+
+HOWTO Python virtualenv Skyline
+===============================
+
+Dependencies
+~~~~~~~~~~~~
+
+Building Python versions from the Python sources in Python virtualenv
+requires the following system dependencies:
+
+-  RedHat family
+
+.. code-block:: bash
+
+    yum -y install epel-release
+    yum -y install autoconf zlib-devel openssl-devel sqlite-devel bzip2-devel \
+      gcc gcc-c++ readline-devel ncurses-devel gdbm-devel compat-readline5 \
+      freetype-devel libpng-devel python-pip wget tar git
+
+-  Debian family
+
+.. code-block:: bash
+
+    apt-get -y install build-essential
+    apt-get -y install autoconf zlib1g-dev libssl-dev libsqlite3-dev libbz2-dev \
+      libreadline6-dev libgdbm-dev libncurses5 libncurses5-dev libncursesw5 \
+      libfreetype6-dev libxft-dev python-pip wget tar git
+
+virtualenv
+~~~~~~~~~~
+
+Regardless of your OS as long as you have pip installed you can install
+virtualenv. *NOTE:* if you are using a version of Python virtualenv
+already, this may not suit your setup.
+
+virtualenv must be >= 15.0.1 and <16.0.0 due to some recent changes in pip,
+setuptools and virtualenv, see **Recent changes in the pip environment** section
+in `Requirements <requirements.html#recent-changes-in-the-pip-environment>`__
+for more details.
+
+This is using your **system** pip at this point only to install virtualenv.
+
+.. code-block:: bash
+
+    pip install 'virtualenv==15.2.0'
+
+Python version
+~~~~~~~~~~~~~~
+
+.. note:: Python-2.7.14 should still work as well, but all documentation has
+  been updated to use Python-2.7.16.
+
+Below we use the path ``/opt/python_virtualenv``, which you can substitute
+with any path you choose.  We are going to use the Python-2.7.16 source and
+build and install an isolated Python-2.7.16, this has no effect on your system
+Python:
+
+.. code-block:: bash
+
+    PYTHON_VERSION="2.7.16"
+    PYTHON_MAJOR_VERSION="2.7"
+    PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
+
+    mkdir -p "${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}"
+    mkdir -p "${PYTHON_VIRTUALENV_DIR}/projects"
+
+    cd "${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}"
+    wget -q "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz"
+    tar -zxvf "Python-${PYTHON_VERSION}.tgz"
+
+    cd ${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}/Python-${PYTHON_VERSION}
+    ./configure --prefix=${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}
+    make
+
+    # Optionally here if you have the time or interest you can run
+    # make test
+
+    make altinstall
+
+You will now have a Python-2.7.16 environment with the Python
+executable: ``/opt/python_virtualenv/versions/2.7.16/bin/python2.7``
+
+Create a Skyline Python virtualenv
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A word of warning relating to pip, setuptools and distribute if you have
+opted not to use the above as you have Python virtualenv already. As of
+virtualenv 15.0.1 the pip community adopted the new pkg\_resources
+\_markerlib package structure, which means the following in the
+virtualenv context:
+
+-  distribute cannot be installed
+-  pip must be >=8.1.0
+-  setuptools must be >=20.2.2
+
+Once again using Python-2.7.16:
+
+.. code-block:: bash
+
+    PYTHON_VERSION="2.7.16"
+    PYTHON_MAJOR_VERSION="2.7"
+    PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
+    PROJECT="skyline-py2716"
+
+    cd "${PYTHON_VIRTUALENV_DIR}/projects"
+    virtualenv --python="${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}/bin/python${PYTHON_MAJOR_VERSION}" "$PROJECT"
+
+
+Make sure to add the ``/etc/skyline/skyline.conf`` file - see
+`Installation <installation.html>`__