Merge pull request #259 from napalm-automation/develop

Prepare release 0.6.0

VERSION

@@ -1 +1 @@
-0.5.0
+0.6.0

docs/device_config/huawei.rst

@@ -0,0 +1,7 @@
+.. _device-configuration-huawei:
+
+==========
+Huawei VRP
+==========
+
+.. versionadded:: 0.6.0

docs/device_config/index.rst

@@ -14,6 +14,9 @@ systems:
    iosxr
    eos
    nxos
+   ios
+   netiron
+   huawei
 
 To see how to configure the network device, check the documents referenced
 above. Note that the examples in each case represents the configuration used to
@@ -30,6 +33,6 @@ partially parsed messages), but this behaviour is disabled by default.
 To allow publishing messages from operating systems that are not supported yet
 by napalm-logs (but they will not be parsed at all), you can configure the
 :ref:`publisher-opts-send-unknown` option on the publisher (i.e.,
-``send_unknow: true``). To publish partially parsed messages from supported
+``send_unknown: true``). To publish partially parsed messages from supported
 operating systems, but without a mapping for a certain class of messages, you
 can use the :ref:`publisher-opts-send-raw` option.

docs/device_config/ios.rst

@@ -0,0 +1,7 @@
+.. _device-configuration-ios:
+
+=========
+Cisco IOS
+=========
+
+.. versionadded:: 0.5.0

docs/device_config/netiron.rst

@@ -0,0 +1,7 @@
+.. _device-configuration-netiron:
+
+===============
+Brocade NetIron
+===============
+
+.. versionadded:: 0.6.0

docs/index.rst

@@ -310,6 +310,7 @@ daemon):
    options/index
    clients/index
    messages/index
+   metrics/index
    authentication/index
    listener/index
    publisher/index

docs/listener/tcp.rst

@@ -8,6 +8,8 @@ Receive the unstructured syslog messages over TCP.
 
 Available options:
 
+.. _listener-tcp-buffer-size:
+
 ``buffer_size``: ``1024``
 -------------------------
 
@@ -21,6 +23,27 @@ Example:
     tcp:
       buffer_size: 2048
 
+
+.. _listener-tcp-reuse-port:
+
+``reuse_port``: ``false``
+-------------------------
+
+.. versionadded:: 0.6.0
+
+Enable or disable ``SO_REUSEPORT`` on listener's socket.
+
+Example:
+
+.. code-block:: yaml
+
+  listener:
+    udp:
+      reuse_port: true
+
+
+.. _listener-tcp-socket-timeout:
+
 ``socket_timeout``: ``60``
 --------------------------
 
@@ -34,6 +57,9 @@ Example:
     tcp:
       socket_timeout: 5
 
+
+.. _listener-tcp-max-clients:
+
 ``max_clients``: ``5``
 ----------------------
 

docs/listener/udp.rst

@@ -8,6 +8,8 @@ Receive the unstructured syslog messages over UDP.
 
 Available options:
 
+.. _listener-udp-buffer-size:
+
 ``buffer_size``: ``1024``
 -------------------------
 
@@ -20,3 +22,21 @@ Example:
   listener:
     udp:
       buffer_size: 2048
+
+
+.. _listener-udp-reuse-port:
+
+``reuse_port``: ``false``
+-------------------------
+
+.. versionadded:: 0.6.0
+
+Enable or disable ``SO_REUSEPORT`` on listener's socket.
+
+Example:
+
+.. code-block:: yaml
+
+  listener:
+    udp:
+      reuse_port: true

docs/messages/UNKNOWN.rst

@@ -19,15 +19,15 @@ Example:
 
 .. code-block:: json
 
-	{
-	  "message_details": {
-		"message": "<28>Jul 10 10:32:00 vmx1 inetd[2397]: /usr/sbin/sshd[89736]: exited, status 255\n"
-	  },
-	  "timestamp": 1501685287,
-	  "ip": "127.0.0.1",
-	  "host": "unknown",
-	  "error": "UNKNOWN",
-	  "os": "unknown",
-	  "model_name": "unknown"
-	}
+    {
+      "message_details": {
+        "message": "<28>Jul 10 10:32:00 vmx1 inetd[2397]: /usr/sbin/sshd[89736]: exited, status 255\n"
+      },
+      "timestamp": 1501685287,
+      "ip": "127.0.0.1",
+      "host": "unknown",
+      "error": "UNKNOWN",
+      "os": "unknown",
+      "model_name": "unknown"
+    }
 

docs/metrics/index.rst

@@ -0,0 +1,94 @@
+.. _metrics:
+
+==================
+Metrics Collection
+==================
+
+Timeseries metrics my be optionally collected and exported from the server.
+This feature is dissabled by default but can be enabled by passing `--enable-metrics`
+when envoking the server.
+
+The metrics offer insight into how the server is processing messages. Each subsystem publishes
+its own set of metrics. These are timeserises data so they are mostly counters of processed
+elements.
+
+We use the Prometheus metrics format and the metrics are exposed by an HTTP server which by default
+run on port tcp/9443. The implementation is fully complient with Prometheus scraping, so you need only
+point your Prometheus server to the exposed metrics to scrape them.
+
+When using the feature, you must make a directory available which will be used to collect and store
+local metrics from the various system processes. The server will need rights to write to this
+directory. By default `/tmp/napalm_logs_metrics` is used. This is configurable with
+`--metrics-dir`. Whatever the value, if the directory does not exist, we attempt to create it
+on startup. Also note than this directory is cleared of all previous metrics entries on each run.
+This means that metric values are not persisted through runs.
+
+Here is a listing of the exposed metrics and their meaning:
+
+Listener Process(es)
+--------------------
+
+napalm_logs_listener_logs_ingested
+  Count of ingested log messages. Labels are used to seperate metrics for each Listener process.
+
+napalm_logs_listener_messages_published
+  Count of published messages. The are messages published to the message queue for processing by the Server Process.
+  Labels are used to seperate metrics for each Listener process.
+
+Server Process
+--------------
+
+napalm_logs_server_messages_received
+  Count of messages received from Listener processes.
+
+napalm_logs_server_messages_with_identified_os
+  Count of messages with positive OS identification. Labels are used to seperate metrics for each Device OS.
+
+napalm_logs_server_messages_without_identified_os
+  Count of messages which fail OS identification.
+
+napalm_logs_server_messages_failed_device_queuing
+  Count of messages per device OS that fail to be queued to a proper Device process. Note these are messages that
+  pass OS identification and we know how to route them but fail to be queued. Labels are used to seperate metrics
+  for each Device OS.
+
+napalm_logs_server_messages_device_queued
+  Count of messages sucessfully queued to Device processes. Labels are used to seperate metrics for each Device OS process.
+
+napalm_logs_server_messages_unknown_queued
+  Count of messages which fail OS indentification and thus we don't know how to route them, but the user has instructed
+  the system to queue them "as-is."
+
+Device Process(es)
+------------------
+
+napalm_logs_device_messages_received
+  Count of messages received from the Server process. Labels are used to seperate metrics for each Device OS process.
+
+napalm_logs_device_raw_published_messages
+  Count of raw type published messages. In this case, the message did not match a configured message type but the
+  user has instructed the system to publish the message in a raw format. Labels are used to seperate metrics for
+  each Device OS process.
+
+napalm_logs_device_published_messages
+  Count of published messages. These are messages which are sucessfully converted to an OpenConfig format. Labels
+  are used to seperate metrics for each Device OS process.
+
+napalm_logs_device_oc_object_failed 
+  Counter of failed OpenConfig object generations. These are messages for which the system attempts to map to a
+  known OpenConfig object model but fails. Labels are used to seperate metrics for each Device OS process.
+
+Publisher Process(es)
+---------------------
+
+napalm_logs_publisher_received_messages
+  Count of messages received by the Publisher from Device Process(es). Labels are used to seperate metrics for
+  each Publisher process.
+
+napalm_logs_publisher_whitelist_blacklist_check_fail
+  Count of messages which fail the whitelist/blacklist check. Labels are used to seperate metrics for each
+  Publisher process.
+
+napalm_logs_publisher_messages_published
+  Count of published messages. These are messages which are published for clients to receive (i.e. output of the
+  system). Labels are used to seperate metrics for each Publisher process.

docs/options/index.rst

@@ -76,6 +76,101 @@ Configuration file example:
 
   auth_port: 2022
 
+.. _configuration-options-enable-metrics:
+
+``enable-metrics``
+------------------
+
+.. versionadded:: 0.6.0
+
+Enable metrics collection exposition (Prometheus metrics).
+
+Default: ``false``
+
+CLI usgae example:
+
+.. code-block:: bash
+
+  $ napalm-logs --enable-metrics
+
+Configuration file example:
+
+.. code-block:: yaml
+
+  metrics_enabled: true
+
+.. _configuration-options-metrics-address:
+
+``metrics-address``
+-------------------
+
+.. versionadded:: 0.6.0
+
+The IP address to use to listen for all incoming metrics requests. This is the
+address that the Prometheus HTTP server exposes metrics from.
+
+Default: ``0.0.0.0``.
+
+CLI usage example:
+
+.. code-block:: bash
+
+  $ napalm-logs --metrics-address 172.17.17.1
+
+Configuration file example:
+
+.. code-block:: yaml
+
+  metrics_address: 172.17.17.1
+
+.. _configuration-options-metrics-port:
+
+``metrics-port``
+----------------
+
+.. versionadded:: 0.6.0
+
+The port to listen on for incoming metrics requests. This is the port that
+the Prometheus HTTP server exposes metrics from.
+
+Default: ``9443``
+
+CLI usgae example:
+
+.. code-block:: bash
+
+  $ napalm-logs --metrics-port 2022
+
+Configuration file example:
+
+.. code-block:: yaml
+
+  metrics_port: 2022
+
+.. _configuration-options-metrics-dir:
+
+``metrics-dir``
+---------------
+
+.. versionadded:: 0.6.0
+
+The directory used by the processes for metrics collection. This directory must
+be writable. If the directory does not exist, we attempt to create it.
+
+Default: ``/tmp/napalm_logs_metrics``
+
+CLI usgae example:
+
+.. code-block:: bash
+
+  $ napalm-logs --metrics-dir /tmp/a_new_dir_for_metrics
+
+Configuration file example:
+
+.. code-block:: yaml
+
+  metrics_dir: /tmp/a_new_dir_for_metrics
+
 .. _configuration-options-certificate:
 
 ``certificate``