documentation update

docs/data_processing.rst

@@ -95,3 +95,48 @@ nmon_helper.sh tasks part5: startup
 * start nmon binary
 * Identify interpreter to be used
 * start the fifo_reader
+
+===================
+Consuming Nmon data
+===================
+
+Consuming the nmon data generated by the nmon_helper.sh and associated scripts (fifo_reader) is in first step operated by the "bin/fifo_consumer.sh" script.
+
+This is a very simple shell script that will recompose the nmon data in the correct order, and stream its content to nmon parsers. (nmon2csv)
+
+This script does as well the files rotation, such that next cycle starts such that the nmon_data.dat file is empty after consumption, which guarantees a low level of CPU and resources usage over each iteration.
+
+.. image:: img/fifo_consumer.png
+   :alt: fifo_consumer.png
+   :align: center
+
+**Steps are:**
+
+* Initialization and configuration loading
+* Verify if fifo1 and fifo2 have rotated data to be proceeded
+* Verify if fifo1 and fifo2 have non empty nmon_data.dat
+* If so, wait at least 5 seconds age file before processing the data
+* Stream nmon data to nmon2csv.sh
+
+=================
+Parsing Nmon data
+=================
+
+This is the final step in the nmon processing tasks, the nmon2csv parsers will consume the data receive in standard input (stdin), and produce final files to be indexed by Splunk.
+
+**There 3 scripts involved in these tasks:**
+
+* bin/nmon2csv.sh: simple shell wrapper that will decide to use Python or Perl parser
+* bin/nmon2csv.py: the Python version parser
+* bin/nmon2csv.pl: the Perl version parser
+
+.. image:: img/nmon2csv.png
+   :alt: nmon2csv.png
+   :align: center
+
+**Steps are:**
+
+* nmon2csv.sh reads data from standard input
+* nmon2csv.sh decides to use Python if version 2.7.x.available, or fall back to Perl
+* nmon2csv.py|.pl parse the data, generates configuration and performance data, as well as internal flag files
+* data is being indexed in batch mode by Splunk, index and delete