add support for indexing SAR data into elasticsearch 1.7.x

[arcolife]

w.r.t.: #227
This PR adds SAR data indexing capability to pbench. Details:

1. index-pbench now calls a module - index-sar.
2. Index name in ElasticSearch for pbench's indices, was modified to accommodate for SAR data mapping.
3. All indexed data goes to pbench.sar-yyyy-mm-dd
4. sar and pbench runs are connected by the md5 of the pbench run.

• sar docs in ES have 2 fields from the pbench run:
  1. sar data hierarchy path (as _metadata.pbench_run_unique_id) depicting path to sar file in results
  2. pbench-run's md5 (as _metadata.pbench_run_md5 )

[arcolife]

okay it seems I forgot about the indexing unittests there. Shall see to that..

[arcolife]

Using this, a POST query made to _all indices (for example), in the following form:

{
  "query": {
    "multi_match": {
      "query": "2e2e8933451409eeda920ab6c35f7cde",
      "fields": [
        "_metadata.md5",
        "_metadata.pbench_run_md5"
      ]
    }
  },
  "from": 0,
  "size": 2,
  "fields": [
    "_metadata.file-name",
    "_metadata.nodename"
  ],
  "sort": [],
  "aggs": {}
}

..would result in :

...
"hits" : [ 
    {
      "_index" : "pbench.2015-10",
      "_type" : "pbench-run",
      "_id" : "2e2e8933451409eeda920ab6c35f7cde",
      "_score" : 3.925263,
      "fields" : {
        "_metadata.file-name" : [ "/var/lib/pbench-agent/benchserver2_sample_pbench_run_2015-10-11.tar.xz" ]
      }
    }, 
    {
      "_index" : "pbench.sar-2015-10-11",
      "_type" : "sar",
      "_id" : "cd703c87c70ea9fe623ce748b4af1ca1",
      "_score" : 2.7668054,
      "fields" : {
        "_metadata.nodename" : [ "benchserver2" ]
      }
    } 
]
...

[portante]

@arcolife, can you rebase this commit?

[portante]

And then add unit tests?

[arcolife]

rebased. will add the unit tests. Although, is https://travis-ci.org/distributed-system-analysis/pbench/jobs/145559700#L907 this related to this PR? I've faced this before.

[portante]

@arcolife, I have seen that before, but I don't remember the root cause. We'll have to debug it.

[arcolife]

update: failure in pbench unit test [ref: https://github.com//pull/176#issuecomment-233359641 ] due to modification of index-pbench by this commit and consequently change in expected line numbers under pbench/bin/gold/test-*.txt (viz-a-viz test case output verification texts) ..and so the unittests:_verify_output() (code ref: https://github.com/distributed-system-analysis/pbench/blob/master/server/pbench/bin/unittests#L93 ) declares failed cases.

[arcolife]

WIP: unit test rework. reference: #303
will be adding a proper gold/test-9.txt and uncomment test-9 from bin/unittests
cc/ @portante

[arcolife]

test-9 (for this PR) fails at https://travis-ci.org/distributed-system-analysis/pbench/jobs/150102944#L784
..passes locally on laptop. A difference of time config on machines possibly an issue. Not sure.

[portante]

@arcolife, I tried this out this morning, and this is due to your timezone. :( I think the sadf command needs to be invoked with TZ=UTC, I'll post a PR with a fix to your repo, so that you can pull that in adding it to this PR.

[portante]

Okay, your branch need to be rebased on top of upstream master before we can work this through. Can you do that so I can send you a PR? @arcolife

[portante]

Any further updates based on the comments made here so far, @arcolife?

[arcolife]

extractsa is now a python script extract_sa.py that handles various scenarios as per #176 (comment)

Usage:

 ./extract_sa.py -h
Usage: extract_sa.py [--config <path-to-pbench-index.cfg>] <path-to-sa-binary>

Options:
  -h, --help            show this help message and exit
  -c CFG_NAME, --config=CFG_NAME
                        Specify config file

Example usage (independent of pbench):

 $ PYTHONPATH=./:$PYTHONPATH ./satools/extract_sa.py -c ../lib/config/pbench-index.cfg.example ~/sar.data 

Total incompatibility when a required specific sadf script is not found:

Invalid system activity file: /home/arcolife/Documents/sar_samples/test/sa04_65_local
File created by sar/sadc from sysstat version 9.0.4
Current sysstat version cannot read the format of this file (0x1170)

Cannot convert the format of this file

compatible OS version for this binary: 65
Appropriate sadf-<type>-<arch> script necessary to process this file not found!
Please provide a path to folder containing sadf script corresponding to Red Hat OS: 65
Path should be configured under [SAR] section in ../lib/config/pbench-index.cfg.example

Total incompatibility but sadf script found to deal with this:

Invalid system activity file: /home/arcolife/Documents/sar_samples/test/sa04_65_local
File created by sar/sadc from sysstat version 9.0.4
Current sysstat version cannot read the format of this file (0x1170)

Cannot convert the format of this file

compatible OS version for this binary: 65

Nodename: local.example.com
XML data saved to: /home/arcolife/Documents/sar_samples/test/sa04_65_local.xml

Partial incompatibility, dealt with conversion:

 ./extract_sa.py -c ../lib/config/pbench-server.cfg.example ~/Documents/sar_samples/test/datafile001 
Invalid system activity file: /home/arcolife/Documents/sar_samples/test/datafile001
File created by sar/sadc from sysstat version 10.1.5
Current sysstat version cannot read the format of this file (0x2171)

file_magic: OK
file_header: OK
file_activity: A_CPU A_PCSW A_IRQ A_SWAP A_PAGE A_IO A_MEMORY A_HUGE A_KTABLES A_QUEUE A_DISK A_NET_DEV A_NET_EDEV A_NET_NFS A_NET_NFSD A_NET_SOCK A_NET_IP A_NET_EIP A_NET_ICMP A_NET_EICMP A_NET_TCP A_NET_ETCP A_NET_UDP A_NET_SOCK6 A_NET_IP6 A_NET_EIP6 A_NET_ICMP6 A_NET_EICMP6 A_NET_UDP6 A_PWR_CPUFREQ A_PWR_FAN A_PWR_TEMP A_PWR_IN A_PWR_USB OK
Statistics: .........................................................................................................................
File successfully converted to sysstat format version 11.2.0

converted binary to current version of sysstat..
None

Nodename: benchserver2
XML data saved to: /home/arcolife/Documents/sar_samples/test/datafile001.xml

Compatible versions:

 ./extract_sa.py -c ../lib/config/pbench-server.cfg.example ~/Documents/sar_samples/test/sa10 

Nodename: localhost.localdomain
XML data saved to: /home/arcolife/Documents/sar_samples/test/sa10.xml

[arcolife]

TODOs

• update unit tests to accommodate this new change.
  • https://travis-ci.org/distributed-system-analysis/pbench/jobs/224676630#L876
  • also check if pbench-sysstat is installed (same version) in Travis as on local machine (nope! it's not up to date on Travis, hence the failure in unit test above. Gotta add checks noted here)
• update sysstat.py for new file magic
• refactor index-pbench
• update satools (if possible/needed) with https://github.com/distributed-system-analysis/satools
  • [WIP] [WIP] Add 0x2173 support to format_magic class_map satools#1
• squash commits after unit tests pass

Questions

1. +# why are we using c_uint8 instead of the designated c_ubyte for representing unsigned char?
2. Although 0x2173 works for fedora 22 / sysstat 11.1.2-3, but I've included 'f24' in class map, and corresponding sysstat package - 11.2.0 .. Is that the right thing to do (show latest compatibility) ?

# (Pdb) fh.dump(magic)
# file_header (0x2173):
# 	sa_ust_time 288 1970-01-01 00:04:48
# 	sa_last_cpu_nr 0
# 	sa_act_nr 0
# 	sa_vol_act_nr 0
# 	sa_day <sysstat.c_ubyte_Array_4 object at 0x7f433d5ae9d8>
# 	sa_month <sysstat.c_ubyte_Array_4 object at 0x7f433d5ae9d8>
# 	sa_year <sysstat.c_ubyte_Array_4 object at 0x7f433d5ae9d8>
# 	sa_sizeof_long b'\x00'
# 	sa_sysname b''
# 	sa_nodename b''
# 	sa_release b'ost.localdomain'
# 	sa_machine b'200.fc22.x86_64'  <------ This shows up in fh dump from a binary probably generated by fedora 22.

[portante]

Replaced by #645.