CMSSW log collection

[smorovic]

CMSSW stderr and stdout is redirected to log files located in /var/log/hltd/pid.
A new script is developed to scan this directory and parse output of appearing log files. Script is started as a child process by hltd.

Messages are parsed into json documents which contain category, severity, module name, instance, function call, framework state, timestamp and message content of logs. Recognized messages are those produced by the MessageLogger (DEBUG,INFO,WARNING,ERROR and FATAL) as well as stack trace from a crash (considered FATAL). Framework report information is currently ignored.

Messages are also scanned to calculate a "lexicalId" hash which can be used, for example, for rate reduction of similar messages later in the chain.
Depending on the "hltd es_cmssw_log_level" parameter in hltd.conf, threshold is set for minimum log level to store in the elasticsearch.

Presently the same index as for other run-related information is used, however this can be changed if necessary.

A tool "test/logprint.py" is also provided, doing time-window queries in elasticsearch and printing collected messages in a way similar to the Handsaw tool.