[feat] Allow custom JSON formatting for log records in the httpjson handler

[vkarak]

This PR introduces two new configuration parameters for the httpjson handler:

1. json_formatter: this is a callable that will be passed the log record and it should return a valid JSON-encoded string. This configuration parameter is only valid for the Python config.
2. debug: if specified, then the server will not be contacted, but instead the encoded record will be dumped in a file in the current working directory.

This PR also changes how check_perfvalues are logged by the httpjson handler. Similarly to the filelog handler these are flattened and a separate batch of properties is created for each performance variable. For example, the various performance variables of the STREAM benchmark (Copy, Scale, etc.) are logged as follows:

{
  "check_perf_Copy_value": 16940.8,
  "check_perf_Copy_ref": 0,
  "check_perf_Copy_lower_thres": null,
  "check_perf_Copy_upper_thres": null,
  "check_perf_Copy_unit": "MB/s",
  "check_perf_Scale_value": 13619.6,
  "check_perf_Scale_ref": 0,
  "check_perf_Scale_lower_thres": null,
  "check_perf_Scale_upper_thres": null,
  "check_perf_Scale_unit": "MB/s",
  "check_perf_Add_value": 14997.2,
  "check_perf_Add_ref": 0,
  "check_perf_Add_lower_thres": null,
  "check_perf_Add_upper_thres": null,
  "check_perf_Add_unit": "MB/s",
  "check_perf_Triad_value": 15254.9,
  "check_perf_Triad_ref": 0,
  "check_perf_Triad_lower_thres": null,
  "check_perf_Triad_upper_thres": null,
  "check_perf_Triad_unit": "MB/s",
}

I have also added some unit test that basically check the initialisation of the handler but not its functionality. For functionality tests, we would need something more advanced as described in #2760.

Todos

• Do not require the port in the URL for the httpjson handler
• Update the documentation

[codecov-commenter]

Codecov Report

Base: 86.61% // Head: 86.56% // Decreases project coverage by -0.05% ⚠️

Coverage data is based on head (c38c1fe) compared to base (ad2d27e).
Patch coverage: 36.61% of modified lines in pull request are covered.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2752      +/-   ##
===========================================
- Coverage    86.61%   86.56%   -0.05%     
===========================================
  Files           60       60              
  Lines        11234    11287      +53     
===========================================
+ Hits          9730     9771      +41     
- Misses        1504     1516      +12     

Impacted Files	Coverage Δ
reframe/core/logging.py	80.64% <36.61%> (-0.35%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ekouts]

Still going through the PR but a small comment, I am getting an error with python 3.6 because of the time_ns function

Traceback (most recent call last):
  File "/users/eirinik/repos/reframe/reframe/core/logging.py", line 795, in log_performance
    self._log_performance(level, task, msg, multiline)
  File "/users/eirinik/repos/reframe/reframe/core/logging.py", line 821, in _log_performance
    self.log(level, msg)
  File "/users/eirinik/repos/reframe/reframe/core/logging.py", line 838, in log
    super().log(level, msg, *args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1674, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1374, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
    self.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/users/eirinik/repos/reframe/reframe/core/logging.py", line 597, in emit
    ts = time.time_ns()
AttributeError: module 'time' has no attribute 'time_ns'

[vkarak]

@ekouts Fixed.

[vkarak]

@jenkins-cscs retry all