Add API JSON log handling #11171

Kondent · 2021-12-02T20:32:19Z

Related issue
This PR closes #10867

Description

Under this development, I made a few adjustments to make API log parsing easier by allowing our API to write them in JSON format, besides plain text.

Changes on this PR:

Added log format configuration block to api.yaml.
Fixed default api.yaml from AIT.
Fixed all related unittest.
- api/api/test/test_alogging.py.
- api/api/test/test_configuration.py.
- framework/wazuh/core/cluster/tests/test_utils.py.
- framework/wazuh/core/tests/test_wlogging.py.
Added few minor changes at different modules:
- api/api/configuration.py.
- api/api/validator.py.
- api/scripts/wazuh-apid.py.
- framework/scripts/wazuh-clusterd.py.
- framework/wazuh/core/wlogging.py.
- framework/requirements.txt.
- api/api/alogging.py.

API Default Configuration Block:

logs:
  level: "info"
  format: "plain"

NOTE: format allowed values are json, plain, json,plain and plain,json

JSON Log Handler

class WazuhJsonFormatter(jsonlogger.JsonFormatter):
    """
    Defines the custom JSON log formatter used by wlogging
    """
    def add_fields(self, log_record, record, message_dict):
        # Request handling
        if record.message is None:
            record.message = {
                'type': 'request',
                'payload': message_dict
                }
        else:
            # Traceback handling
            traceback = message_dict.get('exc_info')
            if traceback is not None:
                record.message = {
                    'type': 'error',
                    'payload': f'{record.message}. {traceback}'
                    }
            else:
                # Plain text messages
                record.message = {
                    'type': 'informative',
                    'payload': record.message
                    }
        log_record['timestamp'] = self.formatTime(record, self.datefmt)
        log_record['levelname'] = record.levelname
        log_record['data'] = record.message

With this customization of python-json-logger we're adding the information to the log entry with the formatting we want.
In other words, API json log will have three keys: timestamp, levelname and data. The last key is a dict with two more keys: type and payload.

type: str. Value could be informative, error or request.
payload: str or dict. Value could be a plain text message or a dict if the value for the type key is request.

NOTE: every log entry will have these three main keys, even with log level set as DEBUG or DEBUG2

API JSON Log Sample:

Starting

{
  "timestamp": "2021/12/14 10:11:36",
  "levelname": "INFO",
  "data": {
    "type": "informative",
    "payload": "Listening on 0.0.0.0:55000.."
  }
}

Request

{
  "timestamp": "2021/12/14 11:12:04",
  "levelname": "INFO",
  "data": {
    "type": "request",
    "payload": {
      "user": "wazuh",
      "ip": "172.19.0.1",
      "http_method": "GET",
      "uri": "GET /security/user/authenticate",
      "parameters": {},
      "body": {},
      "time": "0.276s",
      "status_code": 200
    }
  }
}

{
  "timestamp": "2021/12/14 11:12:04",
  "levelname": "INFO",
  "data": {
    "type": "request",
    "payload": {
      "user": "wazuh",
      "ip": "172.19.0.1",
      "http_method": "GET",
      "uri": "GET /agents",
      "parameters": {
        "offset": "0",
        "limit": "2",
        "select": "id"
      },
      "body": {},
      "time": "0.044s",
      "status_code": 200
    }
  }
}

Unhandled Exception

{
  "timestamp": "2021/12/14 10:13:22",
  "levelname": "ERROR",
  "data": {
    "type": "error",
    "payload": "Error handling request. Traceback (most recent call last):\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.4.0-py3.9.egg/wazuh/core/cluster/control.py\", line 56, in get_nodes\n    result = json.loads(response, object_hook=as_wazuh_object)\n  File \"/var/ossec/framework/python/lib/python3.9/json/__init__.py\", line 359, in loads\n    return cls(**kw).decode(s)\n  File \"/var/ossec/framework/python/lib/python3.9/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/var/ossec/framework/python/lib/python3.9/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/connexion/apis/aiohttp_api.py\", line 50, in problems_middleware\n    response = yield from handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/aiohttp/web_middlewares.py\", line 110, in impl\n    return await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/api-4.4.0-py3.9.egg/api/middlewares.py\", line 140, in response_postprocessing\n    return await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/api-4.4.0-py3.9.egg/api/middlewares.py\", line 33, in set_user_name\n    return await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/api-4.4.0-py3.9.egg/api/middlewares.py\", line 117, in security_middleware\n    return await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/api-4.4.0-py3.9.egg/api/middlewares.py\", line 89, in request_logging\n    return await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/api-4.4.0-py3.9.egg/api/middlewares.py\", line 38, in set_secure_headers\n    resp = await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/aiohttp_cache/middleware.py\", line 59, in cache_middleware\n    return await handler(request)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/connexion/decorators/coroutine_wrappers.py\", line 23, in wrapper\n    connexion_response = yield from connexion_response\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/api-4.4.0-py3.9.egg/api/controllers/cluster_controller.py\", line 33, in get_cluster_node\n    nodes = raise_if_exc(await get_system_nodes())\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.4.0-py3.9.egg/wazuh/core/cluster/control.py\", line 198, in get_system_nodes\n    result = await get_nodes(lc)\n  File \"/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.4.0-py3.9.egg/wazuh/core/cluster/control.py\", line 58, in get_nodes\n    raise WazuhClusterError(3020) if timeout in response else e\nwazuh.core.exception.WazuhClusterError: Error 3020 - Timeout sending request"
  }
}

Starting with `DEBUG2`

{
  "timestamp": "2021/12/14 11:16:04",
  "levelname": "DEBUG",
  "data": {
    "type": "informative",
    "payload": "Loaded API configuration: {host: 0.0.0.0, port: 55000, drop_privileges: True, experimental_features: False, max_upload_size: 10485760, intervals: {request_timeout: 10}, https: {enabled: True, key: /var/ossec/api/configuration/ssl/server.key, cert: /var/ossec/api/configuration/ssl/server.crt, use_ca: False, ca: /var/ossec/api/configuration/ssl/ca.crt, ssl_protocol: TLSv1.2, ssl_ciphers: }, logs: {level: debug2, path: /var/ossec/logs, format: json}, cors: {enabled: False, source_route: *, expose_headers: *, allow_headers: *, allow_credentials: False}, cache: {enabled: True, time: 0.75}, access: {max_login_attempts: 50, block_time: 300, max_request_per_minute: 300}, remote_commands: {localfile: {enabled: True, exceptions: []}, wodle_command: {enabled: True, exceptions: []}}}"
  }
}

{
  "timestamp": "2021/12/14 11:16:04",
  "levelname": "DEBUG",
  "data": {
    "type": "informative",
    "payload": "Loaded security API configuration: {auth_token_exp_timeout: 900, rbac_mode: white}"
  }
}

{
  "timestamp": "2021/12/14 11:16:04",
  "levelname": "DEBUG",
  "data": {
    "type": "informative",
    "payload": "Starting aiohttp HTTP server.."
  }
}

{
  "timestamp": "2021/12/14 11:16:04",
  "levelname": "INFO",
  "data": {
    "type": "informative",
    "payload": "Listening on 0.0.0.0:55000.."
  }
}

Foreground mode - JSON

logs:
 level: "info"
 format: "json"

root@2e4546d21bb5:/var/ossec# ./bin/wazuh-apid -f
Starting API in foreground
2022/02/14 21:44:00 INFO: Listening on 0.0.0.0:55000..
======== Running on https://0.0.0.0:55000 ========
(Press CTRL+C to quit)
2022/02/14 21:44:03 INFO: wazuh 172.30.0.1 "GET /security/user/authenticate" with parameters {} and body {} done in 0.004s: 400

root@2e4546d21bb5:/var/ossec# cat logs/api.json 
{"timestamp": "2022/02/14 21:44:00", "levelname": "INFO", "data": {"type": "informative", "payload": "Listening on 0.0.0.0:55000.."}}
{"timestamp": "2022/02/14 21:44:03", "levelname": "INFO", "data": {"type": "request", "payload": {"user": "wazuh", "ip": "172.30.0.1", "http_method": "GET", "uri": "GET /security/user/authenticate", "parameters": {}, "body": {}, "time": "0.004s", "status_code": 400}}}

Foreground mode - PLAIN

logs:
 level: "info"
 format: "plain"

root@2e4546d21bb5:/var/ossec# ./bin/wazuh-apid -f
Starting API in foreground
2022/02/14 21:45:21 INFO: Listening on 0.0.0.0:55000..
======== Running on https://0.0.0.0:55000 ========
(Press CTRL+C to quit)
2022/02/14 21:45:23 INFO: wazuh 172.30.0.1 "GET /security/user/authenticate" with parameters {} and body {} done in 0.006s: 400

root@2e4546d21bb5:/var/ossec# cat logs/api.log 
2022/02/14 21:45:21 INFO: Listening on 0.0.0.0:55000..
2022/02/14 21:45:23 INFO: wazuh 172.30.0.1 "GET /security/user/authenticate" with parameters {} and body {} done in 0.006s: 400

Foreground mode - BOTH

logs:
 level: "info"
 format: "plain,json"

root@2e4546d21bb5:/var/ossec# ./bin/wazuh-apid -f
Starting API in foreground
2022/02/14 21:46:09 INFO: Listening on 0.0.0.0:55000..
======== Running on https://0.0.0.0:55000 ========
(Press CTRL+C to quit)
2022/02/14 21:46:11 INFO: wazuh 172.30.0.1 "GET /security/user/authenticate" with parameters {} and body {} done in 0.006s: 400

root@2e4546d21bb5:/var/ossec# cat logs/api.log
2022/02/14 21:46:09 INFO: Listening on 0.0.0.0:55000..
2022/02/14 21:46:11 INFO: wazuh 172.30.0.1 "GET /security/user/authenticate" with parameters {} and body {} done in 0.006s: 400

root@2e4546d21bb5:/var/ossec# cat logs/api.json
{"timestamp": "2022/02/14 21:46:09", "levelname": "INFO", "data": {"type": "informative", "payload": "Listening on 0.0.0.0:55000.."}}
{"timestamp": "2022/02/14 21:46:11", "levelname": "INFO", "data": {"type": "request", "payload": {"user": "wazuh", "ip": "172.30.0.1", "http_method": "GET", "uri": "GET /security/user/authenticate", "parameters": {}, "body": {}, "time": "0.006s", "status_code": 400}}}

Unit Test: API

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api
plugins: html-2.1.1, asyncio-0.15.1, metadata-1.11.0, aiohttp-0.3.0, trio-0.7.0, cov-2.10.1
collected 505 items                                                                                                                                                                          

api/controllers/test/test_active_response_controller.py .                                                                                                                              [  0%]
api/controllers/test/test_agent_controller.py ..........................................                                                                                               [  8%]
api/controllers/test/test_cdb_list_controller.py ......                                                                                                                                [  9%]
api/controllers/test/test_ciscat_controller.py .                                                                                                                                       [  9%]
api/controllers/test/test_cluster_controller.py ......................                                                                                                                 [ 14%]
api/controllers/test/test_decoder_controller.py .......                                                                                                                                [ 15%]
api/controllers/test/test_default_controller.py .                                                                                                                                      [ 15%]
api/controllers/test/test_experimental_controller.py ...............                                                                                                                   [ 18%]
api/controllers/test/test_manager_controller.py .................                                                                                                                      [ 22%]
api/controllers/test/test_mitre_controller.py .......                                                                                                                                  [ 23%]
api/controllers/test/test_overview_controller.py .                                                                                                                                     [ 23%]
api/controllers/test/test_rootcheck_controller.py ....                                                                                                                                 [ 24%]
api/controllers/test/test_rule_controller.py ........                                                                                                                                  [ 26%]
api/controllers/test/test_sca_controller.py ..                                                                                                                                         [ 26%]
api/controllers/test/test_security_controller.py ...................................................                                                                                   [ 36%]
api/controllers/test/test_syscheck_controller.py ....                                                                                                                                  [ 37%]
api/controllers/test/test_syscollector_controller.py .........                                                                                                                         [ 39%]
api/controllers/test/test_task_controller.py .                                                                                                                                         [ 39%]
api/controllers/test/test_vulnerability_controller.py ..                                                                                                                               [ 39%]
api/models/test/test_model.py ...........................                                                                                                                              [ 45%]
api/test/test_alogging.py .................                                                                                                                                            [ 48%]
api/test/test_authentication.py ..........                                                                                                                                             [ 50%]
api/test/test_configuration.py ..........................................                                                                                                              [ 58%]
api/test/test_encoder.py ...                                                                                                                                                           [ 59%]
api/test/test_middlewares.py ..........                                                                                                                                                [ 61%]
api/test/test_uri_parser.py ...                                                                                                                                                        [ 61%]
api/test/test_util.py ......................................                                                                                                                           [ 69%]
api/test/test_validator.py ..........................................................................................................................................................  [100%]

============================================================================== 505 passed, 14 warnings in 2.16s ==============================================================================

Unit Test: Framework

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/framework
plugins: html-2.1.1, asyncio-0.15.1, metadata-1.11.0, aiohttp-0.3.0, trio-0.7.0, cov-2.10.1
collected 1841 items                                                                                                                                                                         

wazuh/core/cluster/dapi/tests/test_dapi.py ..............................                                                                                                              [  1%]
wazuh/core/cluster/tests/test_client.py ................                                                                                                                               [  2%]
wazuh/core/cluster/tests/test_cluster.py .............................                                                                                                                 [  4%]
wazuh/core/cluster/tests/test_common.py ............................................................                                                                                   [  7%]
wazuh/core/cluster/tests/test_control.py .....                                                                                                                                         [  7%]
wazuh/core/cluster/tests/test_local_client.py .............                                                                                                                            [  8%]
wazuh/core/cluster/tests/test_local_server.py .......................                                                                                                                  [  9%]
wazuh/core/cluster/tests/test_master.py ..........................................                                                                                                     [ 11%]
wazuh/core/cluster/tests/test_server.py ...................                                                                                                                            [ 12%]
wazuh/core/cluster/tests/test_utils.py .......                                                                                                                                         [ 13%]
wazuh/core/cluster/tests/test_worker.py ...............................                                                                                                                [ 14%]
wazuh/core/tests/test_active_response.py ..............                                                                                                                                [ 15%]
wazuh/core/tests/test_agent.py ..............................................................................................................................................          [ 23%]
wazuh/core/tests/test_cdb_list.py ......................................                                                                                                               [ 25%]
wazuh/core/tests/test_common.py .......                                                                                                                                                [ 25%]
wazuh/core/tests/test_configuration.py ....................................                                                                                                            [ 27%]
wazuh/core/tests/test_database.py .............                                                                                                                                        [ 28%]
wazuh/core/tests/test_decoder.py ................                                                                                                                                      [ 29%]
wazuh/core/tests/test_exception.py ...                                                                                                                                                 [ 29%]
wazuh/core/tests/test_input_validator.py ...                                                                                                                                           [ 29%]
wazuh/core/tests/test_logtest.py ..                                                                                                                                                    [ 29%]
wazuh/core/tests/test_manager.py ...............                                                                                                                                       [ 30%]
wazuh/core/tests/test_mitre.py .............                                                                                                                                           [ 31%]
wazuh/core/tests/test_pyDaemonModule.py .....                                                                                                                                          [ 31%]
wazuh/core/tests/test_results.py ........................................                                                                                                              [ 33%]
wazuh/core/tests/test_rootcheck.py .............                                                                                                                                       [ 34%]
wazuh/core/tests/test_rule.py ......................                                                                                                                                   [ 35%]
wazuh/core/tests/test_sca.py ............                                                                                                                                              [ 36%]
wazuh/core/tests/test_security.py ............                                                                                                                                         [ 36%]
wazuh/core/tests/test_stats.py ............                                                                                                                                            [ 37%]
wazuh/core/tests/test_syscheck.py .......                                                                                                                                              [ 38%]
wazuh/core/tests/test_syscollector.py ...                                                                                                                                              [ 38%]
wazuh/core/tests/test_task.py ........                                                                                                                                                 [ 38%]
wazuh/core/tests/test_utils.py ....................................................................................................................................................... [ 46%]
.............................................................................                                                                                                          [ 51%]
wazuh/core/tests/test_vulnerability.py .                                                                                                                                               [ 51%]
wazuh/core/tests/test_wazuh_queue.py ..................                                                                                                                                [ 52%]
wazuh/core/tests/test_wazuh_socket.py ....................                                                                                                                             [ 53%]
wazuh/core/tests/test_wdb.py ........................                                                                                                                                  [ 54%]
wazuh/core/tests/test_wlogging.py .........                                                                                                                                            [ 54%]
wazuh/rbac/tests/test_auth_context.py ..                                                                                                                                               [ 55%]
wazuh/rbac/tests/test_decorators.py .........................................................................................................                                          [ 60%]
wazuh/rbac/tests/test_default_configuration.py .......................................................                                                                                 [ 63%]
wazuh/rbac/tests/test_orm.py ......................................................                                                                                                    [ 66%]
wazuh/rbac/tests/test_preprocessor.py ...........                                                                                                                                      [ 67%]
wazuh/tests/test_active_response.py ............                                                                                                                                       [ 67%]
wazuh/tests/test_agent.py .....................................................................................................................                                        [ 74%]
wazuh/tests/test_cdb_list.py .....................................................                                                                                                     [ 77%]
wazuh/tests/test_ciscat.py .................................                                                                                                                           [ 78%]
wazuh/tests/test_cluster.py .......                                                                                                                                                    [ 79%]
wazuh/tests/test_decoder.py ...................................                                                                                                                        [ 81%]
wazuh/tests/test_group.py ........                                                                                                                                                     [ 81%]
wazuh/tests/test_logtest.py ......                                                                                                                                                     [ 81%]
wazuh/tests/test_manager.py ....................................                                                                                                                       [ 83%]
wazuh/tests/test_mitre.py .......                                                                                                                                                      [ 84%]
wazuh/tests/test_rootcheck.py ..................................................                                                                                                       [ 87%]
wazuh/tests/test_rule.py ........................................................                                                                                                      [ 90%]
wazuh/tests/test_sca.py .......                                                                                                                                                        [ 90%]
wazuh/tests/test_security.py .........................................................................                                                                                 [ 94%]
wazuh/tests/test_stats.py .......                                                                                                                                                      [ 94%]
wazuh/tests/test_syscheck.py .........................                                                                                                                                 [ 96%]
wazuh/tests/test_syscollector.py ............                                                                                                                                          [ 96%]
wazuh/tests/test_task.py ............................                                                                                                                                  [ 98%]
wazuh/tests/test_vulnerability.py ...............................                                                                                                                      [100%]

====================================================================== 1841 passed, 3775 warnings in 244.43s (0:04:04) =======================================================================

Related Integration tests (dependency installed in a local environment):

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 6 items                                                                                                                                                                            

test_agent_DELETE_endpoints.tavern.yaml ......                                                                                                                                         [100%]

=========================================================================== 6 passed, 1 warnings in 500.74 seconds ===========================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 91 items                                                                                                                                                                           

test_agent_GET_endpoints.tavern.yaml ...........................................................................................                                                       [100%]

========================================================================== 91 passed, 1 warnings in 183.28 seconds ===========================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 5 items                                                                                                                                                                            

test_agent_POST_endpoints.tavern.yaml .....                                                                                                                                            [100%]

=========================================================================== 5 passed, 1 warnings in 333.15 seconds ===========================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 45 items                                                                                                                                                                           

test_cluster_endpoints.tavern.yaml .............................................                                                                                                       [100%]

========================================================================== 45 passed, 1 warnings in 469.03 seconds ===========================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 12 items                                                                                                                                                                           

test_experimental_endpoints.tavern.yaml ...XXXXX.X..                                                                                                                                   [100%]

====================================================================== 6 passed, 6 xpassed, 1 warnings in 95.96 seconds ======================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 15 items                                                                                                                                                                           

test_security_DELETE_endpoints.tavern.yaml ...............                                                                                                                             [100%]

=========================================================================== 15 passed, 1 warnings in 78.89 seconds ===========================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 11 items                                                                                                                                                                           

test_security_GET_endpoints.tavern.yaml ...........                                                                                                                                    [100%]

=========================================================================== 11 passed, 1 warnings in 61.89 seconds ===========================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 7 items                                                                                                                                                                            

test_security_POST_endpoints.tavern.yaml .......                                                                                                                                       [100%]

=========================================================================== 7 passed, 1 warnings in 57.21 seconds ============================================================================
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.5, pytest-5.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /home/kondent/git/wazuh/api/test/integration, inifile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, tavern-1.2.2
collected 9 items                                                                                                                                                                            

test_security_PUT_endpoints.tavern.yaml .........                                                                                                                                      [100%]

=========================================================================== 9 passed, 1 warnings in 66.15 seconds ============================================================================

Dependencies Scanner:

⩘ ~/git/wazuh-qa/tests/scans/dependencies ⩫ master ⨍ python3 -m pytest test_dependencies.py --branch=feature/10867-api-json-log
...
⩘ ~/git/wazuh-qa/tests/scans/dependencies ⩫ master ⨍ grep 'python-json-logger' report_file.json 
⩘ ~/git/wazuh-qa/tests/scans/dependencies ⩫ master ⨍

Full report.json

{
    "report_date": "2022-02-17T14:19:38.484238",
    "vulnerabilities_found": 5,
    "packages": [
        {
            "package_name": "python",
            "package_version": "3.9.5",
            "package_affected_version": ">0",
            "vuln_description": "Lib/zipfile.py in Python allows remote attackers to cause a denial of service (resource consumption) via a ZIP bomb.\r\nhttps://bugs.python.org/issue36260",
            "safety_id": "37832"
        },
        {
            "package_name": "python",
            "package_version": "3.9.5",
            "package_affected_version": ">0",
            "vuln_description": "In difflib module, table header in output of difflib.HtmlDiff.make_table is not escaped and can be rendered as code in the browser, leading potentially to XSS.\r\nhttps://bugs.python.org/issue35603\r\nhttps://github.com/python/cpython/commit/44e36e80456dabaeb59c6e2a93e0c1322bfeb179",
            "safety_id": "42393"
        },
        {
            "package_name": "python",
            "package_version": "3.9.5",
            "package_affected_version": ">=3.9.0a0,<3.9.6",
            "vuln_description": "Python versions 3.6.14, 3.7.11, 3.8.11, 3.9.6 and 3.10.0b2 make urllib.parse sanitize urls containing ASCII newline and tabs.\r\nhttps://bugs.python.org/issue43882",
            "safety_id": "42384"
        },
        {
            "package_name": "python",
            "package_version": "3.9.5",
            "package_affected_version": ">=3.9.0a0,<3.9.6",
            "vuln_description": "Python versions 3.6.14, 3.7.11, 3.8.11, 3.9.6 and 3.10.0b2 include a fix for CVE-2021-3737: An improperly handled HTTP response in the HTTP client code of python may allow a remote attacker, who controls the HTTP server, to make the client script enter an infinite loop, consuming CPU time. The highest threat from this vulnerability is to system availability.\r\nhttps://access.redhat.com/security/cve/CVE-2021-3737\r\nhttps://bugs.python.org/issue44022",
            "safety_id": "42380"
        },
        {
            "package_name": "python",
            "package_version": "3.9.5",
            "package_affected_version": ">=3.9.0a0,<3.9.7",
            "vuln_description": "Python versions 3.6.15, 3.7.12, 3.8.12, 3.9.7 and 3.10.0rc2 fix a smtplib multiple CRLF injection vulnerability.\r\nhttps://bugs.python.org/issue43124",
            "safety_id": "42379"
        }
    ]
}

api/api/alogging.py

api/api/test/test_alogging.py

api/api/alogging.py

api/api/test/test_configuration.py

api/scripts/wazuh-apid.py

framework/requirements.txt

framework/wazuh/core/wlogging.py

mcarmona99

LGTM

* Add API JSON Logs * Fix related unittest * Fix API config from AIT * Fix alogging unittest and wazuhjsonformatter * Add docstrings * Add traceback message handling * Fix wazuhjsonformatter and related unittest * Add few improvements regarding key names and debug2 handling * Fix unittest for alogging * Add minor fixes after the rebase from master * Add minor fixes at related unit test * Add support for logging into both file types at the same time * Fix related unit test * Fix CustomFilter class and add related unit test * Add foreground fix related with default formatter * Fix related unit test * Add log rotation support for json files * Add requested changes: docstrings and minor fixes * Fix log rotation unit test * Remove python-json-logger from requirements file. Added in #11153 Co-authored-by: Alexis Rivas <alexis.rivas@wazuh.com>

…ncy (#13197) * Add python-json-logger to requirements.txt (#11153) * Add API JSON log handling (#11171) * Add API JSON Logs * Fix related unittest * Fix API config from AIT * Fix alogging unittest and wazuhjsonformatter * Add docstrings * Add traceback message handling * Fix wazuhjsonformatter and related unittest * Add few improvements regarding key names and debug2 handling * Fix unittest for alogging * Add minor fixes after the rebase from master * Add minor fixes at related unit test * Add support for logging into both file types at the same time * Fix related unit test * Fix CustomFilter class and add related unit test * Add foreground fix related with default formatter * Fix related unit test * Add log rotation support for json files * Add requested changes: docstrings and minor fixes * Fix log rotation unit test * Remove python-json-logger from requirements file. Added in #11153 Co-authored-by: Alexis Rivas <alexis.rivas@wazuh.com> Co-authored-by: Alexis Rivas <84642680+Kondent@users.noreply.github.com> Co-authored-by: Alexis Rivas <alexis.rivas@wazuh.com>

Kondent self-assigned this Dec 2, 2021

Kondent requested a review from davidjiglesias December 2, 2021 20:32

Kondent mentioned this pull request Dec 3, 2021

Fix API log format references wazuh/wazuh-documentation#4651

Merged

5 tasks

Kondent marked this pull request as ready for review December 10, 2021 19:00

Kondent marked this pull request as draft December 13, 2021 20:43

Kondent force-pushed the feature/10867-api-json-log branch 3 times, most recently from 1b1c8bc to 3560b61 Compare December 14, 2021 13:56

Kondent marked this pull request as ready for review December 14, 2021 15:04

Kondent marked this pull request as draft February 8, 2022 14:42

Kondent force-pushed the feature/10867-api-json-log branch 3 times, most recently from 6337ad3 to 3f1f2f2 Compare February 11, 2022 17:18

Kondent marked this pull request as ready for review February 11, 2022 19:57

Kondent marked this pull request as draft February 14, 2022 15:05

Kondent force-pushed the feature/10867-api-json-log branch 3 times, most recently from e14b911 to 5c78316 Compare February 14, 2022 19:55

Kondent marked this pull request as ready for review February 14, 2022 21:37

Kondent force-pushed the feature/10867-api-json-log branch from 04d3565 to 66f1103 Compare February 15, 2022 19:27

Kondent changed the base branch from master to dev-support-api-json-log February 15, 2022 19:27

Kondent mentioned this pull request Feb 16, 2022

Add QA integration tests for the API JSON log support wazuh/wazuh-qa#2580

Closed

mcarmona99 suggested changes Feb 17, 2022

View reviewed changes

Kondent marked this pull request as draft February 17, 2022 15:52

Kondent requested a review from mcarmona99 February 17, 2022 17:59

Kondent marked this pull request as ready for review February 17, 2022 17:59

mcarmona99 approved these changes Feb 18, 2022

View reviewed changes

Kondent force-pushed the feature/10867-api-json-log branch from a600133 to 43a38e7 Compare February 25, 2022 13:46

This was referenced Mar 4, 2022

API - Test the format of the logs (JSON logs support) wazuh/wazuh-qa#2635

Merged

Test API logs rotation wazuh/wazuh-qa#2646

Closed

Alexis Rivas added 18 commits March 18, 2022 16:13

Fix API config from AIT

3ef4c5a

Fix alogging unittest and wazuhjsonformatter

8ea2df4

Add docstrings

ddf3e0c

Add traceback message handling

939693d

Fix wazuhjsonformatter and related unittest

fe56b04

Add few improvements regarding key names and debug2 handling

1ce588e

Fix unittest for alogging

8c033a7

Add minor fixes after the rebase from master

c09ae3a

Add minor fixes at related unit test

8698337

Add support for logging into both file types at the same time

6d9f724

Fix related unit test

94273c2

Fix CustomFilter class and add related unit test

dd47241

Add foreground fix related with default formatter

0e505e4

Fix related unit test

f1277b5

Add log rotation support for json files

1ff0f05

Add requested changes: docstrings and minor fixes

164e13f

Fix log rotation unit test

b01ef41

Remove python-json-logger from requirements file. Added in #11153

bfba0a8

mcarmona99 force-pushed the feature/10867-api-json-log branch from 9963eef to bfba0a8 Compare March 18, 2022 15:14

mcarmona99 marked this pull request as ready for review March 18, 2022 15:38

davidjiglesias approved these changes Apr 20, 2022

View reviewed changes

davidjiglesias merged commit 1f870bc into dev-support-api-json-log Apr 20, 2022

davidjiglesias deleted the feature/10867-api-json-log branch April 20, 2022 08:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add API JSON log handling #11171

Add API JSON log handling #11171

Kondent commented Dec 2, 2021 •

edited

mcarmona99 left a comment

Add API JSON log handling #11171

Add API JSON log handling #11171

Conversation

Kondent commented Dec 2, 2021 • edited

Description

API Default Configuration Block:

JSON Log Handler

API JSON Log Sample:

Starting

Request

Unhandled Exception

Starting with DEBUG2

Foreground mode - JSON

Foreground mode - PLAIN

Foreground mode - BOTH

Unit Test: API

Unit Test: Framework

Related Integration tests (dependency installed in a local environment):

Dependencies Scanner:

mcarmona99 left a comment

Choose a reason for hiding this comment

Kondent commented Dec 2, 2021 •

edited

Starting with `DEBUG2`