Merge group files endpoints #21653

GGP1 · 2024-01-29T16:31:14Z

Related issue
Closes #20851

Description

Merges the /groups/:group_id/files/:file_name/json and /groups/:group_id/files/:file_name/xml endpoints into a single one that takes the raw parameter which defines whether the response content will be formatted in json or plain text. The response content-type is defined based on the file extension retrieved.

Logs/Alerts example

Get merged.mg in plain text

GET /groups/default/files/merged.mg?raw=true

#default
!228 ar.conf
restart-ossec0 - restart-ossec.sh - 0
restart-ossec0 - restart-ossec.cmd - 0
restart-wazuh0 - restart-ossec.sh - 0
restart-wazuh0 - restart-ossec.cmd - 0
restart-wazuh0 - restart-wazuh - 0
restart-wazuh0 - restart-wazuh.exe - 0
!76 agent.conf
<agent_config>

  <!-- Shared agent configuration here -->

</agent_config>
!28411 cis_apache2224_rcl.txt
...

Get merged.mg in JSON

GET /groups/default/files/merged.mg

{
    "data": [
        {
            "file_name": "ar.conf",
            "file_size": 228,
            "file_content": "restart-ossec0 - restart-ossec.sh - 0\nrestart-ossec0 - restart-ossec.cmd - 0\nrestart-wazuh0 - restart-ossec.sh - 0\nrestart-wazuh0 - restart-ossec.cmd - 0\nrestart-wazuh0 - restart-wazuh - 0\nrestart-wazuh0 - restart-wazuh.exe - 0\n"
        },
        {
            "file_name": "agent.conf",
            "file_size": 76,
            "file_content": "<agent_config>\n\n  <!-- Shared agent configuration here -->\n\n</agent_config>\n"
        },
        ...
 	]
}

Get file content using a configuration type

GET /groups/default/files/agent.conf?type=conf

{
    "data": {
        "total_affected_items": 1,
        "affected_items": [
            {
                "filters": {},
                "config": {}
            }
        ]
    },
    "error": 0
}

Failed checks are not related to the changes introduced.

EduLeon12

Great work @GGP1

LGTM!,

My only question is if it would be better for readibility purposes to change the raw default value to a bool False instead of `None.

GGP1 · 2024-01-31T12:05:54Z

My only question is if it would be better for readibility purposes to change the raw default value to a bool False instead of `None.

I've seen both in the code, I don't think there's a standard but I agree that using False is slightly better so I modified it. Thanks!

Selutario

Looks good! Just some minor changes were requested. Also, I found a strange behaviour. I guess this is expected, but I would like to confirm.

I tried creating a test.json file. When the content-type returned is application/json, it is seen this way:

curl -k -X GET "https://localhost:55000/groups/default/files/test.json?raw=true" -H "Authorization: Bearer $TOKEN" -v

However, this does not happen when xml content is returned as application/xml:

Nor when the json file is returned as plain 🤔:

Selutario · 2024-01-31T17:04:00Z

api/api/controllers/agent_controller.py

@@ -1360,9 +1361,9 @@ async def get_group_files(request, group_id: str, pretty: bool = False, wait_for
    return web.json_response(data=data, status=200, dumps=prettify if pretty else dumps)


-async def get_group_file_json(request, group_id: str, file_name: str, pretty: bool = False,
+async def get_group_file(request, group_id: str, file_name: str, raw: bool = False, pretty: bool = False,
                              wait_for_complete: bool = False) -> web.Response:


Suggested change

wait_for_complete: bool = False) -> web.Response:

wait_for_complete: bool = False) -> web.Response | ConnexionResponse:

api/api/controllers/agent_controller.py

Selutario · 2024-01-31T17:08:18Z

api/api/controllers/agent_controller.py

-    response = ConnexionResponse(body=data["data"], mimetype='application/xml')
-
-    return response
+        return ConnexionResponse(body=data["data"], mimetype=mimetype)


Suggested change

return ConnexionResponse(body=data["data"], mimetype=mimetype)

return ConnexionResponse(body=data["data"], mimetype=mimetype)

Selutario · 2024-01-31T17:30:18Z

framework/wazuh/core/configuration.py

+
+    # ![file_size] [file_name]
+    regex_header = re.compile(r"^!(\d+)\s*(.*)")
+    regex_comment = re.compile(r"^\s*#")


I don't think it's worth skipping comments since the way to mark a comment could be different in each file inside the merged.mg. The agent.conf comments (for example) are still being included while those in rcl files are not.

The first line should be skipped though.

Selutario · 2024-01-31T17:58:50Z

framework/wazuh/core/configuration.py

@@ -770,33 +820,33 @@ def get_agent_conf_multigroup(multigroup_id: str = None, offset: int = 0, limit:
    return {'totalItems': len(data), 'items': cut_array(data, offset=offset, limit=limit)}


-def get_file_conf(filename: str, group_id: str = None, type_conf: str = None, return_format: str = None) -> dict:
-    """Return the configuration file as dictionary.
+def get_file_conf(filename: str, group_id: str = None, type_conf: str = None, raw: bool = False) -> dict:


Suggested change

def get_file_conf(filename: str, group_id: str = None, type_conf: str = None, raw: bool = False) -> dict:

def get_file_conf(filename: str, group_id: str = None, type_conf: str = None, raw: bool = False) -> dict | str:

Selutario · 2024-01-31T18:00:19Z

framework/wazuh/core/configuration.py


    Raises
    ------
    WazuhResourceNotFound(1710)
        Group was not found.
    WazuhError(1006)
-        agent.conf does not exist or there is a problem with the permissions.
+        The file does not exist or there is a problem with the permissions.
    WazuhError(1104)
        Invalid file type.

    Returns
    -------
    dict


Suggested change

dict

dict or str

GGP1 · 2024-02-01T12:31:17Z

Looks good! Just some minor changes were requested. Also, I found a strange behaviour. I guess this is expected, but I would like to confirm.

I tried creating a test.json file. When the content-type returned is application/json, it is seen this way:

curl -k -X GET "https://localhost:55000/groups/default/files/test.json?raw=true" -H "Authorization: Bearer $TOKEN" -v

@Selutario This is expected, the content of the file is returned as a string, and if the Content-Type is json then it will be interpreted as a json string instead of an object.

In order to include all the content inside the string, new lines have to be literal and double quotes have to be escaped. The other content types do not behave the same because they don't need to escape or wrap anything (and thus the output is the actual file content).

Selutario

LGTM. I think we'll need to update the documentation too, at least the RBAC reference.

Please, open a PR for this @GGP1

GGP1 self-assigned this Jan 29, 2024

GGP1 linked an issue Jan 29, 2024 that may be closed by this pull request

default group merged.mg error #20851

Closed

GGP1 force-pushed the fix/20851-merge-group-files-formats branch from 1412554 to 586795c Compare January 29, 2024 18:15

EduLeon12 previously approved these changes Jan 31, 2024

View reviewed changes

GGP1 dismissed EduLeon12’s stale review via f84e142 January 31, 2024 12:04

GGP1 added 7 commits January 31, 2024 09:08

Merge get_group_file endpoints

1d4cecf

Replace xml with plain text

6d76292

Update tests

f8348c9

Replace return_format with raw

525d15e

Add merged.mg json parsing

14fe609

Update docstrings

b301c55

Use false as the default value

cedccbc

GGP1 force-pushed the fix/20851-merge-group-files-formats branch from f84e142 to cedccbc Compare January 31, 2024 12:08

Selutario suggested changes Jan 31, 2024

View reviewed changes

Style fixes

b4f9e3a

GGP1 force-pushed the fix/20851-merge-group-files-formats branch from 02c18e5 to b4f9e3a Compare February 1, 2024 14:14

Selutario approved these changes Feb 1, 2024

View reviewed changes

Selutario merged commit 9e41df9 into master Feb 1, 2024
51 of 53 checks passed

Selutario deleted the fix/20851-merge-group-files-formats branch February 1, 2024 16:39

Selutario mentioned this pull request Feb 1, 2024

default group merged.mg error #20851

Closed

GGP1 mentioned this pull request Feb 1, 2024

Update RBAC reference wazuh/wazuh-documentation#7006

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge group files endpoints #21653

Merge group files endpoints #21653

GGP1 commented Jan 29, 2024 •

edited

Loading

EduLeon12 left a comment

GGP1 commented Jan 31, 2024

Selutario left a comment

Selutario Jan 31, 2024

Selutario Jan 31, 2024

Selutario Jan 31, 2024

Selutario Jan 31, 2024

Selutario Jan 31, 2024

GGP1 commented Feb 1, 2024

Selutario left a comment

	wait_for_complete: bool = False) -> web.Response:
	wait_for_complete: bool = False) -> web.Response \| ConnexionResponse:

	return ConnexionResponse(body=data["data"], mimetype=mimetype)
	return ConnexionResponse(body=data["data"], mimetype=mimetype)

	def get_file_conf(filename: str, group_id: str = None, type_conf: str = None, raw: bool = False) -> dict:
	def get_file_conf(filename: str, group_id: str = None, type_conf: str = None, raw: bool = False) -> dict \| str:

Merge group files endpoints #21653

Merge group files endpoints #21653

Conversation

GGP1 commented Jan 29, 2024 • edited Loading

Description

Logs/Alerts example

EduLeon12 left a comment

Choose a reason for hiding this comment

GGP1 commented Jan 31, 2024

Selutario left a comment

Choose a reason for hiding this comment

Selutario Jan 31, 2024

Choose a reason for hiding this comment

Selutario Jan 31, 2024

Choose a reason for hiding this comment

Selutario Jan 31, 2024

Choose a reason for hiding this comment

Selutario Jan 31, 2024

Choose a reason for hiding this comment

Selutario Jan 31, 2024

Choose a reason for hiding this comment

GGP1 commented Feb 1, 2024

Selutario left a comment

Choose a reason for hiding this comment

GGP1 commented Jan 29, 2024 •

edited

Loading