Migrate configuration to ossec.conf #22782

nico-stefani · 2024-04-05T19:32:17Z

Related issue
#20939

Description

This PR closes #20939. Migrate the HAProxy Helper to the ossec.conf.

The full set of options added are:

<haproxy_helper>
        <haproxy_disabled>no</haproxy_disabled>
        <haproxy_address>wazuh-proxy</haproxy_address>
        <haproxy_user>haproxy</haproxy_user>
        <haproxy_password>haproxy</haproxy_password>
         <!-- Optional parameters -->
        <haproxy_port>5555</haproxy_port>
        <haproxy_protocol>http</haproxy_protocol>
        <haproxy_backend>wazuh_cluster</haproxy_backend>
        <frequency>60</frequency>
        <agent_chunk_size>10</agent_chunk_size>
        <agent_reconnection_time>10</agent_reconnection_time>
        <agent_reconnection_stability_time>60</agent_reconnection_stability_time>
        <imbalance_tolerance>0.1</imbalance_tolerance>
        <remove_disconnected_node_after>1</remove_disconnected_node_after>
    </haproxy_helper>

Logs/Alerts example

The Helper starts with mandatory parameters

<cluster>
    <name>wazuh</name>
    <node_name>master-node</node_name>
    <node_type>master</node_type>
    <key>9d273b53510fef702b54a92e9cffc82e</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>wazuh-master</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
    <haproxy_helper>
        <haproxy_disabled>no</haproxy_disabled>
        <haproxy_address>wazuh-proxy</haproxy_address>
        <haproxy_user>haproxy</haproxy_user>
        <haproxy_password>haproxy</haproxy_password>
    </haproxy_helper>
  </cluster>

Logs

2024/04/05 19:23:06 DEBUG: [Cluster] [Main] Removing '/var/ossec/queue/cluster/'.
2024/04/05 19:23:06 DEBUG: [Cluster] [Main] Removed '/var/ossec/queue/cluster/'.
2024/04/05 19:23:06 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock
2024/04/05 19:23:06 DEBUG: [Local Server] [Keep alive] Calculating.
2024/04/05 19:23:06 DEBUG: [Local Server] [Keep alive] Calculated.
2024/04/05 19:23:06 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516)
2024/04/05 19:23:06 DEBUG: [Master] [Keep alive] Calculating.
2024/04/05 19:23:06 DEBUG: [Master] [Keep alive] Calculated.
2024/04/05 19:23:06 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:23:06 INFO: [Master] [Local agent-groups] Sleeping 30s before starting the agent-groups task, waiting for the workers connection.
2024/04/05 19:23:06 INFO: [HAPHelper] [Main] Proxy was initialized
2024/04/05 19:23:06 INFO: [HAPHelper] [Main] Ensuring only exists one HAProxy process. Sleeping 12s before start...
2024/04/05 19:23:06 INFO: [Master] [Local integrity] Finished in 0.090s. Calculated metadata of 34 files.
2024/04/05 19:23:14 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:23:14 INFO: [Master] [Local integrity] Finished in 0.005s. Calculated metadata of 34 files.
2024/04/05 19:23:18 DEBUG2: [HAPHelper] [Proxy] Obtained proxy backends
2024/04/05 19:23:18 DEBUG2: [HAPHelper] [Proxy] Obtained proxy frontends
2024/04/05 19:23:18 INFO: [HAPHelper] [Main] Starting HAProxy Helper
2024/04/05 19:23:18 DEBUG2: [HAPHelper] [Proxy] Obtained proxy servers
2024/04/05 19:23:18 DEBUG: [Local 24024] [Main] Connection received in local server.
2024/04/05 19:23:18 DEBUG: [HAPHelper] [D API] Receiving parameters {}
2024/04/05 19:23:18 DEBUG: [Local Client] [D API] Starting to execute request locally
2024/04/05 19:23:18 DEBUG: [Local 24024] [Main] Disconnected 24024.
2024/04/05 19:23:18 DEBUG: [Local 412462] [Main] Connection received in local server.
2024/04/05 19:23:18 DEBUG: [Local 412462] [Main] Disconnected 412462.
2024/04/05 19:23:18 DEBUG: [Local Client] [D API] Finished executing request locally
2024/04/05 19:23:18 DEBUG: [Local Client] [D API] Time calculating request result: 0.005s
2024/04/05 19:23:18 DEBUG2: [HAPHelper] [Proxy] Obtained proxy servers
2024/04/05 19:23:18 INFO: [HAPHelper] [Main] Load balancer backend is up to date
2024/04/05 19:23:18 DEBUG2: [HAPHelper] [Proxy] Obtained server 'master-node' stats
2024/04/05 19:23:18 DEBUG: [HAPHelper] [Main] Checking for agent balance. Current connections distribution: {'master-node': 2}
2024/04/05 19:23:18 DEBUG: [HAPHelper] [Main] Current balance is under tolerance
2024/04/05 19:23:18 DEBUG2: [HAPHelper] [Proxy] Obtained server 'master-node' stats
2024/04/05 19:23:18 DEBUG: [HAPHelper] [Main] Current backend stats: {'master-node': 2}
2024/04/05 19:23:18 INFO: [HAPHelper] [Main] Load balancer backend is balanced
2024/04/05 19:23:18 DEBUG: [HAPHelper] [Main] Sleeping 60s...

The Helper will not start when is disabled

<cluster>
    <name>wazuh</name>
    <node_name>master-node</node_name>
    <node_type>master</node_type>
    <key>9d273b53510fef702b54a92e9cffc82e</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>wazuh-master</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
    <haproxy_helper>
        <haproxy_disabled>yes</haproxy_disabled>
        <haproxy_address>wazuh-proxy</haproxy_address>
        <haproxy_user>haproxy</haproxy_user>
        <haproxy_password>haproxy</haproxy_password>
    </haproxy_helper>
  </cluster>

Logs

2024/04/05 19:25:56 DEBUG: [Cluster] [Main] Removing '/var/ossec/queue/cluster/'.
2024/04/05 19:25:56 DEBUG: [Cluster] [Main] Removed '/var/ossec/queue/cluster/'.
2024/04/05 19:25:57 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock
2024/04/05 19:25:57 DEBUG: [Local Server] [Keep alive] Calculating.
2024/04/05 19:25:57 DEBUG: [Local Server] [Keep alive] Calculated.
2024/04/05 19:25:57 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516)
2024/04/05 19:25:57 DEBUG: [Master] [Keep alive] Calculating.
2024/04/05 19:25:57 DEBUG: [Master] [Keep alive] Calculated.
2024/04/05 19:25:57 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:25:57 INFO: [Master] [Local agent-groups] Sleeping 30s before starting the agent-groups task, waiting for the workers connection.
2024/04/05 19:25:57 INFO: [Master] [Local integrity] Finished in 0.107s. Calculated metadata of 34 files.
2024/04/05 19:26:05 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:26:05 INFO: [Master] [Local integrity] Finished in 0.004s. Calculated metadata of 34 files.
2024/04/05 19:26:13 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:26:13 INFO: [Master] [Local integrity] Finished in 0.004s. Calculated metadata of 34 files.

The Helper will not start when one mandatory parameter is missing

<cluster>
    <name>wazuh</name>
    <node_name>master-node</node_name>
    <node_type>master</node_type>
    <key>9d273b53510fef702b54a92e9cffc82e</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>wazuh-master</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
    <haproxy_helper>
        <haproxy_disabled>no</haproxy_disabled>
        <haproxy_address>wazuh-proxy</haproxy_address>
        <haproxy_user>haproxy</haproxy_user>
    </haproxy_helper>
  </cluster>

Logs

2024/04/05 19:28:07 DEBUG: [Cluster] [Main] Removing '/var/ossec/queue/cluster/'.
2024/04/05 19:28:07 DEBUG: [Cluster] [Main] Removed '/var/ossec/queue/cluster/'.
2024/04/05 19:28:08 ERROR: [HAPHelper] [Main] Missing configuration 'haproxy_password'. The helper cannot start.
Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/core/cluster/hap_helper/hap_helper.py", line 496, in start
    password=helper_config['haproxy_password'],
KeyError: 'haproxy_password'
2024/04/05 19:28:08 INFO: [HAPHelper] [Main] Process ended
2024/04/05 19:28:08 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock
2024/04/05 19:28:08 DEBUG: [Local Server] [Keep alive] Calculating.
2024/04/05 19:28:08 DEBUG: [Local Server] [Keep alive] Calculated.
2024/04/05 19:28:08 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516)
2024/04/05 19:28:08 DEBUG: [Master] [Keep alive] Calculating.
2024/04/05 19:28:08 DEBUG: [Master] [Keep alive] Calculated.
2024/04/05 19:28:08 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:28:08 INFO: [Master] [Local agent-groups] Sleeping 30s before starting the agent-groups task, waiting for the workers connection.
2024/04/05 19:28:08 INFO: [Master] [Local integrity] Finished in 0.102s. Calculated metadata of 34 files.
2024/04/05 19:28:16 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:28:16 INFO: [Master] [Local integrity] Finished in 0.004s. Calculated metadata of 34 files.
2024/04/05 19:28:24 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:28:24 INFO: [Master] [Local integrity] Finished in 0.004s. Calculated metadata of 34 files.
2024/04/05 19:28:32 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:28:32 INFO: [Master] [Local integrity] Finished in 0.004s. Calculated metadata of 34 files.

The Helper starts with all parameters

<cluster>
    <name>wazuh</name>
    <node_name>master-node</node_name>
    <node_type>master</node_type>
    <key>9d273b53510fef702b54a92e9cffc82e</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>wazuh-master</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
    <haproxy_helper>
        <haproxy_disabled>no</haproxy_disabled>
        <haproxy_address>wazuh-proxy</haproxy_address>
        <haproxy_user>haproxy</haproxy_user>
        <haproxy_password>haproxy</haproxy_password>
        <haproxy_port>5555</haproxy_port>
        <haproxy_protocol>http</haproxy_protocol>
        <haproxy_backend>wazuh_cluster</haproxy_backend>
        <frequency>60</frequency>
        <agent_chunk_size>10</agent_chunk_size>
        <agent_reconnection_time>10</agent_reconnection_time>
        <agent_reconnection_stability_time>60</agent_reconnection_stability_time>
        <imbalance_tolerance>0.1</imbalance_tolerance>
        <remove_disconnected_node_after>1</remove_disconnected_node_after>
    </haproxy_helper>
  </cluster>

Logs

2024/04/05 19:30:55 DEBUG: [Cluster] [Main] Removing '/var/ossec/queue/cluster/'.
2024/04/05 19:30:55 DEBUG: [Cluster] [Main] Removed '/var/ossec/queue/cluster/'.
2024/04/05 19:30:56 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock
2024/04/05 19:30:56 DEBUG: [Local Server] [Keep alive] Calculating.
2024/04/05 19:30:56 DEBUG: [Local Server] [Keep alive] Calculated.
2024/04/05 19:30:56 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516)
2024/04/05 19:30:56 DEBUG: [Master] [Keep alive] Calculating.
2024/04/05 19:30:56 DEBUG: [Master] [Keep alive] Calculated.
2024/04/05 19:30:56 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:30:56 INFO: [Master] [Local agent-groups] Sleeping 30s before starting the agent-groups task, waiting for the workers connection.
2024/04/05 19:30:56 INFO: [HAPHelper] [Main] Proxy was initialized
2024/04/05 19:30:56 INFO: [HAPHelper] [Main] Ensuring only exists one HAProxy process. Sleeping 12s before start...
2024/04/05 19:30:56 INFO: [Master] [Local integrity] Finished in 0.090s. Calculated metadata of 34 files.
2024/04/05 19:31:04 INFO: [Master] [Local integrity] Starting.
2024/04/05 19:31:04 INFO: [Master] [Local integrity] Finished in 0.004s. Calculated metadata of 34 files.
2024/04/05 19:31:08 DEBUG2: [HAPHelper] [Proxy] Obtained proxy backends
2024/04/05 19:31:08 DEBUG2: [HAPHelper] [Proxy] Obtained proxy frontends
2024/04/05 19:31:08 INFO: [HAPHelper] [Main] Starting HAProxy Helper
2024/04/05 19:31:08 DEBUG2: [HAPHelper] [Proxy] Obtained proxy servers
2024/04/05 19:31:08 DEBUG: [Local 919910] [Main] Connection received in local server.
2024/04/05 19:31:08 DEBUG: [HAPHelper] [D API] Receiving parameters {}
2024/04/05 19:31:08 DEBUG: [Local Client] [D API] Starting to execute request locally
2024/04/05 19:31:08 DEBUG: [Local 919910] [Main] Disconnected 919910.
2024/04/05 19:31:08 DEBUG: [Local 194254] [Main] Connection received in local server.
2024/04/05 19:31:08 DEBUG: [Local 194254] [Main] Disconnected 194254.
2024/04/05 19:31:08 DEBUG: [Local Client] [D API] Finished executing request locally
2024/04/05 19:31:08 DEBUG: [Local Client] [D API] Time calculating request result: 0.005s
2024/04/05 19:31:08 DEBUG2: [HAPHelper] [Proxy] Obtained proxy servers
2024/04/05 19:31:08 INFO: [HAPHelper] [Main] Load balancer backend is up to date
2024/04/05 19:31:08 DEBUG2: [HAPHelper] [Proxy] Obtained server 'master-node' stats
2024/04/05 19:31:08 DEBUG: [HAPHelper] [Main] Checking for agent balance. Current connections distribution: {'master-node': 2}
2024/04/05 19:31:08 DEBUG: [HAPHelper] [Main] Current balance is under tolerance
2024/04/05 19:31:08 DEBUG2: [HAPHelper] [Proxy] Obtained server 'master-node' stats
2024/04/05 19:31:08 DEBUG: [HAPHelper] [Main] Current backend stats: {'master-node': 2}
2024/04/05 19:31:08 INFO: [HAPHelper] [Main] Load balancer backend is balanced
2024/04/05 19:31:08 DEBUG: [HAPHelper] [Main] Sleeping 60s...
2024/04/05 19:31:12 INFO: [Master] [Local integrity] Starting.

framework/wazuh/core/cluster/hap_helper/hap_helper.py

framework/scripts/tests/test_wazuh_clusterd.py

framework/wazuh/core/cluster/utils.py

src/config/cluster-config.c

framework/wazuh/core/cluster/utils.py

GGP1

LGTM

Selutario

Great job @nico-stefani. I requested only a few things. Also, although this is not a common scenario, if all active managers are excluded, this unexpected error is raised:

2024/04/18 12:12:16 CRITICAL: [Local Client] [Main] Unexpected exception: division by zero
Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/core/cluster/hap_helper/hap_helper.py", line 547, in start
    await helper.manage_wazuh_cluster_nodes()
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/core/cluster/hap_helper/hap_helper.py", line 411, in manage_wazuh_cluster_nodes
    await self.set_hard_stop_after(wait_connection_retry=False, reconnect_agents=False)
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/core/cluster/hap_helper/hap_helper.py", line 459, in set_hard_stop_after
    await self.proxy.set_hard_stop_after_value(
  File "/var/ossec/framework/python/lib/python3.10/site-packages/wazuh/core/cluster/hap_helper/proxy.py", line 577, in set_hard_stop_after_value
    hard_stop_after = (active_agents / (n_managers * chunk_size)) * n_managers * agent_reconnection_time + (
ZeroDivisionError: division by zero
2024/04/18 12:12:16 INFO: [Local Client] [Main] Process ended

Even if all nodes are excluded, it could change in the future after connecting a new one (for example, if only master is excluded and workers are installed and connected later). We should handle this better.

framework/wazuh/core/cluster/utils.py

framework/wazuh/core/cluster/hap_helper/hap_helper.py

framework/wazuh/core/cluster/cluster.py

Selutario

LGTM, good job!

nico-stefani added 3 commits April 4, 2024 16:35

Add validations for HAProxy helper parameters

0c702a7

Get configuration from ossec.conf

16a07bf

Add field for haproxy protocol

5da12ff

nico-stefani self-assigned this Apr 5, 2024

nico-stefani linked an issue Apr 5, 2024 that may be closed by this pull request

Convert HAProxy helper configuration to XML and integrate it in ossec.conf #20939

Closed

5 tasks

nico-stefani commented Apr 5, 2024

View reviewed changes

framework/wazuh/core/cluster/hap_helper/hap_helper.py Outdated Show resolved Hide resolved

Clean up old configuration utils

ce59e37

nico-stefani force-pushed the 20939-implement-xml-configuration branch from d96fcb7 to ce59e37 Compare April 5, 2024 19:38

Fix UT

2d4416d

GGP1 requested changes Apr 9, 2024

View reviewed changes

Suggestions from CR

df3dc29

nico-stefani force-pushed the 20939-implement-xml-configuration branch from c0843fc to df3dc29 Compare April 10, 2024 18:51

nico-stefani requested a review from GGP1 April 10, 2024 18:53

GGP1 approved these changes Apr 11, 2024

View reviewed changes

Selutario requested changes Apr 18, 2024

View reviewed changes

nico-stefani added 3 commits April 19, 2024 13:58

Fill empty values with defaults

691aa41

Set default number of managers in case all are excluded

9b7acf1

Improve configuration validation

16afec2

nico-stefani force-pushed the 20939-implement-xml-configuration branch from ce66b3f to 16afec2 Compare April 22, 2024 14:50

nico-stefani requested a review from Selutario April 22, 2024 14:56

Update default values

d06682a

Selutario reviewed Apr 23, 2024

View reviewed changes

framework/wazuh/core/cluster/cluster.py Outdated Show resolved Hide resolved

Fix constant name

d94e420

nico-stefani requested a review from Selutario April 23, 2024 16:18

Selutario approved these changes Apr 23, 2024

View reviewed changes

Selutario merged commit fb18876 into epic-20887-migrate-haproxy-helper Apr 23, 2024
16 of 18 checks passed

Selutario deleted the 20939-implement-xml-configuration branch April 23, 2024 16:23

nico-stefani mentioned this pull request Apr 30, 2024

Update the dashboard panel where HAProxy helper configuration is shown wazuh/wazuh-dashboard#175

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate configuration to ossec.conf #22782

Migrate configuration to ossec.conf #22782

nico-stefani commented Apr 5, 2024

GGP1 left a comment

Selutario left a comment

Selutario left a comment

Migrate configuration to ossec.conf #22782

Migrate configuration to ossec.conf #22782

Conversation

nico-stefani commented Apr 5, 2024

Description

Logs/Alerts example

GGP1 left a comment

Choose a reason for hiding this comment

Selutario left a comment

Choose a reason for hiding this comment

Selutario left a comment

Choose a reason for hiding this comment