Create consolidated vulnerability state

[havidarou]

Description

As part of the vulnerability detector refactor, the global vulnerability state will be consolidated by the API. If there's a Wazuh manager cluster running in the environment this will only be performed by the master node.

Wazuh manager will synchronize vulnerability state information to a specific Wazuh indexer index. In a Wazuh manager cluster scenario, the API will use the Wazuh agent connection information and vulnerability state indices to build the global vulnerability state.

Concepts:

• Global vulnerability state. The global vulnerability state only contains the most up-to-date vulnerability information that belongs to agents registered in a Wazuh environment (considering active agents' information).

Functional requirements

• Users will be able to search global vulnerability state information in a single Wazuh indexer index. Only if the vulnerability detector module is enabled.
• Users will use the vulnerability detector placeholder setting to disable the global vulnerability state synchronization.
• Users will be able to change the global state synchronization interval.

Non-functional requirements

• Performance. API queries to both global.db and Wazuh indexer should be as performant as possible.
• Performance. Capacity testing with 50K agents. 25 managers. 2 vd state changes per agent per second. 500 initial vd state elements per agent.
• None of the implemented features will delay the Wazuh manager initialization process.

Implementation restrictions

• Only execution errors will be logged. A significant delay will be considered an error (2m).
• The Wazuh API will spawn a new child process in charge of the synchronization.
• Wazuh API should be modified as little as possible.
• To communicate with the Wazuh indexer API, the pip package opensearch-dsl will be used.
• The synchronization interval minimum value should guarantee global.db and Wazuh indexer performance (10s).
• The synchronization interval maximum value should guarantee that the global vulnerability state is usable (10m).
• Wazuh indexer connection information will be fetched from the wazuh-connector configuration block in the ossec.conf file.
• In a Wazuh manager cluster scenario, this synchronization will only take place in the master node.
• The reindex API should be considered (all fields that may change should be hashed and stored in the document ID).
• To clean deleted agents in the global vulnerability state index, a different, hardcoded interval (24h) will be used due to performance issues with global.db.
• The global vulnerability index creation will be managed by the API. The name of the cluster will be wazuh-states-vulnerabilities-manager_cluster_name. If there's no cluster enabled we should use the default name as manager_cluster_name (wazuh?).

Plan

• Spike. (2 days) @wazuh/devel-framework @wazuh/devel-indexer
  • Spike - Synchronize VD state from all nodes index to a common index #23068
• MVP design and implementation. (3 days) @wazuh/devel-framework
  • Include opensearch-py in the embedded Python interpreter #23097
• Feature complete implementation. (2 days) @wazuh/devel-framework
• Integration (1 day) @wazuh/devel-framework @wazuh/devel-core2
• Capacity testing (? days) @wazuh/devel-qa

RTM

Type	Responsible(s)	Description	Plan Part
Functional	@wazuh/devel-framework	Enable search of global vulnerability state in a single indexer index only when the vulnerability detector is enabled.	MVP design and implementation
Functional	@wazuh/devel-framework	Allow users to disable global vulnerability state synchronization using the vulnerability detector placeholder setting.	MVP design and implementation
Functional	@wazuh/devel-framework	Provide capability to change the synchronization interval for global vulnerability state.	MVP design and implementation
Non-functional	@wazuh/devel-framework	Ensure that API queries to both global.db and Wazuh indexer are optimized for performance.	Integration
Non-functional	@wazuh/devel-framework	Ensure that no feature implementation delays the Wazuh manager initialization process.	Throughout all phases
Implementation	@wazuh/devel-framework	Log only execution errors; treat significant delays as errors.	Throughout all phases
Implementation	@wazuh/devel-framework	Create minimal changes to existing Wazuh API for introducing new synchronization features.	Throughout all phases
Implementation	@wazuh/devel-framework	Utilize opensearch-dsl for communicating with the Wazuh indexer API.	Spike, MVP design
Implementation	@wazuh/devel-framework	Set synchronization interval minimum at 10s and maximum at 10m to balance performance and usability.	MVP design and implementation
Implementation	@wazuh/devel-framework	Fetch Wazuh indexer connection info from the wazuh-connector configuration block in ossec.conf.	Spike, MVP design
Implementation	@wazuh/devel-framework	Implement reindexing strategy using the reindex API to handle changes in the document ID.	MVP design and implementation
Implementation	@wazuh/devel-framework	Manage cleanup of deleted agents in the global vulnerability index using a hardcoded interval of 24h.	MVP design and implementation
Non-functional	@wazuh/devel-qa	Capacity testing with 50K agents. 25 managers. 2 vd state changes per agent per second. 500 initial vd state elements per agent.	Capacity testing

Approved by

• DRI name: @davidjiglesias @gdiazlo @Selutario @havidarou
• Objective: Vulnerability detector refactor.

[nico-stefani]

The opensearch-dsl library was deprecated. The recommended way is to use opensearch-py instead.

[davidjiglesias]

Due to the algorithmic nature of the new vulnerability detector module, the solution at hand can lead to an unrecoverable scenario.

We will favor the solution proposed in #22867.