Increase in differents metrics by Wazuh-modulesd

[MARCOSD4]

Wazuh version	Component	Install type	Install method	Platform
4.8.0-beta5	Wazuh manager	Manager	Packages	CentOS

Description

It has been detected a significant increase of Wazuh-modulesd in different metrics for all footprint tests of 4.8.0-beta5

Important

This uptick isn't directly tied to alterations in the vulnerability detection module, as it's disabled in the majority of the tests affected.

Related issues:

• Highly Increase in Wazuh-Modulesd Values for v4.8.0-beta4 #22566

Evidences

We've opted for the SCA footprint test, yet this enhancement can be observed across all issues related to footprint tests.

Plots compared to #22296

Memory usage

Release/Stage	RSS	Issue
4.8.0-Beta5		#22552
4.7.3-RC2		#22296

Release/Stage	PSS	Issue
4.8.0-Beta5		#22552
4.7.3-RC2		#22296

File descriptors

Release/Stage	FD	Issue
4.8.0-Beta5		#22788
4.7.3-RC2		#22296

[vikman90]

Let's compare 4.7.3, 4.8.0-beta4 and 4.8.0-beta5:

Memory usage

4.7.3	4.8.0-beta4	4.8.0-beta5

The memory usage has decreased from ~80MiB (4.8.0-beta4) to ~75 MiB (4.8.0-beta5). However, it's still far from the ~50 MiB in 4.7.3.

File descriptors

4.7.3	4.8.0-beta4	4.8.0-beta5

[santipadilla]

Increase of modulesd also noticed in Release 4.8.0 - Beta 5 - Footprint Metrics - LOGCOLLECTOR,SYSCOLLECTOR

Release/Stage	CentOS Disk_read	Issue
4.8.0-Beta5		#22797
4.7.3-RC2		#22313

Release/Stage	CentOS Disk	Issue
4.8.0-Beta5		#22797
4.7.3-RC2		#22313

Release/Stage	Ubuntu Disk_read	Issue
4.8.0-Beta5		#22797
4.7.3-RC2		#22313

[Dwordcito]

Probably related to memory fragmentation.

@sebasfalcone could you upload heaptrack memory usage on standby?

[sebasfalcone]

After processing the whole feed, modulesd consumption settles at 118MB

Most of the allocations are between 17 and 32 Bytes, followed by 15 and 128 Bytes

[Dwordcito]

At the initial point the test differ from the 4.7.3, because Vulnerability detection is enabled by default.

2024/04/05 17:25:46 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/04/05 17:25:46 wazuh-modulesd:vulnerability-scanner: INFO: Starting database file decompression.
2024/04/05 17:25:46 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities', retrying until the connection is successful.
2024/04/05 17:26:53 wazuh-modulesd:vulnerability-scanner: INFO: Database decompression finished.
2024/04/05 17:26:54 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started
2024/04/05 17:26:55 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process
2024/04/05 17:26:55 wazuh-modulesd:vulnerability-scanner: INFO: Processing file: queue/vd_updater/tmp/contents/431557-api_file.json
2024/04/05 17:27:27 wazuh-modulesd:vulnerability-scanner: INFO: Processing file: queue/vd_updater/tmp/contents/432557-api_file.json
2024/04/05 17:28:04 wazuh-modulesd:vulnerability-scanner: INFO: Processing file: queue/vd_updater/tmp/contents/433557-api_file.json
2024/04/05 17:28:32 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module.
2024/04/05 17:28:32 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed
2024/04/05 17:28:54 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/04/05 17:28:54 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module is disabled

Memory usage.
As I mentioned previously, VD is enabled, and when this module is active, some write and read cache's are used.
For example to avoid multiple read operations, rocksdb (the database engine for VD) have an LRU cache, to read the information directly from the memory space. This have the benefit of less IO, but have the contra in the memory usage.

The read memory is limited ? YES, 16MBs

For the write operations rocksdb have another cache (Write cache buffer), and this works similar to the read cache.

The write memory is limited ? YES 64, MBs

File Descriptors
Also the fds are impacted by VD, Rocksdb works with LSM and the information is partitioned in multiple files, and to read this information we need open multiple files.

The rocksdb open files have some limit? YES, 256.

Closed, this consumption is part of the design and strategy to get more scanner performance.