Increase in Disk_Read by Wazuh-syscheckd

[MARCOSD4]

Wazuh version	Component	Install type	Install method	Platform
4.8.0-beta4	Wazuh Manager	Manager	Packages	CentOS

During testing in Release 4.8.0 - Beta 4 - Footprint Metrics - MACOS-SOLARIS (2.5d) it has been detected an increase in Disk_Read by Wazuh-syscheckd. This behavior has not been detected in previous versions, but it has been detected in 4.8.0 Beta 3, although it had not been reported since footprint was not analyzed at that stage. An increase in Wazuh-modulesd can also be seen but this is already reported in this issue #22566

The plots are compared to #22314.

4.8.0	4.7.3

4.8.0 - Beta 3 plot:

[nbertoldo]

Issue update

I have started by analysing the test reports, where the following can be seen:

4.7.3

Both the graph and the CSV file show that the disk read value remains at zero from the start until 3/2/2024 11:27:28. Then it starts to increase to a value of 28*10e6.

Graph

CSV file

4.8.0

In both beta3 and beta4 the initial value remains almost constant from the beginning to the end of the test. In both cases this value is much higher than in 4.7.3.

• 525*10e6 for beta3
• 265*10e6 for beta4.

Beta 3

Graph

CSV file

Beta 4

Graph

CSV file

Something to note is that in all cases, the syscheck curve follows the same pattern as modulesd, so they are probably related. I will continue with the further analysis.

[nbertoldo]

As reported in #22565, Vulnerability Detector is enabled when the test starts (default configuration) and this affects modulesd disk read values.
Although I can't find the relationship, based on the graphs of various tests I've reviewed, this seems to be affecting syscheck as well.

Since the disk read value is cumulative, we can calculate the difference between the initial and final values :

Module	Version	Initial value	Final value	Difference
syscheck	4.7.3	0	27963392	27963392
	4.8.0 - Beta3	525189120	527110144	1921024
	4.8.0 - Beta4	265097216	267681792	2584576
modulesd	4.7.3	0	12783616	12783616
	4.8.0 - Beta3	106254336	127107072	20852736
	4.8.0 - Beta4	146331136	167421440	21090304

And we can see the following:

• syscheck: the value decreases considerably for 4.8.0, being less than 10% of the value in 4.7.3.
• modulesd: the value increases by 65% in 4.8.0.

Therefore I propose to re-launch the test with the appropriate configuration so that the Vulnerability detector does not affect the measurement.

[nbertoldo]

In order to clarify the previous comment, we can see that at the beginning of the 4.8.0 test Vulnerability Detector is enabled because it is taking a default Wazuh configuration from a template.
We can see in the log that the module starts processing:

2024/03/12 17:41:26 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/03/12 17:41:26 wazuh-modulesd:vulnerability-scanner: INFO: Starting database file decompression.
2024/03/12 17:42:24 wazuh-modulesd:vulnerability-scanner: INFO: Database decompression finished.
2024/03/12 17:42:25 wazuh-modulesd:content-updater: INFO: Starting scheduled action for 'vulnerability_feed_manager'
2024/03/12 17:42:25 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:42:25 wazuh-modulesd:vulnerability-scanner: INFO: Processing message
2024/03/12 17:42:25 wazuh-modulesd:vulnerability-scanner: INFO: Processing file: queue/vd_updater/tmp/contents/245855-api_file.json
2024/03/12 17:42:25 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:42:25 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started
2024/03/12 17:42:36 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:42:36 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:42:36 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:19 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:19 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:19 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:31 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:31 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:31 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:33 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:33 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:33 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:36 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:36 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:36 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:36 wazuh-modulesd:vulnerability-scanner: INFO: Processing file: queue/vd_updater/tmp/contents/246579-api_file.json
2024/03/12 17:43:37 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:37 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:37 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:39 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:39 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:39 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:42 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:42 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:42 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:57 wazuh-modulesd:content-updater: INFO: Starting on-demand action for 'vulnerability_feed_manager'
2024/03/12 17:43:57 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' started
2024/03/12 17:43:57 wazuh-modulesd:content-updater: INFO: Action for 'vulnerability_feed_manager' finished
2024/03/12 17:43:57 wazuh-modulesd:vulnerability-scanner: INFO: Message processed

The configuration is replaced and the manager is restarted:

2024/03/12 17:53:24 wazuh-modulesd:syscollector: INFO: Stop received for Syscollector.
2024/03/12 17:53:24 wazuh-modulesd:syscollector: INFO: Module finished.
2024/03/12 17:53:24 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module.
2024/03/12 17:53:28 indexer-connector: WARNING: Error initializing IndexerConnector for index 'wazuh-states-vulnerabilities': No available server. Retrying in 60 seconds. Maximum wait time: 60 seconds.
2024/03/12 17:53:30 wazuh-modulesd:router: INFO: Stopping router module.
2024/03/12 17:53:30 wazuh-modulesd:content_manager: INFO: Stopping content_manager module.
2024/03/12 17:53:30 wazuh-modulesd:content-updater: INFO: Scheduler stopped for 'vulnerability_feed_manager'
2024/03/12 17:53:30 wazuh-monitord: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:30 wazuh-logcollector: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:30 wazuh-remoted: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:30 wazuh-syscheckd: INFO: (1756): Shutdown received. Releasing resources.
2024/03/12 17:53:30 wazuh-syscheckd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:31 wazuh-analysisd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:31 wazuh-execd: INFO: (1314): Shutdown received. Deleting responses.
2024/03/12 17:53:31 wazuh-execd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:31 wazuh-db: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:32 wazuh-authd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/03/12 17:53:32 wazuh-authd: INFO: Exiting...

And with this new configuration we see that Vulnerability Detector is disabled:

2024/03/12 17:53:43 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/03/12 17:53:43 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module is disabled

Unlike 4.7.3 where it is always disabled. This affects the measurement and makes the graphs not comparable. Therefore, Vulnerability Detector should be disabled before the manager starts.

[vikman90]

I think that comparing the read_ops value would be more accurate, as read_bytes implies the storage layer:

proc	psutil	Meaning
rchar	❌	Bytes requested to read
wchar	❌	Bytes requested to write
syscr	read_ops	Calls to read
syscw	write_ops	Calls to write
read_bytes	read_bytes	Bytes effectively read from the storage layer
write_bytes	write_bytes	Bytes effectively written to the storage layer

Please consider comparing read_ops:

4.7.3	4.8.0

Conclusion

I don't think this represent an issue in the product.