Validate Memory leaks in Vulnerability Detector after disk errors fix

[Deblintrake09]

Target version	Related issue	Related PR
4.4.1	wazuh/wazuh#16265	wazuh/wazuh#16478

Description

This issue aims to test the fix to a Memory Leak related to VDT in version 4.4.0-1a The leak was detected that when disk errors are caused by some functions, like wm_vuldet_index_feed, the function would not complete and not free the memory that it is using, causing memory to build up until it crashes the wazuh-modulesd daemon.

The Issue was detected in the Stress tests pipeline, and is not easily reproduced manually.

Proposed checks

• Memory leaks in Vulnerability Detector after disk errors

Steps to reproduce

To reproduce the error, disk or database error must be forced, for this, the cve.db file will be installed to a folder in a custom partiton, so that SQL error: 'database or disk is full' error appears

• Create custom partiton

  $ touch 2gbarea
  $ truncate -s 2Mb 2mbarea
  $ mke2fs -t ext4 -F 2mbarea
  $ mkdir /targetfolder
  $ sudo mount 2gbarea /targetfolder    
  

• Clone repository
• Modify the line that configures the location of cve.db to the configured folder
• Install manager from sources
• Configure VDT with custom feeds for faster downloads.
• Stop wazuh-modulesd
• Execute Valgrind command

  valgrind --track-fds=yes --show-leak-kinds=all --leak-check=full /var/ossec/bin/wazuh-modulesd -ddf
  

[Deblintrake09]

Review data

Tester	PR commit
@Deblintrake09	wazuh/wazuh@ed6ef18

Testing environment

OS	OS version	Deployment	Image/AMI	Notes
Centos	8	Local | Vagrant	qactl/centos8

Tested packages

Manager was built from sources since Jenkins is down

Status

• In progress
• Pending Review
• QA team leader approved @damarisg
• Development team leader approved @MarcelKemp

[Deblintrake09]

Testing results

Testing in 4.4 - Reproduce error

Memory leaks in Vulnerability Detector after disk errors

• Setup manager
• Configure VDT
• Stop wazuh-modulesd
• Remove the database
• Execute wazuh-modules with valgrind

#  valgrind --track-fds=yes --show-leak-kinds=all --leak-check=full /var/ossec/bin/wazuh-modulesd -ddf
==175455== LEAK SUMMARY:
==175455==    definitely lost: 1,008 bytes in 30 blocks
==175455==    indirectly lost: 8,371 bytes in 147 blocks
==175455==      possibly lost: 3,984 bytes in 13 blocks
==175455==    still reachable: 226,145 bytes in 151 blocks
==175455==         suppressed: 0 bytes in 0 blocks
==175455== 
==175455== For lists of detected and suppressed errors, rerun with: -s
==175455== ERROR SUMMARY: 13 errors from 13 contexts (suppressed: 0 from 0)

• Check ossec.log

2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:4915 at wm_vuldet_fetch_feed(): DEBUG: (5403): Fetching feed from '/var/ossec/custom-feeds/nvd/custom_nvd_feed.json'
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:7801 at wm_vuldet_index_json(): DEBUG: (5408): Updating from '/var/ossec/custom-feeds/nvd/custom_nvd_feed.json'
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:4672 at wm_vuldet_index_feed(): DEBUG: (5414): Refreshing 'National Vulnerability Database' databases.
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:3182 at wm_vuldet_insert(): DEBUG: (5415): Inserting vulnerabilities.
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:3228 at wm_vuldet_insert(): DEBUG: (5419): Inserting NVD vulnerabilities section.
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector_nvd.c:1012 at wm_vuldet_insert_nvd_metadata(): ERROR: (5503): SQL error: 'database or disk is full'
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:2788 at wm_vuldet_sql_error(): ERROR: (5503): SQL error: 'database or disk is full'
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:4675 at wm_vuldet_index_feed(): ERROR: (5506): Could not refresh the 'NVD' DB.
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:4683 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp): No such file or directory
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:4686 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp-fitted): No such file or directory
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:4689 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp.bz2): No such file or directory
2023/03/28 19:03:17 wazuh-modulesd:vulnerability-detector[175455] wm_vuln_detector.c:5015 at wm_vuldet_check_feed(): DEBUG: (5521): Failed when updating 'nvd provider' database. Retrying in '300' seconds.

Testing on de branch

UPDATE INSTALL - Memory leaks in Vulnerability Detector after disk errors

• Update manager from 4.4 to dev branch
• Stop wazuh-modulesd
• Remove the database
• Execute wazuh-modules with valgrind

#  valgrind --track-fds=yes --show-leak-kinds=all --leak-check=full /var/ossec/bin/wazuh-modulesd -ddf

==234363== LEAK SUMMARY:
==234363==    definitely lost: 0 bytes in 0 blocks
==234363==    indirectly lost: 0 bytes in 0 blocks
==234363==      possibly lost: 3,984 bytes in 13 blocks
==234363==    still reachable: 226,145 bytes in 151 blocks
==234363==         suppressed: 0 bytes in 0 blocks

• Check ossec.log

2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:5153 at wm_vuldet_check_feed(): INFO: (5400): Starting 'National Vulnerability Database' database update.
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:5045 at wm_vuldet_fetch_feed(): DEBUG: (5484): Cleaning metadata for target 'NVD'
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:5061 at wm_vuldet_fetch_feed(): DEBUG: (5403): Fetching feed from '/var/ossec/custom-feeds/nvd/custom_nvd_feed.json'
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:7950 at wm_vuldet_index_json(): DEBUG: (5408): Updating from '/var/ossec/custom-feeds/nvd/custom_nvd_feed.json'
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:4814 at wm_vuldet_index_feed(): DEBUG: (5414): Refreshing 'National Vulnerability Database' databases.
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:3189 at wm_vuldet_insert(): DEBUG: (5415): Inserting vulnerabilities.
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:3242 at wm_vuldet_insert(): DEBUG: (5419): Inserting NVD vulnerabilities section.
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector_nvd.c:1577 at wm_vuldet_insert_nvd_cve_match(): ERROR: (5503): SQL error: 'database or disk is full'
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:2795 at wm_vuldet_sql_error(): ERROR: (5503): SQL error: 'database or disk is full'
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:4817 at wm_vuldet_index_feed(): ERROR: (5506): Could not refresh the 'NVD' DB.
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:4825 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp): No such file or directory
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:4828 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp-fitted): No such file or directory
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:4831 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp.bz2): No such file or directory
2023/03/28 19:46:59 wazuh-modulesd:vulnerability-detector[234363] wm_vuln_detector.c:5161 at wm_vuldet_check_feed(): DEBUG: (5521): Failed when updating 'nvd provider' database. Retrying in '300' seconds.

CLEAN INSTALL - Memory leaks in Vulnerability Detector after disk errors

• Install manager
• Configure VDT
• Stop wazuh-modulesd
• Remove the database
• Execute wazuh-modules with valgrind

#  valgrind --track-fds=yes --show-leak-kinds=all --leak-check=full /var/ossec/bin/wazuh-modulesd -ddf
==234401== LEAK SUMMARY:
==234401==    definitely lost: 0 bytes in 0 blocks
==234401==    indirectly lost: 0 bytes in 0 blocks
==234401==      possibly lost: 3,952 bytes in 13 blocks
==234401==    still reachable: 226,145 bytes in 151 blocks
==234401==         suppressed: 0 bytes in 0 blocks
==234401== 

• Check ossec.log

2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:7950 at wm_vuldet_index_json(): DEBUG: (5408): Updating from '/var/ossec/custom-feeds/nvd/custom_nvd_feed.json'
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:4814 at wm_vuldet_index_feed(): DEBUG: (5414): Refreshing 'National Vulnerability Database' databases.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:3189 at wm_vuldet_insert(): DEBUG: (5415): Inserting vulnerabilities.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:3242 at wm_vuldet_insert(): DEBUG: (5419): Inserting NVD vulnerabilities section.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector_nvd.c:1012 at wm_vuldet_insert_nvd_metadata(): ERROR: (5503): SQL error: 'database or disk is full'
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:2795 at wm_vuldet_sql_error(): ERROR: (5503): SQL error: 'database or disk is full'
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:4817 at wm_vuldet_index_feed(): ERROR: (5506): Could not refresh the 'NVD' DB.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:4825 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp): No such file or directory
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:4828 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp-fitted): No such file or directory
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:4831 at wm_vuldet_index_feed(): DEBUG: remove(tmp/vuln-temp.bz2): No such file or directory
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:5161 at wm_vuldet_check_feed(): DEBUG: (5521): Failed when updating 'nvd provider' database. Retrying in '300' seconds.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:5155 at wm_vuldet_check_feed(): DEBUG: (5400): Starting 'Wazuh CPE dictionary' database update.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:7196 at wm_vuldet_fetch_wazuh_cpe(): DEBUG: (5407): The feed 'Wazuh CPE dictionary' is outdated. Fetching the last version.
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:7969 at wm_vuldet_index_json(): DEBUG: (5408): Updating from 'tmp/vuln-temp-fitted'
2023/03/28 19:51:55 wazuh-modulesd:vulnerability-detector[234401] wm_vuln_detector.c:7242 at wm_vuldet_json_wcpe_parser(): DEBUG: (5407): The feed 'Wazuh CPE dictionary' is outdated. Fetching the last version.

[Deblintrake09]

Conclusion 🟢

Testing shows the Memory Leaks have been fixed. At the current time, it is not possible to run the Stress tests to replicate verify in the pipeline, but testing consistently shows no memleaks are present when the disk is full.