-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes to limit the memory usage in write/read operations with rocksdb. #22693
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Code 🟢
- Some minor details regarding documentation and atributes
- Test ⚪
- Pending
src/wazuh_modules/vulnerability_scanner/src/databaseFeedManager/databaseFeedManager.hpp
Outdated
Show resolved
Hide resolved
src/wazuh_modules/vulnerability_scanner/src/databaseFeedManager/databaseFeedManager.hpp
Outdated
Show resolved
Hide resolved
src/wazuh_modules/vulnerability_scanner/src/databaseFeedManager/databaseFeedManager.hpp
Outdated
Show resolved
Hide resolved
src/wazuh_modules/vulnerability_scanner/src/databaseFeedManager/databaseFeedManager.hpp
Outdated
Show resolved
Hide resolved
src/wazuh_modules/vulnerability_scanner/src/databaseFeedManager/databaseFeedManager.hpp
Outdated
Show resolved
Hide resolved
src/wazuh_modules/vulnerability_scanner/tests/unit/databaseFeedManager_test.cpp
Outdated
Show resolved
Hide resolved
- Fix behavior if the feed have less or invalid data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scan with content (this branch)
cat ossec.log | grep "publish key" | wc -l
50
Scan with content (4.8.0)
cat ossec.log | grep "publish key" | wc -l
50
Scan without content (this branch)
Sync events and feed processing is done in parallel, this leads to errors at the scan
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:423 at handleRequest(): DEBUG: Vulnerability scan for package 'libreoffice-help-de' on Agent '000' has completed.
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] scanOrchestrator.hpp:167 at run(): DEBUG: Event type: 1 processed
2024/04/04 16:15:20 wazuh-modulesd:syscollector[239792] logging_helper.c:40 at taggedLogFunction(): DEBUG: Sync sent: {"component":"syscollector_packages","data":{"attributes":{"architecture":"amd64","checksum":"a15cae8254677e9418fb013e59dfd542ef78c146","description":"simple thesaurus library","format":"deb","groups":"libs","install_time":" ","item_id":"60fbb3da57a48c87e2f95a780abd91ab7be32ba4","location":" ","multiarch":"same","name":"libmythes-1.2-0","priority":"optional","scan_time":"2024/04/04 14:15:20","size":32,"source":"mythes","vendor":"Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>","version":"2:1.2.4-4build1"},"index":"60fbb3da57a48c87e2f95a780abd91ab7be32ba4","timestamp":""},"type":"state"}
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:397 at handleRequest(): DEBUG: Initiating a vulnerability scan for package 'libmythes-1.2-0' (deb) (Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>) with CVE Numbering Authorities (CNA) 'nvd' on Agent 'Thinkpad-Sebas' (ID: '000', Version: 'v4.8.0').
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:415 at handleRequest(): WARNING: Failed to scan package: 'libmythes-1.2-0', CVE Numbering Authorities (CNA): 'nvd', Error: 'Couldn't find column family: 'nvd''
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:423 at handleRequest(): DEBUG: Vulnerability scan for package 'libmythes-1.2-0' on Agent '000' has completed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Code 🟢
- Tests 🟠
- Errors detected on this PR will be addressed at:
Description
Considerations
Test
Update offsets from 257xxx to 429xxx
Before changes:
Memory:
Temporary allocations:
Summary:
Disk usage ->
Take 1:53 hours to update all the data.
After changes:
After changes:
Memory usage:
Temporary allocations:
Summary:
Disk usage
Take 18 minutes.