Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to limit the memory usage in write/read operations with rocksdb. #22693

Merged
merged 6 commits into from
Apr 4, 2024

Conversation

Dwordcito
Copy link
Member

@Dwordcito Dwordcito commented Mar 30, 2024

Related issue
#22581
#22529
#22528

Description

Considerations

Test

Update offsets from 257xxx to 429xxx

Before changes:
Memory:
image

Temporary allocations:
image

Summary:
image

Disk usage ->
image

Take 1:53 hours to update all the data.

After changes:

After changes:
Memory usage:
image

Temporary allocations:
image

Summary:
image

Disk usage
image

Take 18 minutes.

@Dwordcito Dwordcito linked an issue Apr 2, 2024 that may be closed by this pull request
5 tasks
Copy link
Member

@sebasfalcone sebasfalcone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Code 🟢
    • Some minor details regarding documentation and atributes
  • Test ⚪
    • Pending

Copy link
Member

@sebasfalcone sebasfalcone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scan with content (this branch)

cat ossec.log | grep "publish key" | wc -l
50

Scan with content (4.8.0)

cat ossec.log | grep "publish key" | wc -l
50

Scan without content (this branch)

Sync events and feed processing is done in parallel, this leads to errors at the scan

2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:423 at handleRequest(): DEBUG: Vulnerability scan for package 'libreoffice-help-de' on Agent '000' has completed.
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] scanOrchestrator.hpp:167 at run(): DEBUG: Event type: 1 processed
2024/04/04 16:15:20 wazuh-modulesd:syscollector[239792] logging_helper.c:40 at taggedLogFunction(): DEBUG: Sync sent: {"component":"syscollector_packages","data":{"attributes":{"architecture":"amd64","checksum":"a15cae8254677e9418fb013e59dfd542ef78c146","description":"simple thesaurus library","format":"deb","groups":"libs","install_time":" ","item_id":"60fbb3da57a48c87e2f95a780abd91ab7be32ba4","location":" ","multiarch":"same","name":"libmythes-1.2-0","priority":"optional","scan_time":"2024/04/04 14:15:20","size":32,"source":"mythes","vendor":"Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>","version":"2:1.2.4-4build1"},"index":"60fbb3da57a48c87e2f95a780abd91ab7be32ba4","timestamp":""},"type":"state"}
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:397 at handleRequest(): DEBUG: Initiating a vulnerability scan for package 'libmythes-1.2-0' (deb) (Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>) with CVE Numbering Authorities (CNA) 'nvd' on Agent 'Thinkpad-Sebas' (ID: '000', Version: 'v4.8.0').
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:415 at handleRequest(): WARNING: Failed to scan package: 'libmythes-1.2-0', CVE Numbering Authorities (CNA): 'nvd', Error: 'Couldn't find column family: 'nvd''
2024/04/04 16:15:20 wazuh-modulesd:vulnerability-scanner[239792] packageScanner.hpp:423 at handleRequest(): DEBUG: Vulnerability scan for package 'libmythes-1.2-0' on Agent '000' has completed.

Copy link
Member

@sebasfalcone sebasfalcone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Dwordcito Dwordcito merged commit 68435a5 into 4.8.0 Apr 4, 2024
89 of 90 checks passed
@Dwordcito Dwordcito deleted the fix/22581-fix-memory-management-of-rocksdb branch April 4, 2024 15:51
@Dwordcito Dwordcito restored the fix/22581-fix-memory-management-of-rocksdb branch April 8, 2024 13:58
@Dwordcito Dwordcito deleted the fix/22581-fix-memory-management-of-rocksdb branch April 13, 2024 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants