Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

To reduce ram usage and performance, Replaced String with other crate #793

Merged
merged 37 commits into from
Nov 5, 2022
Merged

To reduce ram usage and performance, Replaced String with other crate #793

merged 37 commits into from
Nov 5, 2022

Conversation

hitenkoku
Copy link
Collaborator

@hitenkoku hitenkoku commented Nov 5, 2022 •
edited
Loading

Closes #790 , Closes #789

What Changed

  • To improve memory usage, removed a few call static variables.
  • To improve memory usage, replaced Vec with Nested.
  • To improve memory usage, replaced String Object in DetectInfo with CompactString.
  • Updated changelog.

Evidence

analyzed result to 2GB evtx file by the following environment.

  • CPU: Ryzen 5700G
  • RAM: 16GB
  • OS: Windows 11
step prev memory usage in 4701a41 memory usage(prev step diff percent) improved in fba1688
start process 31% 24% - 29%
finished analyze 94%(+63%) 78%(+54%) 18% 82%(+53%)
finished output 94%(0%) 80%(+2%) 16% 83%(+1%)

output time ( in 4701a41)

Elapsed time: 00:03:28.181
Saved file: output.csv (610.9 MB)

output time(in fba1688 )

Elapsed time: 00:03:26.145
Saved file: output-2.csv (610.9 MB)

@hitenkoku
To reduce used ram. Replaced LinkedHashMap to Nested<Vec<String>>
65cbdcf
@hitenkoku
cargo fmt
c0e8af1
@hitenkoku
to fix ext_field message parse.
2d71fa2
@hitenkoku
removed unnecessary collect process
6da984e
@hitenkoku
To improved memory usage, replaced Vec<String> to Nested<String>
1898746
@hitenkoku
to improve speed and memory usage, changed process of related to tags.
7d6e2d9 
target is following tags.

- %MitreTactics%
- %MitreTags%
- %OtherTags%
@hitenkoku
To improve memory usage, Replaced Vec<String> to Nested<String>
d7cea61
@hitenkoku
To improve memory usage, replaced Vec<String> to Nested<String>
30d29f3
@hitenkoku
to improve memory usage, replaced Vec<String> to Nested<String>
c017deb
@hitenkoku
removed unnecessary static termilan_size variable
9c1a3a3
@hitenkoku
To reduce memory usage, Removed ERROR_LOG_PATH
7169bde
@hitenkoku
To reduce memory usage, moved static variable to local
de2717a
@hitenkoku
to reduce memory usage, removed few call static variable (EXCLUDE_STA…
2b2029c 
…TUS)
@hitenkoku
to reduce memory usage, Removed few used static variable
181de14 
(TARGET_EXTENSIONS)
@hitenkoku
to reduce memory usage, Removed few used static variable (LEVEL_MAP)
1b510cd
@hitenkoku
cargo fmt
2248947
@hitenkoku
added compressed_string crate to improve memory usage
f9e46c4
@hitenkoku
to improve memory usage, replaced String to ComprString
b36a089
@hitenkoku
cargo fmt
5b49bd2
@hitenkoku
to reduce memory usage, replaced String to ComprString in DetectInfo
7f53580
@hitenkoku
fixed warning in not windows environment
c0cf508
@hitenkoku
Revert "to reduce memory usage, replaced String to ComprString in Det…
9600aa2 
…ectInfo"

This reverts commit 7f53580.
@hitenkoku
To improve performance, Replaced Vec<Vec<String>> to Nested<Vec<String>>
cda9ac8 
(get_profile_list)
@hitenkoku
cargo fmt
f0ea9ab
@hitenkoku
to improve memory usage, changed String in static hashmap variable to
d0f10ae 
static str
@hitenkoku
to improve performance, removed unused lazy static variable
96e3bd3
@hitenkoku
to improve memory performace, removed clone in selectionnodes #787
ec3a902
@hitenkoku
to improve memory usage and performance, replaced compressed string to
ce4f981 
compact_str
@hitenkoku
cargo fmt
9cb386c
@hitenkoku
Merge branch 'main' into test-reduce_ram_usage_remove_linked_hashmap
b204414
@hitenkoku hitenkoku added the enhancement New feature or request label Nov 5, 2022
@hitenkoku hitenkoku added this to the v1.8.0 milestone Nov 5, 2022
@hitenkoku hitenkoku self-assigned this Nov 5, 2022
YamatoSecurity
YamatoSecurity approved these changes Nov 5, 2022
View reviewed changes
Copy link
Collaborator

@YamatoSecurity YamatoSecurity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@hitenkoku hitenkoku marked this pull request as ready for review November 5, 2022 14:46
@hitenkoku hitenkoku changed the title Test reduce ram usage remove linked hashmap To reduce ram usage and performance, Replaced String with other crate Nov 5, 2022
@hitenkoku hitenkoku merged commit e4980ff into main Nov 5, 2022
@hitenkoku hitenkoku deleted the test-reduce_ram_usage_remove_linked_hashmap branch November 5, 2022 15:01
@hitenkoku
Copy link
Collaborator Author

hitenkoku commented Nov 5, 2022 •
edited
Loading

thank you for your review. I merged it.

@fukusuket
Copy link
Collaborator

fukusuket commented Nov 5, 2022 •
edited
Loading

Compared to the main branch before merging, I have confirmed the following performance improvements!!
(6.1GB evtx)

  • before-fix: Mem(avg):6.85 Mem(max):9.48
  • after-fix : Mem(avg):6.45 Mem(max):8.89
    (Since the OS uses about 3GB of memory, the actual usage is -3GB.)

Mem(avg): -0.40GB Mem(max):-0.59GB Great imporvements ! 🚀
(and I also confirmed that the number of detections was the same before and after fix.)

LGTM:)

@fukusuket fukusuket mentioned this pull request Feb 18, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YamatoSecurity YamatoSecurity YamatoSecurity approved these changes

@fukusuket fukusuket Awaiting requested review from fukusuket

Assignees

@hitenkoku hitenkoku

Labels
enhancement New feature or request
Projects
None yet
Milestone
v1.8.0
3 participants
@hitenkoku @fukusuket @YamatoSecurity