Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

optimize max detect records #1176

Merged
merged 1 commit into from
Sep 7, 2023
Merged

optimize max detect records #1176

merged 1 commit into from
Sep 7, 2023

Conversation

YamatoSecurity
Copy link
Collaborator

closes #1175

@fukusuket @hitenkoku
Could you two take some benchmarks to see if the speed increases?

@YamatoSecurity YamatoSecurity added the enhancement New feature or request label Sep 7, 2023
@YamatoSecurity YamatoSecurity added this to the v2.9.0 milestone Sep 7, 2023
@codecov
Copy link

codecov bot commented Sep 7, 2023

Codecov Report

Patch coverage has no change and project coverage change: -0.03% ⚠️

Comparison is base (9418eb2) 83.87% compared to head (f491103) 83.85%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1176      +/-   ##
==========================================
- Coverage   83.87%   83.85%   -0.03%     
==========================================
  Files          26       26              
  Lines       23404    23373      -31     
==========================================
- Hits        19630    19599      -31     
  Misses       3774     3774
Files Changed Coverage Δ
src/main.rs 68.68% <ø> (ø)

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fukusuket
fukusuket approved these changes Sep 7, 2023
View reviewed changes
Copy link
Collaborator

@fukusuket fukusuket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed following improvements :) (macOS/6.1GB EVTX) LGTM!!🚀

This PR

Elapsed time: 00:06:28.835
Rule Parse Processing Time: 00:00:01.697
Analysis Processing Time: 00:06:16.912
Output Processing Time: 00:00:10.223

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     6.0 GiB     6.0 GiB     0           6.0 GiB
 committed:     1.0 GiB     6.0 GiB   803.8 GiB  -797.8 GiB                          ok

main

Elapsed time: 00:06:33.924
Rule Parse Processing Time: 00:00:01.768
Analysis Processing Time: 00:06:21.798
Output Processing Time: 00:00:10.354

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     6.0 GiB     6.0 GiB     0           6.0 GiB
 committed:     1.0 GiB     6.0 GiB   225.0 GiB  -219.0 GiB                          ok

@hitenkoku
Copy link
Collaborator

executed ./hayabusa.exe csv-timeline -d ../hayabusa-sample-evtx -o hse.csv -q -C -p super-verbose

  • Thils PR

Saved file: hse.csv (39.2 MB)

Elapsed time: 00:00:07.412
Rule Parse Processing Time: 00:00:01.619
Analysis Processing Time: 00:00:05.277
Output Processing Time: 00:00:00.515

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:   699.7 MiB     1.1 GiB   618.5 MiB   592.0 MiB
     reset:     0
    purged:   342.4 MiB
   touched:   128.5 KiB     4.7 MiB     8.1 GiB    -8.1 GiB                          ok
  segments:    12          76          65          11                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0          55.8 Ki    -55.8 Ki                           ok
-abandoned:     3           3           0           3                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     6.9 Ki
    resets:     0
    purges:   332
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:     7.418 s
   process: user: 22.562 s, system: 0.421 s, faults: 224057, rss: 615.7 MiB, commit: 704.6 MiB
  • main
Saved file: hse.csv (39.2 MB)

Elapsed time: 00:00:08.039
Rule Parse Processing Time: 00:00:01.631
Analysis Processing Time: 00:00:05.894
Output Processing Time: 00:00:00.513

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:   728.2 MiB     1.3 GiB   746.8 MiB   590.9 MiB
     reset:     0
    purged:   420.6 MiB
   touched:   128.5 KiB     4.9 MiB     8.0 GiB    -8.0 GiB                          ok
  segments:    13          79          68          11                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0          54.9 Ki    -54.9 Ki                           ok
-abandoned:     3           3           0           3                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     7.1 Ki
    resets:     0
    purges:   260
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:     8.082 s
   process: user: 31.359 s, system: 0.437 s, faults: 235675, rss: 630.4 MiB, commit: 733.1 MiB

@hitenkoku
Copy link
Collaborator

executed ./hayabusa.exe csv-timeline -d ../all-evtx -o hse.csv -q -C -p super-verbose
(6.1GB evtx)

  • Thils PR
Saved file: hse.csv (1.6 GB)

Elapsed time: 00:04:45.927
Rule Parse Processing Time: 00:00:01.688
Analysis Processing Time: 00:04:22.850
Output Processing Time: 00:00:21.388

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    10.0 GiB    10.0 GiB     0          10.0 GiB
 committed:     7.2 GiB    44.1 GiB    37.0 GiB     7.1 GiB
     reset:     0
    purged:    13.0 GiB
   touched:   128.5 KiB     4.3 MiB    74.4 GiB   -74.4 GiB                          ok
  segments:    14          69          58          11                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0         861.4 Ki   -861.4 Ki                           ok
-abandoned:     3           3           0           3                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   310.8 Ki
    resets:     0
    purges:    20.3 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   285.938 s
   process: user: 234.390 s, system: 5.796 s, faults: 7469319, rss: 7.1 GiB, commit: 7.2 GiB
  • main
Saved file: hse.csv (1.6 GB)

Elapsed time: 00:04:52.230
Rule Parse Processing Time: 00:00:02.329
Analysis Processing Time: 00:04:30.465
Output Processing Time: 00:00:19.434

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     8.0 GiB     8.0 GiB     0           8.0 GiB
 committed:     8.5 GiB    36.2 GiB    29.2 GiB     7.0 GiB
     reset:     0
    purged:    12.6 GiB
   touched:   128.5 KiB    14.0 MiB    60.0 GiB   -60.0 GiB                          ok
  segments:    14         224         213          11                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0         753.7 Ki   -753.7 Ki                           ok
-abandoned:     3           3           0           3                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   259.4 Ki
    resets:     0
    purges:     8.7 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   292.266 s
   process: user: 2728.140 s, system: 11.343 s, faults: 6135739, rss: 7.1 GiB, commit: 7.2 GiB

@hitenkoku
Copy link
Collaborator

executed ./hayabusa.exe csv-timeline -d ../Big-Security.evtx -o hse.csv -q -C -p super-verbose
(2.1GB evtx)

  • this PR
Saved file: hse.csv (1.9 GB)

Elapsed time: 00:04:00.231
Rule Parse Processing Time: 00:00:02.347
Analysis Processing Time: 00:02:53.885
Output Processing Time: 00:01:03.997

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    16.0 GiB    16.0 GiB     0          16.0 GiB
 committed:    13.7 GiB    39.7 GiB    26.1 GiB    13.5 GiB
     reset:   362.9 MiB
    purged:    10.9 GiB
   touched:   128.5 KiB     6.4 MiB    56.6 GiB   -56.6 GiB                          ok
  segments:    16         103          92          11                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0         679.2 Ki   -679.2 Ki                           ok
-abandoned:     3           3           0           3                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   238.9 Ki
    resets:     8
    purges:    17.0 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   240.268 s
   process: user: 155.984 s, system: 6.312 s, faults: 9865734, rss: 11.6 GiB, commit: 13.8 GiB
  • main
Saved file: hse.csv (1.9 GB)

Elapsed time: 00:04:26.400
Rule Parse Processing Time: 00:00:02.159
Analysis Processing Time: 00:03:08.472
Output Processing Time: 00:01:15.764

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    16.0 GiB    16.0 GiB     0          16.0 GiB
 committed:    13.7 GiB    36.6 GiB    23.0 GiB    13.5 GiB
     reset:   362.9 MiB
    purged:     8.1 GiB
   touched:   128.5 KiB     6.5 MiB    45.4 GiB   -45.4 GiB                          ok
  segments:    17         104          92          12                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0         591.3 Ki   -591.3 Ki                           ok
-abandoned:     3           3           0           3                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   305.9 Ki
    resets:     8
    purges:     5.4 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   266.410 s
   process: user: 2070.937 s, system: 24.171 s, faults: 10368889, rss: 11.6 GiB, commit: 13.7 GiB

hitenkoku
hitenkoku approved these changes Sep 7, 2023
View reviewed changes
Copy link
Collaborator

@hitenkoku hitenkoku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed improved result by following comment.
LGTM.

@hitenkoku hitenkoku merged commit 3ab8c9b into main Sep 7, 2023
9 checks passed
@hitenkoku hitenkoku deleted the 1175-optimize-max-detect-records branch September 7, 2023 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hitenkoku hitenkoku hitenkoku approved these changes

@fukusuket fukusuket fukusuket approved these changes

Assignees

@YamatoSecurity YamatoSecurity

Labels
enhancement New feature or request
Projects
None yet
Milestone
v2.9.0
Development

Successfully merging this pull request may close these issues.

Optimize max records to load
3 participants
@YamatoSecurity @hitenkoku @fukusuket