Reduce memory usage by removing unnecessary regex

[fukusuket]

What Changed

• Reduced memory usage(about 200-300MB
  in the data below) by removing regex compilation process when the search can be replaced with the fast search below.

  • Improving speed by using eq_ignore_ascii_case() before regular expression match #884
  • Improving speed by adding string length match before regular expression match #883

• Changed pipe startswith/endswith matching from regex to Imporving speed by changing wildcard search process from regular expression match to starts_with/ends_with match #890 match.

  • This change didn't speed up matching, but it is changed for matching process refactoring

Evidence

Environment

• OS: Windows 10 Home edition
• Hard: Memory 16GB , Core 8, SSD, laptop

Benchmark1

I ran a benchmark using this procedure(6.1GB evtx) and the results were as follows.

Version	Elapsed time	Memory(peak)	Events with hits / Total events	Output file size(bytes)
before	00:10:44.026	4.7 GiB	1,593,715 / 4,817,181	575085389
This PR	00:10:42.195	4.4 GiB	1,593,715 / 4,817,181	575085385

Console output

before

PS C:\tmp\hayabusa-2.1.0-win-64-bit> .\hayabusa.exe csv-timeline -d ..\hayabusa-big-evtx\ -o 1.csv --debug
Results Summary:

Events with hits / Total events: 1,593,715 / 4,817,181 (Data reduction: 3,223,466 events (66.92%))

Total | Unique detections: 1,627,284 | 150
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,044 (0.74%) | 20 (13.33%)
Total | Unique medium detections: 11,118 (0.68%) | 38 (25.33%)
Total | Unique low detections: 1,053,623 (64.75%) | 42 (28.00%)
Total | Unique informational detections: 550,499 (33.83%) | 50 (33.33%)

...

Saved file: 1.csv (575.1 MB)
Elapsed time: 00:10:44.026
Rule Parse Processing Time: 00:00:20.191
Analysis Processing Time: 00:10:02.174
Output Processing Time: 00:00:21.658

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    4.7 GiB    4.7 GiB   83.0 MiB    4.6 GiB
 committed:    4.3 GiB   57.6 GiB   53.4 GiB    4.1 GiB

This PR

PS C:\tmp\hayabusa-2.1.0-win-64-bit> .\hayabusa.exe csv-timeline -d ..\hayabusa-big-evtx\ -o 1.csv --debug
Results Summary:

Events with hits / Total events: 1,593,715 / 4,817,181 (Data reduction: 3,223,466 events (66.92%))

Total | Unique detections: 1,627,284 | 150
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,044 (0.74%) | 20 (13.33%)
Total | Unique medium detections: 11,118 (0.68%) | 38 (25.33%)
Total | Unique low detections: 1,053,623 (64.75%) | 42 (28.00%)
Total | Unique informational detections: 550,499 (33.83%) | 50 (33.33%)
...

Saved file: 1.csv (575.1 MB)
Elapsed time: 00:10:42.195
Rule Parse Processing Time: 00:00:20.234
Analysis Processing Time: 00:10:00.446
Output Processing Time: 00:00:21.513

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    4.4 GiB    4.4 GiB   83.0 MiB    4.4 GiB
 committed:    4.0 GiB   65.0 GiB   61.2 GiB    3.8 GiB

Benchmark2

I ran a benchmark using hayabusa-sample-evtx and the results were as follows.

Version	Elapsed time	Memory(peak)	Events with hits / Total events	Output file size(bytes)
before	00:00:16.158	1.9 GiB	19,606 / 47,458	16447790
This PR	00:00:15.422	1.6 GiB	19,606 / 47,458	16447790

I would appreciate it if you could review🙏

[codecov]

Codecov Report

Base: 73.84% // Head: 73.92% // Increases project coverage by +0.08% 🎉

Coverage data is based on head (47af259) compared to base (6cb871b).
Patch coverage: 96.12% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #894      +/-   ##
==========================================
+ Coverage   73.84%   73.92%   +0.08%     
==========================================
  Files          23       23              
  Lines       14576    14643      +67     
==========================================
+ Hits        10763    10825      +62     
- Misses       3813     3818       +5     

Impacted Files	Coverage Δ
src/detections/rule/matchers.rs	96.45% <96.12%> (-0.15%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[hitenkoku]

@fukusuket Thanks for your pull request.

LGTM

[fukusuket]

@hitenkoku
Thank you so much for quick review🙇 I fixed typo :) 30d78d6

[YamatoSecurity]

@fukusuket PRありがとうございます！
Macのベンチマークではメモリが300MB減っていますが、処理時間が少し増えています。
21:54.523 -> 22:26.437
１回目はメモリ使用 7.5GB -> 7.2GB、２回目は 7.5GB -> 7.4GBで、処理時間は二回目も同じ22:26でした。
明日Windowsの方でも試してみるので、少々お待ち下さい。

[fukusuket]

@YamatoSecurity
ベンチマーク早速取得いただきありがとうございます🙇
マッチ速度に影響しない想定の変更でしたが、すみません！どこかでデグレさせてしまっていそうです、確認いたします！
(またデータにより遅くなるパターンがありそうでしたら、こちらのPR一旦取り下げます！)

[fukusuket]

6.1GB evtxに対して手元のMac環境で試したところ(それぞれ測定前に再起動, 2回ずつ測定)、以下の結果でした🤔
なので、データのパターンによりスピード劣化してしまうケースがあるということかもしれません.... 😭　

Environment

• OS: macOS montery version 13.1
• Hard: Macbook Air(M1, 2020) , Memory 8GB, Core 8

before

Elapsed time: 00:06:22.257
Rule Parse Processing Time: 00:00:01.861
Analysis Processing Time: 00:06:15.218
Output Processing Time: 00:00:05.178

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    4.9 GiB    4.9 GiB   83.0 MiB    4.8 GiB
 committed:    4.5 GiB   49.8 GiB   45.5 GiB    4.3 GiB

This PR

Elapsed time: 00:06:12.900
Rule Parse Processing Time: 00:00:01.310
Analysis Processing Time: 00:06:06.634
Output Processing Time: 00:00:04.955

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    4.6 GiB    4.6 GiB   83.0 MiB    4.5 GiB
 committed:    4.1 GiB   49.6 GiB   45.7 GiB    3.9 GiB

[YamatoSecurity]

@fukusuket Hummm, yea, it might just be my environment so I will check tonight on a different computer.

[YamatoSecurity]

LGTM!
When checking on my Windows environment.
Speed improvement: 25:25 -> 24:45
Memory reduction: 7.6GB -> 7.2GB

[fukusuket]

Thank you for your review and benchmark :)