Create developer's guide for performance enhancements

[YamatoSecurity]

I would like to create some documentation about Rust performance for future developers as well as Rust programmers interested in making their programs very fast and memory efficient.
We can put it under doc/RustPerformance-English.md and doc/RustPerformance-Japanese.md.
@fukusuket Could I ask you to write the first draft? (No hurry, so whenever you have extra time.)
You can just document all of your performance enhancements you made recently in Japanese and I can handle the English translations when it is finished.
I would like to include the old sample code versus new sample code and roughly how much it improves speed and/or memory.
Maybe group by category. For example:

1. Avoid regular expressions as much as possible. Here are some ways that we filter with String comparisons first, etc..
2. Use Iterators instead of Vec.
   Iterators can reduce memory usage because memory allocation only occurs one record at a time.
   Also, change the return type to Box to make Iterator the return value of the function.
   Iterator is trait (not structs), so need to be wrapped in a Box to return from a function
3. Avoid using clone
4. Use compactstr to avoid heap allocation
5. Use mimalloc for Windows, xxx for linux/mac
6. etc...

What do you think?

[fukusuket]

That's sounds nice :) I'd love to write a draft document 💪
I'd start by sorting through the recent performance improvements.

Also, it would be better to be able to describe how to measure performance? (heaptrack, mimalloc-stats, memory layout ... etc)

[YamatoSecurity]

@fukusuket Thank you so much! I will assign you then.
Indeed, we should probably explain how we are measuring performance. Recently I have been putting my benchmarks in your pull requests and noting them in the changelog as well. We can use your or my benchmarks. I don't think it has to be perfect measurements. Just as long as we give people an idea of how much performance changed in our project.

[fukusuket]

memo:

Category of performance improvement

Memory

by removing unnecessary fields and string

• Reduct DetectInfo fields memory #806
• Reduced memory usage of DetectInfo/EvtxRecordInfo #837
• Reduce memory usage by removing unnecessary regex #894

by reducing clone() and to_string()

• Reduce used memory and Skipped rule author, detect counts aggregation when --no-summary option is used #782

by using iterator instead of vec

• Reduce memory usage when reading JSONL file #921
• Reduce memory usage when reading json file #906
• メモリの利用量が大きい #276

by using external crate

• To reduce ram usage and performance, Replaced String with other crate #793
• HashMapのcrate変更による速度向上 #368

by changing global memory allocator(mimalloc)

• chg: build.rs(for vc runtime) to rustflags in config.toml and replace default global memory allocator with mimalloc. #777

Speed

by reducing IO process

• Improve speed by removing IO process before insert_message() #858

by changing regex match to std:string method match

• Improving speed by changing EventID matching process from regular expression to exact string matching #882
• Improving speed by adding string length match before regular expression match #883
• Improving speed by using eq_ignore_ascii_case() before regular expression match #884
• Imporving speed by changing wildcard search process from regular expression match to starts_with/ends_with match #890
• [SIGMAルール] contains,startsWith,endsWith対応 #79

by changing global memory allocator(mimalloc)

• chg: build.rs(for vc runtime) to rustflags in config.toml and replace default global memory allocator with mimalloc. #777

by changing buffrerd IO

• バッファIOに変更したい。 #253

by chacheing regex

• Regexクラスのインスタンスをキャッシュしたい #75
• ルールファイルの読み込みが遅い #254
• whitelistキーワードかregexesキーワードを含むルールが遅い。 #173

by using multithread

• マルチスレッド対応 #68

Benchmarking

• heaptrack
  • https://medium.com/lumen-engineering-blog/tutorial-profiling-cpu-and-ram-usage-of-rust-micro-services-running-on-kubernetes-fbc32714da93
  • https://www.brendangregg.com/flamegraphs.html
• mimalloc-stats
  • add --debug option for printing mimalloc memory stats #822
• about memory layout
  • https://www.youtube.com/watch?v=rDoqT-a6UFg
  • https://doc.rust-lang.org/std/mem/fn.size_of.html

Reference

Rust performance guide

• https://nnethercote.github.io/perf-book/introduction.html

enum vs trait

• https://users.rust-lang.org/t/performance-implications-of-box-trait-vs-enum-delegation/11957
• https://github.com/Britefury/rust-enum-vs-trait-benchmark
• https://www.possiblerust.com/guide/enum-or-trait-object
• https://qiita.com/taiki-e/items/39688f6c86b919988222
• https://qiita.com/carrotflakes/items/896ce7f49931c64a2954

[fukusuket]

@YamatoSecurity
I'm starting to write a performance guide on this branch.
I'm thinking of writing with the above structure,🤔 what do you think? I would appreciate it if you could comment if there is anything that should be changed in the paragraph. 🙏 (It's still a draft of a draft, so anything can be changed :) )

[YamatoSecurity]

@fukusuket Looks great!
What about changing the order from 1. メモリ使用量の削減 2. 速度の改善 3. ベンチマークの取得 to 1. 速度の改善 2. メモリ使用量の削減 3. ベンチマークの取得 ?
I think this might be better as most people want to hear about speed improvements first and 2 & 3 go together more. (reducing memory and checking with benchmarks)
Everything else looks great.
It would be nice to have a screenshot of the mimalloc --debug output and explain which numbers are the most important and how the latest version of mimalloc seems to have a bug that gives results like -128GB but just to ignore that as MS needs to fix it.

[fukusuket]

Thank you for reviewing :) I'll update the order paragraphs and put a screenshot!
I'm still working on it, so if you have any other suggestions, feel free to let me know 💪

[fukusuket]

@YamatoSecurity
I have finished writing the Japanese version of the draft document on this branch :)
I would appreciate it if you could do a review and an English translation when you have time🙏

[YamatoSecurity]

@fukusuket Thank you so much! It might take a little while as I have to prepare for teaching SANS next week but can definitely do it after that.

@hitenkoku Can you add whatever performance advice you have as well?

[hitenkoku]

@fukusuket Thank you for writing. It is very useful for me.

@YamatoSecurity I have nothing to add.