added new feature `-P, --proven-rules` #1120

hitenkoku · 2023-07-03T10:12:25Z

What Changed

Added -P, --proven-rules option in csv-timeline and json-timeline command. To scan only target rule ids that there are detection rules for defined in ./rules/config/proven_rules.txt.

I would appreciate it if you could review when you have time.

hitenkoku · 2023-07-03T10:14:46Z

Evidence

this PR(without -P)

> ./1115.exe csv-timeline -d ..\hayabusa-sample-evtx\ -o 1115-no-pr.csv --debug -C -q   

Total event log files: 584
Total file size: 137.1 MB

Loading detections rules. Please wait.

Excluded rules: 31
Noisy rules: 12 (Disabled)

Deprecated rules: 165 (4.50%) (Disabled)
Experimental rules: 1973 (53.77%)
Stable rules: 229 (6.24%)
Test rules: 1467 (39.98%)
Unsupported rules: 45 (1.23%) (Disabled)

Hayabusa rules: 159
Sigma rules: 3510
Total enabled detection rules: 3669

Output profile: standard

...

Results Summary:
Events with hits / Total events: 19,642 / 47,472 (Data reduction: 27,830 events (58.62%))

Total | Unique detections: 32,228 | 561
Total | Unique critical detections: 56 (0.17%) | 20 (3.57%)
Total | Unique high detections: 6,043 (18.75%) | 245 (43.67%)
Total | Unique medium detections: 1,894 (5.88%) | 184 (32.80%)
Total | Unique low detections: 6,002 (18.62%) | 56 (9.98%)
Total | Unique informational detections: 18,233 (56.58%) | 56 (9.98%)   

...

Saved file: 1115-no-pr.csv (32.0 MB)

Elapsed time: 00:00:07.950
Rule Parse Processing Time: 00:00:01.432
Analysis Processing Time: 00:00:06.077
Output Processing Time: 00:00:00.439

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB             

 committed:   845.1 MiB     1.3 GiB   841.4 MiB   525.9 MiB             

     reset:     0
    purged:   376.6 MiB
   touched:    64.2 KiB     1.1 MiB     3.5 GiB    -3.5 GiB             
             ok
  segments:    16          19           9          10                   
             not all freed!
-abandoned:     0           0           0           0                   
             ok
   -cached:     0           0           0           0                   
             ok
     pages:     0           0          33.1 Ki    -33.1 Ki              
             ok
-abandoned:     0           0           0           0                   
             ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     7.0 Ki
    resets:     0
    purges:   301
   threads:    32          32           0          32                   
             not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:     7.954 s
   process: user: 27.703 s, system: 0.781 s, faults: 261488, rss: 684.9 MiB, commit: 824.0 MiB

this PR(with -P)

> ./1115.exe csv-timeline -d ..\hayabusa-sample-evtx\ -o 1115-pr.csv --debug -P -C -q
Total event log files: 584
Total file size: 137.1 MB

Loading detections rules. Please wait.

Excluded rules: 1401
Noisy rules: 12 (Disabled)

Deprecated rules: 85 (3.55%) (Disabled)
Experimental rules: 1368 (57.10%)
Stable rules: 76 (3.17%)
Test rules: 952 (39.73%)
Unsupported rules: 28 (1.17%) (Disabled)

Hayabusa rules: 20
Sigma rules: 2376
Total enabled detection rules: 2396

...

Results Summary:
Events with hits / Total events: 54 / 47,472 (Data reduction: 47,418 events (99.89%))

Total | Unique detections: 98 | 4
Total | Unique critical detections: 1 (1.02%) | 1 (25.00%)
Total | Unique high detections: 0 (0.00%) | 0 (0.00%)
Total | Unique medium detections: 3 (3.06%) | 1 (25.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 94 (95.92%) | 2 (50.00%)       

...

Saved file: 1115-no-pr.csv (37.3 KB)

Elapsed time: 00:00:04.082
Rule Parse Processing Time: 00:00:00.953
Analysis Processing Time: 00:00:03.088
Output Processing Time: 00:00:00.040

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB             

 committed:   457.3 MiB   957.4 MiB   589.2 MiB   368.1 MiB             

     reset:     0
    purged:   265.1 MiB
   touched:    64.2 KiB   835.2 KiB     3.1 GiB    -3.1 GiB             
             ok
  segments:     9          13           6           7                   
             not all freed!
-abandoned:     0           0           0           0                   
             ok
   -cached:     0           0           0           0                   
             ok
     pages:     0           0          25.3 Ki    -25.3 Ki              
             ok
-abandoned:     0           0           0           0                   
             ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     4.5 Ki
    resets:     0
    purges:   173
   threads:    32          32           0          32                   
             not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:     4.088 s
   process: user: 12.765 s, system: 0.468 s, faults: 162312, rss: 361.6 MiB, commit: 454.3 MiB

codecov · 2023-07-03T10:17:35Z

Codecov Report

Patch coverage: 70.23% and project coverage change: -0.02 ⚠️

Comparison is base (fea4d14) 82.47% compared to head (d3336b5) 82.45%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1120      +/-   ##
==========================================
- Coverage   82.47%   82.45%   -0.02%     
==========================================
  Files          24       24              
  Lines       20820    20890      +70     
==========================================
+ Hits        17171    17225      +54     
- Misses       3649     3665      +16

Impacted Files	Coverage Δ
src/detections/configs.rs	`65.64% <50.00%> (-0.26%)`	⬇️
src/yaml.rs	`88.54% <50.00%> (-0.37%)`	⬇️
src/main.rs	`64.93% <66.66%> (+0.09%)`	⬆️
src/afterfact.rs	`71.30% <100.00%> (+0.10%)`	⬆️
src/detections/detection.rs	`75.28% <100.00%> (+0.07%)`	⬆️
src/detections/rule/condition_parser.rs	`96.98% <100.00%> (+<0.01%)`	⬆️
src/detections/rule/count.rs	`93.58% <100.00%> (+<0.01%)`	⬆️
src/detections/rule/matchers.rs	`97.07% <100.00%> (+<0.01%)`	⬆️
src/detections/rule/mod.rs	`94.69% <100.00%> (+<0.01%)`	⬆️
src/detections/rule/selectionnodes.rs	`92.20% <100.00%> (+0.01%)`	⬆️
... and 3 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

fukusuket

@hitenkoku @YamatoSecurity
I confirmed that the default proven_rules.txt has fewer detection rules to load :)

I have a question about the detection result after changing proven_rules.txt!
I expect that only 1 rule will be detected when only 1 rule is described in proven_rules.txt as follows,

hayabusa-2.6.0-all-platforms % cat ./rules/config/proven_rules.txt
002bdb95-0cf1-46a6-9e08-d38c128a6127

However, when comparing the detection results for hayabusa-sample-evtx, there is no difference as shown below.
(no-p.csv == without -P option, with-p.csv == with -P option)

hayabusa-2.6.0-all-platforms % ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -o no-p.csv -q -C
...
hayabusa-2.6.0-all-platforms % ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -o with-p.csv -q -C -P
...
hayabusa-2.6.0-all-platforms % diff no-p.csv with-p.csv
hayabusa-2.6.0-all-platforms %

Is the above expected behavior?
I would appreciate it if you could check it🙏

YamatoSecurity · 2023-07-04T00:47:11Z

@fukusuket
The number of rules in proven_rules.txt is less than the number of rules available. I made this list by scanning all of the evtx files I have available. If you or @hitenkoku have proven detection rules that are not in this list, please add the rule ID to this file.
If, for example, you only put one rule 002bdb95-0cf1-46a6-9e08-d38c128a6127 inside this file and use the -P option, then hayabusa should only find detections with that rule ID. (Usually just one rule, but might be 2 if it is a process_creation rule that is actually two different yml files)
So it is indeed strange that the unique detections would decrease so much.
In all of our tests, the detections should be the same but I want to decrease the amount of loaded rules which will decrease the processing time and memory usage.

hitenkoku · 2023-07-04T01:12:16Z

The proven_rules.txt file is the latest version from the hayabusa-rules repository and has not been modified.

I will check how it looks like when only one element is used.

…statement #1115

hitenkoku · 2023-07-04T14:47:41Z

We are very sorry. There was an error in the condition when filtering the rules at ea444a8 Corrected.

> cat .\rules\config\proven_rules.txt
002bdb95-0cf1-46a6-9e08-d38c128a6127
>  ./1115.exe csv-timeline -d ../hayabusa-sample-evtx -o with-p.csv -q -C -P
...
Total event log files: 584
Total file size: 137.1 MB

Loading detections rules. Please wait.

Excluded rules: 3909
Noisy rules: 12 (Disabled)

Experimental rules: 1 (100.00%)

Sigma rules: 1
Total enabled detection rules: 1

Output profile: standard

Scanning in progress. Please wait.

584 / 584 [========================================================] 100.00 %

Scanning finished. Please wait while the results are being saved.


Results Summary:
Events with hits / Total events: 0 / 47,472 (Data reduction: 47,472 events (100.00%))

Total | Unique detections: 0 | 0
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 0 (0.00%) | 0 (0.00%)
Total | Unique medium detections: 0 (0.00%) | 0 (0.00%)
Total | Unique low detections: 0 (0.00%) | 0 (0.00%)
Total | Unique informational detections: 0 (0.00%) | 0 (0.00%)

Dates with most total detections:
critical: n/a, high: n/a, medium: n/a, low: n/a, informational: n/a

Top 5 computers with most unique detections:
critical: n/a
high: n/a
medium: n/a
low: n/a
informational: n/a

...
Saved file: with-p.csv (0 B)

Elapsed time: 00:00:01.488

fukusuket

@hitenkoku @YamatoSecurity
Thank you so much for a quick fix :)
I confirmed that only rules listed in proven_rules.txt are loaded! LGTM!🚀

Although It the issue on the hayabusa_rule(proven_rules.txt) side, when I checked the diff, it was as follows.

hayabusa-2.6.0-all-platforms % cat old.csv | awk -F"," '{print $5, $7}' | sort | uniq -c > old-rule-count.csv
hayabusa-2.6.0-all-platforms % cat fix.csv | awk -F"," '{print $5, $7}' | sort | uniq -c > fix-rule-count.csv
hayabusa-2.6.0-all-platforms % diff old-rule-count.csv fix-rule-count.csv
3d2
<    1 "crit" "Audit CVE Event"
291c290
<  243 "info" "Net Conn"
---
>  151 "info" "Net Conn"
302d300
<    2 "info" "RDP Denied"
462d459
<    3 "med" "Potential RDP Session Hijacking Activity"

Based on the above diff results, I think we should add the following to proven_rules.txt, what do you think?

YamatoSecurity

@hitenkoku Thank you for the update! I confirmed that it is working well for me too.
@fukusuket Thank you for checking. I am currently in the process of rechecking the rules to enable. I will be sure to add the rules that you mentioned. I'll submit a PR to hayabusa-rules later to fix this. I think we can merge this for now.

hitenkoku · 2023-07-05T08:16:42Z

@YamatoSecurity @fukusuket Thanks for your review. I will merge it.

hitenkoku added 3 commits July 3, 2023 19:01

feat: added proven-rules option in csv-timeline and json-timeline #1115

a762b63

test: adjusted test to add -P --proven-rules option #1115

fe9ded7

docs(CHANGELOG): added #1115

c0aaf98

hitenkoku added the enhancement New feature or request label Jul 3, 2023

hitenkoku requested a review from YamatoSecurity July 3, 2023 10:12

hitenkoku self-assigned this Jul 3, 2023

hitenkoku linked an issue Jul 3, 2023 that may be closed by this pull request

New feature: -P, --proven-rules #1115

Closed

fix: fixed proven-rule excluded condition #1115

e3d412d

YamatoSecurity requested a review from fukusuket July 3, 2023 10:46

fukusuket reviewed Jul 4, 2023

View reviewed changes

YamatoSecurity and others added 2 commits July 4, 2023 10:16

update changelog

dffc042

fix(yaml): fixed proven-rules option filtering condition wrong in if …

ea444a8

…statement #1115

Merge branch 'main' into 1115-new-feature-p-proven-rules

41064ca

fukusuket self-requested a review July 4, 2023 23:57

fukusuket approved these changes Jul 4, 2023

View reviewed changes

YamatoSecurity approved these changes Jul 5, 2023

View reviewed changes

Merge branch 'main' into 1115-new-feature-p-proven-rules

d3336b5

hitenkoku merged commit e6e8afc into main Jul 5, 2023
9 checks passed

hitenkoku deleted the 1115-new-feature-p-proven-rules branch July 5, 2023 08:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added new feature `-P, --proven-rules` #1120

added new feature `-P, --proven-rules` #1120

hitenkoku commented Jul 3, 2023

hitenkoku commented Jul 3, 2023 •

edited

Loading

codecov bot commented Jul 3, 2023 •

edited

Loading

fukusuket left a comment •

edited

Loading

YamatoSecurity commented Jul 4, 2023

hitenkoku commented Jul 4, 2023

hitenkoku commented Jul 4, 2023

fukusuket left a comment •

edited

Loading

YamatoSecurity left a comment

hitenkoku commented Jul 5, 2023

added new feature -P, --proven-rules #1120

added new feature -P, --proven-rules #1120

Conversation

hitenkoku commented Jul 3, 2023

What Changed

hitenkoku commented Jul 3, 2023 • edited Loading

Evidence

codecov bot commented Jul 3, 2023 • edited Loading

Codecov Report

fukusuket left a comment • edited Loading

Choose a reason for hiding this comment

YamatoSecurity commented Jul 4, 2023

hitenkoku commented Jul 4, 2023

hitenkoku commented Jul 4, 2023

fukusuket left a comment • edited Loading

Choose a reason for hiding this comment

YamatoSecurity left a comment

Choose a reason for hiding this comment

hitenkoku commented Jul 5, 2023

added new feature `-P, --proven-rules` #1120

added new feature `-P, --proven-rules` #1120

hitenkoku commented Jul 3, 2023 •

edited

Loading

codecov bot commented Jul 3, 2023 •

edited

Loading

fukusuket left a comment •

edited

Loading

fukusuket left a comment •

edited

Loading