Deleted return characters in output of search command #1018

hitenkoku · 2023-04-27T16:13:24Z

What Changed

Deleted return characters in output(CSV/display) of search command

Evidence

Please describe the functionality gained by merging this pull-request and the evidence that the bug has been fixed.

main branch output
PS > ./main.exe search -d ..\hayabusa-sample-evtx\ --keywords "null" -o main.csv
...
Total event log files: 584
Total file size: 137.1 MB

584 / 584 [==========================================================================] 100.00 %

Total findings: 44144

Elapsed time: 00:00:09.375

This PR

PS > ./1003.exe search -d ..\hayabusa-sample-evtx\ --keywords "null" -o 1003.csv

...
Total event log files: 584
Total file size: 137.1 MB

584 / 584 [==========================================================================] 100.00 %

Total findings: 44144


Elapsed time: 00:00:09.778

File Contents

> cat main.csv

...
2013-10-23T16:18:08.000000Z,37L4247D28-05,App,8212,71,-,Binary: 2D20436F64653A20575254575254494330303030303832332D2043616C6C3A20575254575254494330303030303737342D205049443A202030303030313532382D205449443A202030303030313538302D20434D443A2020433A5C57696E646F77735C73797374656D33325C76737376632E6578652020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820 ¦ Data: Operation:    Initializing Writer  Context:    Writer Class Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}    Writer Name: Registry Writer ¦ Data: Registry Writer ¦ Data: {afbab4a2-367d-4d15-a586-71dbb18f8485},..\hayabusa-sample-evtx\DeepBlueCLI\many-events-application.evtx
2014-11-25T22:38:30.979875Z,IE8Win7,Sec,4672,4132,Admin logon,PrivilegeList: SeAssignPrimaryTokenPrivilege     SeTcbPrivilege     SeSecurityPrivilege     SeTakeOwnershipPrivilege     SeLoadDriverPrivilege     SeBackupPrivilege     SeRestorePrivilege     SeDebugPrivilege     SeAuditPrivilege     SeSystemEnvironmentPrivilege     SeImpersonatePrivilege ¦ SubjectDomainName: NT AUTHORITY ¦ SubjectLogonId: 0x3e7 ¦ SubjectUserName: SYSTEM ¦ SubjectUserSid: S-1-5-18,..\hayabusa-sample-evtx\DeepBlueCLI\many-events-security.evtx
2016-08-20T16:33:55.819679Z,IE10Win7,Sec,4688,7095,Process created,"CommandLine: ""C:\Program Files\Google\Update\GoogleUpdate.exe"" /svc ¦ NewProcessId: 0x984 ¦ NewProcessName: C:\Program Files\Google\Update\GoogleUpdate.exe ¦ ProcessId: 0x1d4 ¦ SubjectDomainName: WORKGROUP ¦ SubjectLogonId: 0x3e7 ¦ SubjectUserName: IE10WIN7$ ¦ SubjectUserSid: S-1-5-18 ¦ TokenElevationType: %%1936",..\hayabusa-sample-evtx\DeepBlueCLI\many-events-security.evtx
2019-05-22T04:02:11.307031Z,IEWIN7,Sysmon,1,839,Process Creation,"CommandLine: cmd.exe ¦ Company: Microsoft Corporation ¦ CurrentDirectory: C:\Users\IEUser\Desktop\ ¦ Description: Windows Command Processor ¦ FileVersion: 6.1.7601.17514 (win7sp1_rtm.101119-1850) ¦ Hashes: SHA1=EE8CBF12D87C4D388F09B4F69BED2E91682920B5,MD5=AD7B9C14083B52BC532FBA5948342B98,SHA256=17F746D82695FA9B35493B41859D39D786D32B23A9D2E00F4011DEC7A02402AE,IMPHASH=CEEFB55F764020CC5C5F8F23349AB163 ¦ Image: C:\Windows\System32\cmd.exe ¦ IntegrityLevel: High ¦ LogonGuid: 365ABB72-C32E-5CE4-0000-00205DF00000 ¦ LogonId: 0xf05d ¦ ParentCommandLine: ""C:\Program Files\Internet Explorer\iexplore.exe"" SCODEF:1600 CREDAT:275470 /prefetch:2 ¦ ParentImage: C:\Program Files\Internet Explorer\iexplore.exe ¦ ParentProcessGuid: 365ABB72-C9C1-5CE4-0000-00100B222E00 ¦ ParentProcessId: 3156 ¦ ProcessGuid: 365ABB72-C9C3-5CE4-0000-00101F422E00 ¦ ProcessId: 2888 ¦ Product: Microsoft® Windows® Operating System ¦ RuleName:  ¦ TerminalSessionId: 1 ¦ User: IEWIN7\IEUser ¦ UtcTime: 2019-05-22 04:02:11.287",..\hayabusa-sample-evtx\EVTX-ATTACK-SAMPLES\Execution\exec_driveby_cve-2018-15982_sysmon_1_10.evtx

...

> cat 1003.csv
...
2013-10-23T16:18:08.000000Z,37L4247D28-05,App,8212,71,-,Binary: 2D20436F64653A20575254575254494330303030303832332D2043616C6C3A20575254575254494330303030303737342D205049443A202030303030313532382D205449443A202030303030313538302D20434D443A2020433A5C57696E646F77735C73797374656D33325C76737376632E6578652020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820 ¦ Data: Operation: Initializing Writer Context: Writer Class Id: {afbab4a2-367d-4d15-a586-71dbb18f8485} Writer Name: Registry Writer ¦ Data: Registry Writer ¦ Data: {afbab4a2-367d-4d15-a586-71dbb18f8485},..\hayabusa-sample-evtx\DeepBlueCLI\many-events-application.evtx
2014-11-25T22:38:30.979875Z,IE8Win7,Sec,4672,4132,Admin logon,PrivilegeList: SeAssignPrimaryTokenPrivilege SeTcbPrivilege SeSecurityPrivilege SeTakeOwnershipPrivilege SeLoadDriverPrivilege SeBackupPrivilege SeRestorePrivilege SeDebugPrivilege SeAuditPrivilege SeSystemEnvironmentPrivilege SeImpersonatePrivilege ¦ SubjectDomainName: NT AUTHORITY ¦ SubjectLogonId: 0x3e7 ¦ SubjectUserName: SYSTEM ¦ SubjectUserSid: S-1-5-18,..\hayabusa-sample-evtx\DeepBlueCLI\many-events-security.evtx
2016-08-20T16:33:55.819679Z,IE10Win7,Sec,4688,7095,Process created,"CommandLine: ""C:\Program Files\Google\Update\GoogleUpdate.exe"" /svc ¦ NewProcessId: 0x984 ¦ NewProcessName: C:\Program Files\Google\Update\GoogleUpdate.exe ¦ ProcessId: 0x1d4 ¦ SubjectDomainName: WORKGROUP ¦ SubjectLogonId: 0x3e7 ¦ SubjectUserName: IE10WIN7$ ¦ SubjectUserSid: S-1-5-18 ¦ TokenElevationType: %%1936",..\hayabusa-sample-evtx\DeepBlueCLI\many-events-security.evtx
2019-05-22T04:02:11.307031Z,IEWIN7,Sysmon,1,839,Process Creation,"CommandLine: cmd.exe ¦ Company: Microsoft Corporation ¦ CurrentDirectory: C:\Users\IEUser\Desktop\ ¦ Description: Windows Command Processor ¦ FileVersion: 6.1.7601.17514 (win7sp1_rtm.101119-1850) ¦ Hashes: SHA1=EE8CBF12D87C4D388F09B4F69BED2E91682920B5,MD5=AD7B9C14083B52BC532FBA5948342B98,SHA256=17F746D82695FA9B35493B41859D39D786D32B23A9D2E00F4011DEC7A02402AE,IMPHASH=CEEFB55F764020CC5C5F8F23349AB163 ¦ Image: C:\Windows\System32\cmd.exe ¦ IntegrityLevel: High ¦ LogonGuid: 365ABB72-C32E-5CE4-0000-00205DF00000 ¦ LogonId: 0xf05d ¦ ParentCommandLine: ""C:\Program Files\Internet Explorer\iexplore.exe"" SCODEF:1600 CREDAT:275470 /prefetch:2 ¦ ParentImage: C:\Program Files\Internet Explorer\iexplore.exe ¦ ParentProcessGuid: 365ABB72-C9C1-5CE4-0000-00100B222E00 ¦ ParentProcessId: 3156 ¦ ProcessGuid: 365ABB72-C9C3-5CE4-0000-00101F422E00 ¦ ProcessId: 2888 ¦ Product: Microsoft® Windows® Operating System ¦ RuleName: ¦ TerminalSessionId: 1 ¦ User: IEWIN7\IEUser ¦ UtcTime: 2019-05-22 04:02:11.287",..\hayabusa-sample-evtx\EVTX-ATTACK-SAMPLES\Execution\exec_driveby_cve-2018-15982_sysmon_1_10.evtx
...

I would appreciate it if you could review.

codecov · 2023-04-27T16:16:52Z

Codecov Report

Patch coverage has no change and project coverage change: -0.03 ⚠️

Comparison is base (9ab1d35) 73.89% compared to head (0ebdba9) 73.87%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1018      +/-   ##
==========================================
- Coverage   73.89%   73.87%   -0.03%     
==========================================
  Files          25       24       -1     
  Lines       17792    17784       -8     
==========================================
- Hits        13148    13138      -10     
- Misses       4644     4646       +2

Impacted Files	Coverage Δ
src/afterfact.rs	`45.39% <ø> (-0.24%)`	⬇️
src/detections/configs.rs	`52.85% <ø> (ø)`
src/timeline/search.rs	`4.61% <0.00%> (-0.03%)`	⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

YamatoSecurity · 2023-04-27T21:36:46Z

@hitenkoku ありがとうございます！
文字を一個ずつ空白にすると、文字列の間のスペースが大きくなるので、以下のregexのようにできますか？

>>> import re
>>> s = 'Copyright ©\n\t\t\t\n\t\t\t2019\n\t\t\tApple Inc. All rights reserved.'
>>> s = re.sub("\s+", " ", s)
>>> s
'Copyright © 2019 Apple Inc. All rights reserved.'

参考: https://stackoverflow.com/questions/55123626/regex-python-replace-any-combination-of-line-breaks-tabs-spaces-by-single-s

csv-timelineの方も上記のようにしたいのですが、csv-timelineは別のissueにしましょうか？

hitenkoku · 2023-04-28T04:04:39Z

@YamatoSecurity 確認ありがとうございます。searchの方は連続した空白は一つの空白となるように変更してみていますはずですが、どこかで問題ありましたでしょうか？
csv-timelineについては確かに、そのままになっていたかもしれないのでここで確認をして、修正が必要であれば一旦別のbranchで対応させてもらいますね

hitenkoku · 2023-04-28T07:16:09Z

csv-timelineでは行っていなかったようなのでissue新しく切ってそちらで管理させていただければと思います

YamatoSecurity · 2023-04-28T08:49:02Z

@hitenkoku あ、なるほど。
例えば、./target/release/hayabusa search -d ../hayabusa-sample-evtx -k "Access"を実行したら、

AccessListフィールドのデータの間にスペースが６個入っています。可能であれば、それを１個にしたいです。

元のAccessListフィールドを調べたら、スペースが一個になっているように見えますが、Hayabusaの内部では間にタブ文字が入ってたりしますか？（確か、間にタブや改行文字が入っていた気がしました）
例: <Data Name="AccessList">%%1541 %%4416 %%4423</Data>

hitenkoku · 2023-04-28T09:15:21Z

ご指摘ありがとうございます。今回対処した範囲だとCSVの出力のみ対応していました……画面出力についてもすぐに対応します

…ommand #1003

hitenkoku · 2023-04-28T09:28:18Z

@YamatoSecurity

ご指摘いただき誠にありがとうございます。頂いたコマンドを元に d1a957c で修正して画面表示でも複数の空白を1つの空白にまとめる形で対応完了しました！

YamatoSecurity

@hitenkoku そういうことだったんですね。標準出力を確認したら、LGTM!
良いと思います。ありがとうございました！
ChangelogとHayabusaバージョンを更新しておきました。

hitenkoku · 2023-04-29T02:58:44Z

ありがとうございます！修正いただきありがとうございます。修正いただいた内容で問題ないと思います。それではマージさせていただきます！

hitenkoku added 5 commits April 28, 2023 00:29

docs(CHANGELOG): updated changelog #1004

4641bc3

docs(CHANGELOG): fixed typo

fca9c2a

refactor(afterfact): removed unnecessary comment

ee59f43

feat(search): delete return characters in output #1003

e55a5ee

docs(CHANGELOG): updated changelog #1003

d1d71fe

hitenkoku added the enhancement New feature or request label Apr 27, 2023

hitenkoku requested review from itiB, fukusuket and YamatoSecurity April 27, 2023 16:13

hitenkoku self-assigned this Apr 27, 2023

hitenkoku linked an issue Apr 27, 2023 that may be closed by this pull request

Search enhancement: delete return characters in output #1003

Closed

hitenkoku mentioned this pull request Apr 28, 2023

Deleted return characters in output of csv-timeline command #1019

Closed

feat(search): deleted return characters in display output of search c…

d1a957c

…ommand #1003

YamatoSecurity added 4 commits April 29, 2023 09:32

update changelog

e116549

update cargo

bb26f9f

update haybusa version number to dev

ed78c7d

update rules link

0ebdba9

YamatoSecurity approved these changes Apr 29, 2023

View reviewed changes

hitenkoku merged commit d1f5aa5 into main Apr 29, 2023

hitenkoku deleted the 1003-search-enhancement-delete-return-characters-in-output branch April 29, 2023 02:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deleted return characters in output of search command #1018

Deleted return characters in output of search command #1018

hitenkoku commented Apr 27, 2023 •

edited

Loading

codecov bot commented Apr 27, 2023 •

edited

Loading

YamatoSecurity commented Apr 27, 2023

hitenkoku commented Apr 28, 2023

hitenkoku commented Apr 28, 2023 •

edited

Loading

YamatoSecurity commented Apr 28, 2023

hitenkoku commented Apr 28, 2023 •

edited

Loading

hitenkoku commented Apr 28, 2023

YamatoSecurity left a comment

hitenkoku commented Apr 29, 2023 •

edited

Loading

Deleted return characters in output of search command #1018

Deleted return characters in output of search command #1018

Conversation

hitenkoku commented Apr 27, 2023 • edited Loading

What Changed

Evidence

File Contents

codecov bot commented Apr 27, 2023 • edited Loading

Codecov Report

YamatoSecurity commented Apr 27, 2023

hitenkoku commented Apr 28, 2023

hitenkoku commented Apr 28, 2023 • edited Loading

YamatoSecurity commented Apr 28, 2023

hitenkoku commented Apr 28, 2023 • edited Loading

hitenkoku commented Apr 28, 2023

YamatoSecurity left a comment

Choose a reason for hiding this comment

hitenkoku commented Apr 29, 2023 • edited Loading

hitenkoku commented Apr 27, 2023 •

edited

Loading

codecov bot commented Apr 27, 2023 •

edited

Loading

hitenkoku commented Apr 28, 2023 •

edited

Loading

hitenkoku commented Apr 28, 2023 •

edited

Loading

hitenkoku commented Apr 29, 2023 •

edited

Loading