-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Non-ASCII characters can't be output in CSV. #81
Labels
Bug
Something isn't working
Comments
14 tasks
T0pCyber
added a commit
that referenced
this issue
Jan 21, 2022
Fix encoding of outputs from the Export-Csv cmdlet #81
T0pCyber
added a commit
that referenced
this issue
Feb 17, 2022
T0pCyber
added a commit
that referenced
this issue
Feb 17, 2022
…_Csv Revert "Fix encoding of outputs from the Export-Csv cmdlet #81"
This was referenced Feb 21, 2022
T0pCyber
added a commit
that referenced
this issue
Apr 7, 2022
…Csv_UTF8BOM Fix encoding of outputs from the Export-Csv cmdlet with UTF8(BOM). #81
Merged
15 tasks
Updates in 3.0.0 and merged to master. Updated in Gallery. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Non-ASCII characters can't output in CSV.
HAWK outputs CSV files with only ASCII characters.
All non-ASCII characters such as Chinese, Japanese, Russian are converted to symbols of
?
.The reason for this is using Export-Csv cmdlets without specifying encoding in the Out-MultipleFileType.ps1. The cmdlet uses the ASCII encoding.
According to the Microsoft Docs and my research, each cmdlet uses the following encodings by default.
-Append
)Currently, the HAWK's output encoding of
xml
is UTF-16LE(BOM), andtxt
is UTF-16LE(BOM), andcsv
is ASCII.So, HAWK's output of
xml
andtxt
files can contain non-ASCII characters, butcsv
files can't.To fix it
It could be resolve by using Export-Csv cmdlet with encoding option
-Encoding <encoding name>
.I recommend using UTF-16LE(BOM) with specifying
-Encoding Unicode
.(However, UTF-8(BOM) may be better to work with Excel.)
The pros and cons of encodings are as below.
From Text/CSV Import
button can correctly import such CSV files. So it doesn't seem to be a big deal.To Reproduce
Steps to reproduce the behavior:
テストabc1
.Get-HawkUserInvestigation <your mail address>
Exchange_Mailbox_Audit_<username>.csv
.The CSV file will be ASCII encoding and contain some
?
symbols in the log record of deleting operation.All non-ASCII characters are replaced with
?
symbols.Expected (better) behavior
HAWK can output CSV files containing non-ASCII characters, such as
テストabc1
.Screenshots
N/A
File (please complete the following information):
Additional context
N/A
The text was updated successfully, but these errors were encountered: