Fix #13798, Fix #14128, fix utf8 encoding issues on meterpreter #14888

timwr · 2021-03-13T11:32:18Z

This change fixes #13798 and #14128 which are caused by unicode file/directory names.
It seems the directory listing returned from the meterpreter TLV here: https://github.com/rapid7/metasploit-framework/blob/master/lib/rex/post/meterpreter/extensions/stdapi/fs/dir.rb#L65
is ASCII-8BIT.
This change ensure it's encoded as UTF-8 before passing it to the database, or to Rex::Table (for ls).

Verification

List the steps needed to make sure this thing works

Start msfconsole
Get any meterpreter session (I was able to reproduce on both Android and linux/x64 (mettle))
Ensure you have a database connected (required for # Chinese file name recursive download error #14128)
Verify the following:

meterpreter > mkdir testunicode
Creating directory: testunicode
meterpreter > cd testunicode
meterpreter > mkdir 🔥
Creating directory: 🔥
meterpreter > cd 🔥
meterpreter > edit 💩
meterpreter > ls
Listing: /metasploit-framework/testunicode/🔥
==============================================================

Mode              Size  Type  Last modified              Name
----              ----  ----  -------------              ----
100664/rw-rw-r--  0     fil   2021-03-13 11:13:19 +0000  💩

meterpreter > cd ..
meterpreter > download *
[*] mirroring  : ./🔥 -> /metasploit-framework/🔥
[*] downloading: ./🔥/💩 -> /metasploit-framework/🔥/💩
[*] download   : ./🔥/💩 -> /metasploit-framework/🔥/💩
[*] mirrored   : ./🔥 -> /metasploit-framework/🔥
meterpreter > exit

Before the fix:

meterpreter > mkdir testunicode
Creating directory: testunicode
meterpreter > cd testunicode
meterpreter > mkdir 🔥
Creating directory: 🔥
meterpreter > cd 🔥
meterpreter > edit 💩
meterpreter > ls
[-] Error running command ls: Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8
meterpreter > cd ..
meterpreter > download *
[*] mirroring  : ./🔥 -> /metasploit-framework/🔥
[-] Error running command download: Encoding::CompatibilityError incompatible character encodings: UTF-8 and ASCII-8BIT

…tf-8 filenames

…code directory

gwillcox-r7 · 2021-03-13T21:26:30Z

Thanks for making this PR @timwr, this should hopefully help to fix some long standing UTF-8 related bugs that I was looking into but wasn't sure how to fix appropriately. Appreciate you taking the time to try fix this! 👍

gwillcox-r7 · 2021-03-14T06:20:09Z

Hmm so this is definitely better but I'm not sure its a complete solution:

msf6 exploit(multi/handler) > run

[*] Started bind TCP handler against 172.18.62.83:4444
[*] Sending stage (200262 bytes) to 172.18.62.83
[*] Meterpreter session 1 opened (0.0.0.0:0 -> 172.18.62.83:4444) at 2021-03-14 00:12:35 -0600

meterpreter > mkdir testunicode
Creating directory: testunicode
meterpreter > mkdir 🔥
Creating directory: 🔥
meterpreter >cd ���
[-] Unknown command: cdmcd.
meterpreter > cd ���
[-] stdapi_fs_chdir: Operation failed: 1113
meterpreter > cd 🔥
meterpreter > pwd
C:\Users\normal\OneDrive\Desktop\🔥
meterpreter > shell
Process 5976 created.
Channel 1 created.
Microsoft Windows [Version 10.0.19042.631]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Users\normal\OneDrive\Desktop\??>touch 🔥
touch 🔥
'touch' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\normal\OneDrive\Desktop\??>echo 'A' > 🔥.txt
echo 'A' > 🔥.txt

C:\Users\normal\OneDrive\Desktop\??>cat 🔥.txt
cat 🔥.txt
'cat' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\normal\OneDrive\Desktop\??>type 🔥.txt
type 🔥.txt
'A' 

C:\Users\normal\OneDrive\Desktop\??>ls
ls
'ls' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\normal\OneDrive\Desktop\??>dir
dir
 Volume in drive C has no label.
 Volume Serial Number is 8891-6632

 Directory of C:\Users\normal\OneDrive\Desktop\??

03/13/2021  10:14 PM    <DIR>          .
03/13/2021  10:14 PM    <DIR>          ..
03/13/2021  10:14 PM                 6 🔥.txt
               1 File(s)              6 bytes
               2 Dir(s)  109,683,220,480 bytes free

C:\Users\normal\OneDrive\Desktop\??>exit
exit
meterpreter > download
Usage: download [options] src1 src2 src3 ... destination

Downloads remote files and directories to the local machine.

OPTIONS:

    -a        Enable adaptive download buffer size
    -b <opt>  Set the initial block size for the download
    -c        Resume getting a partially-downloaded file
    -h        Help banner
    -l <opt>  Set the limit of retries (0 unlimits)
    -r        Download recursively
    -t        Timestamp downloaded files

meterpreter > download *
[*] downloading: .\≡ƒöÑ.txt -> /home/gwillcox/git/metasploit-framework/≡ƒöÑ.txt
[*] download   : .\≡ƒöÑ.txt -> /home/gwillcox/git/metasploit-framework/≡ƒöÑ.txt
meterpreter >

So current errors appear to be that the download command on Windows will still have errors, and backspacing will cause issues like the cd �� output shown above. Also the dir command seems to have a mixed output where some of the output is correct Unicode output, and other parts aren't.

timwr · 2021-03-15T20:23:51Z

Oops I didn't test this on Windows, so the reproduction steps were not suited to Windows.
Basically there is a unicode problem in meterpreter > shell on Windows, so when you do:

meterpreter > shell
Process 7144 created.
Channel 3 created.
Microsoft Windows [Version 10.0.18363.1198]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\testunicode>echo a > 🔥.txt
echo a > 🔥.txt

You instead end up creating a file called ƒöÑ.txt

If instead of meterpreter > shell you run meterpreter > edit 🔥.txt then it works as expected on Windows:

meterpreter > mkdir 🔥
Creating directory: 🔥
meterpreter > cd 🔥
meterpreter > edit 💩
meterpreter > ls
Listing: C:\testunicode\🔥
=========================

Mode              Size  Type  Last modified              Name
----              ----  ----  -------------              ----
100666/rw-rw-rw-  6     fil   2021-03-15 20:16:38 +0000  💩

meterpreter > cd ..
meterpreter > download *
[*] mirroring  : .\🔥 -> /metasploit-framework/🔥
[*] downloading: .\🔥\💩 -> /metasploit-framework/🔥/💩
[*] download   : .\🔥\💩 -> /metasploit-framework/🔥/💩
[*] mirrored   : .\🔥 -> /metasploit-framework/🔥

timwr · 2021-03-15T20:26:25Z

I've updated the description, sorry about that!

gwillcox-r7 · 2021-03-15T20:29:43Z

Ah ok no worries so long as we are aware this isn't a complete fix and that further updates will be needed to make a complete solution for Windows, that was the main point I wanted to make clear here. This is a brilliant fix but I don't want people thinking its the golden bullet for all UTF8 issues 😄

Will retest again now!

gwillcox-r7 · 2021-03-15T21:38:42Z

Works like a charm 😄

msf6 > db_status
[*] Connected to remote_data_service: (https://localhost:5443). Connection type: http. Connection name: local-https-data-service.
msf6 > use multi/handler
[*] Using configured payload generic/shell_reverse_tcp
msf6 exploit(multi/handler) > set payload windows/x64/meterpreter/bind_tcp
payload => windows/x64/meterpreter/bind_tcp
msf6 exploit(multi/handler) > set RHOST 172.28.49.78
RHOST => 172.28.49.78
msf6 exploit(multi/handler) > show options

Module options (exploit/multi/handler):

   Name  Current Setting  Required  Description
   ----  ---------------  --------  -----------


Payload options (windows/x64/meterpreter/bind_tcp):

   Name      Current Setting  Required  Description
   ----      ---------------  --------  -----------
   EXITFUNC  process          yes       Exit technique (Accepted: '', seh, thread, process, none)
   LPORT     4444             yes       The listen port
   RHOST     172.28.49.78     no        The target address


Exploit target:

   Id  Name
   --  ----
   0   Wildcard Target


msf6 exploit(multi/handler) > run

[*] Started bind TCP handler against 172.28.49.78:4444
[*] Sending stage (200262 bytes) to 172.28.49.78
[*] Meterpreter session 1 opened (0.0.0.0:0 -> 172.28.49.78:4444) at 2021-03-15 16:33:42 -0500

meterpreter > pwd
C:\Users\normal\OneDrive\Desktop
meterpreter > mkdir testunicode
Creating directory: testunicode
[-] stdapi_fs_mkdir: Operation failed: Cannot create a file when that file already exists.
meterpreter > cd testunicode
meterpreter > ls
No entries exist in C:\Users\normal\OneDrive\Desktop\testunicode
meterpreter > dir
No entries exist in C:\Users\normal\OneDrive\Desktop\testunicode
meterpreter > mkdir 🔥
Creating directory: 🔥
meterpreter > cd 🔥
meterpreter > edit ⚡
meterpreter > ls
Listing: C:\Users\normal\OneDrive\Desktop\testunicode\🔥
=======================================================

Mode              Size  Type  Last modified              Name
----              ----  ----  -------------              ----
100666/rw-rw-rw-  15    fil   2021-03-15 16:36:14 -0500  ⚡

meterpreter > cat ⚡
testing right?
meterpreter > cd ..
meterpreter > download *
[*] mirroring  : .\🔥 -> /home/gwillcox/git/metasploit-framework/🔥
[*] downloading: .\🔥\⚡ -> /home/gwillcox/git/metasploit-framework/🔥/⚡
[*] download   : .\🔥\⚡ -> /home/gwillcox/git/metasploit-framework/🔥/⚡
[*] mirrored   : .\🔥 -> /home/gwillcox/git/metasploit-framework/🔥
meterpreter > back
[-] Unknown command: back.
meterpreter > background
[*] Backgrounding session 1...
msf6 exploit(multi/handler) > cat /home/gwillcox/git/metasploit-framework/🔥/⚡
[*] exec: cat /home/gwillcox/git/metasploit-framework/🔥/⚡

testing right?
msf6 exploit(multi/handler) >

gwillcox-r7 · 2021-03-15T21:41:38Z

Confirmation #13798 is fixed:

meterpreter > mkdir عکس
Creating directory: عکس
meterpreter > cd عکس
meterpreter > pwd
C:\Users\normal\OneDrive\Desktop\testunicode\عکس
meterpreter >

gwillcox-r7 · 2021-03-15T21:44:38Z

Confirmation #14128 is fixed:

meterpreter > mkdir 贏得
Creating directory: 贏得
meterpreter > cd 贏得
meterpreter > edit 贏得贏得.txt
meterpreter > cd ..
meterpreter > download *
[*] mirroring  : .\贏得 -> /home/gwillcox/git/metasploit-framework/贏得
[*] downloading: .\贏得\贏得贏得.txt -> /home/gwillcox/git/metasploit-framework/贏得/贏得贏得.txt
[*] download   : .\贏得\贏得贏得.txt -> /home/gwillcox/git/metasploit-framework/贏得/贏得贏得.txt
[*] mirrored   : .\贏得 -> /home/gwillcox/git/metasploit-framework/贏得
meterpreter > background
[*] Backgrounding session 1...
msf6 exploit(multi/handler) > cat /home/gwillcox/git/metasploit-framework/贏得/贏得贏得.txt
[*] exec: cat /home/gwillcox/git/metasploit-framework/贏得/贏得贏得.txt

foobar
msf6 exploit(multi/handler) >

gwillcox-r7 · 2021-03-15T21:45:16Z

Looks good @timwr will land this now, thanks for all your work on this, much appreciated!

gwillcox-r7 · 2021-03-15T21:51:15Z

Release Notes

Fixed two Unicode related bugs preventing recursive download of files or folders containing UTF8 characters, or otherwise open or interact with these files, via Meterpreter. This has now been addressed for common commands such as edit, download and cd.

timwr · 2021-03-15T21:51:38Z

Thanks for testing and landing @gwillcox-r7 !!

gwillcox-r7 · 2021-03-22T20:41:45Z

Hey @timwr quick update on this but tried running this command today for the demo meeting and ran into another error:

meterpreter > cd 🔥
meterpreter > edit ⚡
meterpreter > cat ⚡
who is this?
meterpreter > pwd
C:\Users\test\Desktop\🔥
meterpreter > cd ..
meterpreter > download 🔥
[-] Error running command download: Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8

timwr · 2021-03-22T21:19:55Z

Urg I feel like this bug might be the gift that keeps on giving. I can put in a quick fix for that particular case but I wonder if there are others hiding too.

gwillcox-r7 · 2021-03-22T21:29:36Z

All good, I pointed it out in the demo meeting just to make people aware that we still have other areas that need improvement. Well aware though that we have lots of different areas in the framework that need fixing (just try UTF-8 with the creds database and watch it topple over).

timwr added 2 commits March 12, 2021 19:51

Fix rapid7#13798, Fix rapid7#14128, fix meterpreter > download * on u…

1a77047

…tf-8 filenames

Fix rapid7#13798, Fix rapid7#14128, fix meterpreter > ls within a uni…

54c07c3

…code directory

gwillcox-r7 added bug library labels Mar 13, 2021

gwillcox-r7 self-assigned this Mar 13, 2021

gwillcox-r7 merged commit 50ef32c into rapid7:master Mar 15, 2021

gwillcox-r7 added the rn-fix release notes fix label Mar 15, 2021

timwr deleted the fix_download_glob_encoding branch March 15, 2021 21:51

timwr added a commit to timwr/metasploit-framework that referenced this pull request Mar 22, 2021

Fix rapid7#14888, fix downloading a utf-8 directory directly

afecde2

timwr mentioned this pull request Mar 22, 2021

Fix #14888, fix downloading a utf-8 directory directly #14934

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #13798, Fix #14128, fix utf8 encoding issues on meterpreter #14888

Fix #13798, Fix #14128, fix utf8 encoding issues on meterpreter #14888

timwr commented Mar 13, 2021 •

edited by gwillcox-r7

Loading

gwillcox-r7 commented Mar 13, 2021

gwillcox-r7 commented Mar 14, 2021

timwr commented Mar 15, 2021

timwr commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021 •

edited by pbarry-r7

Loading

timwr commented Mar 15, 2021

gwillcox-r7 commented Mar 22, 2021

timwr commented Mar 22, 2021

gwillcox-r7 commented Mar 22, 2021

Fix #13798, Fix #14128, fix utf8 encoding issues on meterpreter #14888

Fix #13798, Fix #14128, fix utf8 encoding issues on meterpreter #14888

Conversation

timwr commented Mar 13, 2021 • edited by gwillcox-r7 Loading

Verification

Before the fix:

gwillcox-r7 commented Mar 13, 2021

gwillcox-r7 commented Mar 14, 2021

timwr commented Mar 15, 2021

timwr commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021

gwillcox-r7 commented Mar 15, 2021 • edited by pbarry-r7 Loading

Release Notes

timwr commented Mar 15, 2021

gwillcox-r7 commented Mar 22, 2021

timwr commented Mar 22, 2021

gwillcox-r7 commented Mar 22, 2021

timwr commented Mar 13, 2021 •

edited by gwillcox-r7

Loading

gwillcox-r7 commented Mar 15, 2021 •

edited by pbarry-r7

Loading