Unicode issues (outside of Needle, in paramiko)

[tghosth]

Issue

Hi @marco-lancini, there is some sort of issue reading special characters in paramiko. I came across it again after I added from __future__ import unicode_literals to the top of each file in the project.

Currently, in the develop branch of needle, you don't see the issue below because it throws the error separately described here before it even gets to this issue.

This appears to be an upstream issue in paramiko described here and unfortunately there doesn't seem to be a suggested fix right now. The problem occurs with filenames with non UTF-8 characters in.

Expected behaviour

needle should pull the IPA from the device.

Actual behaviour

An error is thrown apparently by paramiko related to special characters

Steps to reproduce

1. Install this application: https://itunes.apple.com/us/app/discount-business+dysqwnt/id444480867?mt=8
2. Use the this branch of needle. As can be seen here, the only difference is that I have added from __future__ import unicode_literals to the top of each header.
3. Try running the binary/pull_binary module
4. needle throws the error below.

needle error logs

root@kali:/mnt/hgfs/Work# python needle/needle/needle.py

             __   _ _______ _______ ______         _______
             | \  | |______ |______ |     \ |      |______
             |  \_| |______ |______ |_____/ |_____ |______
        
                   Needle v0.1.1 [mwr.to/needle]                  
  [MWR InfoSecurity (@MWRLabs) - Marco Lancini (@LanciniMarco)]   

[needle] > set IP 10.0.0.7
IP => 10.0.0.7
[needle] > set PORT 22
PORT => 22
[needle] > set VERBOSE True
VERBOSE => True
[needle] > set DEBUG True
DEBUG => True
[needle] > set APP il.co.discountbank.discountbankSME
APP => il.co.discountbank.discountbankSME
[needle] > use binary/pull_ipa
[needle][pull_ipa] > run
[D] Setup local output folder: /root/.needle/output
[?] Attention! The folder chosen to store local output is not empty: /root/.needle/output
[?] Do you want to back it up first?
[?] Y: the content will be archived in a different location, then the folder will be emptied
[?] N: no action will be taken (destination files might be overwritten in case of filename clash)
[y/n]: 
[*] Checking connection with device...
[V] Connection not present, creating a new instance
[V] Setting up SSH connection...
[+] Connected to: 10.0.0.7
[V] Creating temp folder: /var/root/needle/
[D] [REMOTE CMD] Remote Command: if [ -d /var/root/needle/ ]; then echo "yes"; else echo "no" ; fi
[+] Target app: il.co.discountbank.discountbankSME
[*] Retrieving app's metadata...
[D] [REMOTE CMD] Remote Command: if [ -f /var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist ]; then echo "yes"; else echo "no" ; fi
[V] Refreshing list of installed apps...
[D] [REMOTE CMD] Remote Command: /bin/su mobile -c /usr/bin/uicache
[D] Copying the plist to temp: /var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist -> /root/.needle/tmp/plist
[*] Pulling: /var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist -> /root/.needle/tmp/plist
[D] Downloading: "/var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist" -> /root/.needle/tmp/plist
[D] [LOCAL CMD] Local Command: sshpass -p "alpine" scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 22 root@10.0.0.7:"/var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist" /root/.needle/tmp/plist
[D] Copying the plist to temp: '/private/var/mobile/Containers/Bundle/Application/B6EAD8C2-6A5C-495D-86EA-3E9A1A0FA58C/דיסקונט עסקים.app/Info.plist' -> /root/.needle/tmp/plist
[*] Pulling: '/private/var/mobile/Containers/Bundle/Application/B6EAD8C2-6A5C-495D-86EA-3E9A1A0FA58C/דיסקונט עסקים.app/Info.plist' -> /root/.needle/tmp/plist
[D] Downloading: "'/private/var/mobile/Containers/Bundle/Application/B6EAD8C2-6A5C-495D-86EA-3E9A1A0FA58C/דיסקונט עסקים.app/Info.plist'" -> /root/.needle/tmp/plist
[D] [LOCAL CMD] Local Command: sshpass -p "alpine" scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 22 root@10.0.0.7:"'/private/var/mobile/Containers/Bundle/Application/B6EAD8C2-6A5C-495D-86EA-3E9A1A0FA58C/דיסקונט עסקים.app/Info.plist'" /root/.needle/tmp/plist
[D] [REMOTE CMD] Remote Command: lipo -info '/private/var/mobile/Containers/Bundle/Application/B6EAD8C2-6A5C-495D-86EA-3E9A1A0FA58C/דיסקונט עסקים.app/דיסקונט עסקים'
[D] [REMOTE CMD] Remote Command: if [ -d '/private/var/mobile/Containers/Bundle/Application/B6EAD8C2-6A5C-495D-86EA-3E9A1A0FA58C/דיסקונט עסקים.app/PlugIns' ]; then echo "yes"; else echo "no" ; fi
[D] No Plugins found
[*] Decrypting the binary...
[D] [REMOTE CMD] Remote Command: Clutch2 -d il.co.discountbank.discountbankSME 2>&1
[D] [REMOTE CMD] Remote Command: cp '/private/var/mobile/Documents/Dumped/il.co.discountbank.discountbankSME-iOS6.0-(Clutch-2.0.4).ipa' /var/root/needle/decrypted.ipa
[D] [REMOTE CMD] Remote Command: if [ -f '/private/var/mobile/Documents/Dumped/il.co.discountbank.discountbankSME-iOS6.0-(Clutch-2.0.4).ipa' ]; then echo "yes"; else echo "no" ; fi
[D] [REMOTE CMD] Remote Command: rm '/private/var/mobile/Documents/Dumped/il.co.discountbank.discountbankSME-iOS6.0-(Clutch-2.0.4).ipa' 2> /dev/null
[V] Decrypted IPA stored at: /var/root/needle/decrypted.ipa
[D] [REMOTE CMD] Remote Command: if [ -d /var/root/needle/Payload ]; then echo "yes"; else echo "no" ; fi
[D] [REMOTE CMD] Remote Command: if [ -d /var/root/needle/Payload ]; then echo "yes"; else echo "no" ; fi
[D] [REMOTE CMD] Remote Command: rm -rf /var/root/needle/Payload 2> /dev/null
[D] [REMOTE CMD] Remote Command: if [ -f /var/root/needle/iTunesArtwork ]; then echo "yes"; else echo "no" ; fi
[D] [REMOTE CMD] Remote Command: if [ -f /var/root/needle/iTunesArtwork ]; then echo "yes"; else echo "no" ; fi
[D] [REMOTE CMD] Remote Command: rm /var/root/needle/iTunesArtwork 2> /dev/null
[*] Unpacking the IPA...
[D] [REMOTE CMD] Remote Command: unzip /var/root/needle/decrypted.ipa -d /var/root/needle/
------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/hgfs/Work/needle/needle/core/framework/module.py", line 117, in do_run
    self.module_run()
  File "/mnt/hgfs/Work/needle/needle/modules/binary/pull_ipa.py", line 38, in module_run
    self.fname_binary = self.device.app.decrypt(self.APP_METADATA)
  File "/mnt/hgfs/Work/needle/needle/core/device/app.py", line 220, in decrypt
    fname_binary = self.unpack_ipa(app_metadata, fname_decrypted)
  File "/mnt/hgfs/Work/needle/needle/core/device/app.py", line 238, in unpack_ipa
    out = self._device.remote_op.command_blocking(cmd)
  File "/mnt/hgfs/Work/needle/needle/core/device/remote_operations.py", line 95, in command_blocking
    out, err = self._device._exec_command_ssh(cmd, internal)
  File "/mnt/hgfs/Work/needle/needle/core/device/device.py", line 128, in _exec_command_ssh
    out = stdout.readlines()
  File "/usr/lib/python2.7/dist-packages/paramiko/file.py", line 327, in readlines
    line = self.readline()
  File "/usr/lib/python2.7/dist-packages/paramiko/file.py", line 312, in readline
    return line if self._flags & self.FLAG_BINARY else u(line)
  File "/usr/lib/python2.7/dist-packages/paramiko/py3compat.py", line 53, in u
    return s.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xce in position 38: invalid continuation byte
------------------------------------------------------------
[!] UnicodeDecodeError: 'utf8' codec can't decode byte 0xce in position 38: invalid continuation byte.
[needle][pull_ipa] > 

Environment

Workstation Operating System

Kali Linux 2016.2

Python Version

2.7.12+

Python Packages (pip freeze)

Device iOS Version

iPad Mini 4 running iOS 9.2.1

[tghosth]

Hi @marco-lancini,

I spent quite some time this morning trying to find a way of dealing with this. I experimented with a solution where we try and individually catch unicode issues on a file by file basis, see here for example.

Unfortunately, right now it seems like anything we do will be an ugly hack that will mean things being missed. For example, in the example in the previous comment, the IPA name has non-ASCII characters and it appears that paramiko can just not cope with that and errors on everything.

Any thoughts would be greatly appreciated as I am not sure where to go with this!!!!