Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test whether appstraction and cyanoacrylate work on Windows (under WSL) #25

Closed
baltpeter opened this issue Apr 9, 2023 · 39 comments
Closed
Assignees

Comments

@baltpeter
Copy link
Member

We are pretty sure that CA won't work on Windows due to the IPC mechanism used:

https://github.com/tweaselORG/cyanoacrylate/blob/b59010c79fa4939307b81bc58db0f52147e285fc/README.md?plain=1#L14

But we haven't actually verified that. And maybe it does work under WSL at least…

@baltpeter baltpeter self-assigned this Apr 10, 2023
@baltpeter baltpeter changed the title Test whether appstraction and cyanoacrylate work under Windows (under WSL) Test whether appstraction and cyanoacrylate work on Windows (under WSL) Apr 10, 2023
@baltpeter
Copy link
Member Author

OK, I can confirm that an MRE for the IPC communication doesn't work on Windows.

index.js:

import { execa } from 'execa';

(async () => {
    const proc = execa('python', ['script.py'], { stdio: ['pipe', 'pipe', 'pipe', 'ipc'] });

    proc.on('message', (message) => {
        console.log(111, message);
    });

    const res = await proc;
    console.log(222, res);
})();

script.py:

import os
import json

ipcPipeFd = 3

res = os.write(ipcPipeFd, bytes(json.dumps("hi") + '\n', 'utf8'))

print(f"res: {res}")

This will never print 111 hi on Windows, even though it does on Linux.

@baltpeter
Copy link
Member Author

Curiously, the Python process does print res: 5 to stdout though, which is the number of bytes written, indicating that it did write somewhere:

222 {
  command: 'python script.py',
  escapedCommand: 'python script.py',
  exitCode: 0,
  stdout: 'res: 5',
  stderr: '',
  all: undefined,
  failed: false,
  timedOut: false,
  isCanceled: false,
  killed: false
}

If I instead change ipcPipeFd to 4, the Python process errors with OSError: [Errno 9] Bad file descriptor, further suggesting that there is actually something listening at fd 3.

(Changing it to 1 or 2 prints to stdout or stderr, respectively.)

@baltpeter
Copy link
Member Author

I got a little closer with https://www.npmjs.com/package/@kldzj/named-pipes, which is an abstraction over named pipes for both Linux and Windows. The script is mostly adapted from the README.

index-pipe.ts:

import { createNamedPipe } from '@kldzj/named-pipes';
import { execa } from 'execa';

const pipe = createNamedPipe();
console.log('Path to socket:', pipe.path);

const sender = pipe.createSender();
const receiver = pipe.createReceiver();

await sender.connect();
await receiver.connect();

receiver.on('data', (c) => console.log(c.toString()));
sender.once('connected', () => {
    execa('python', ['script-pipe.py', pipe.path]).then(() => pipe.destroy());
});

script-pipe.py:

import json
import sys

pipe_path = sys.argv[1]

with open(pipe_path, 'wb', buffering=0) as pipe:
    message = json.dumps("hi").encode('utf-8')
    pipe.write(message)

This still works fine on Linux and it correctly creates a pipe on Windows, but the Python code for writing to it is wrong.

@baltpeter
Copy link
Member Author

A few things I tried, unsuccessfully:

@zner0L
Copy link

zner0L commented Apr 11, 2023

Seems like you did all the work I did again. When I tried this, I came to the conclusion, that the easiest and most elegant solution would be to just use libuv pipes in python as well, since this is what nodejs uses to create the pipes. This is also why you are able to write to the file descriptor. Somehow libuv creates a named pipe at this file descriptor in Windows, but I don't really understand how they write to it.

There is already python library for libuv: https://github.com/saghul/pyuv, but the maintainer doesn't really seem to care about it anymore saghul/pyuv#276, so we can not really use it anymore sadly. But the part for the pipe is pretty simple and should be easy to reuse from pyuv.

@zner0L
Copy link

zner0L commented Apr 11, 2023

Are the pipes also failing on WSL, or did you only test them on pure Windows? I think they are not our only problem when porting to Windows. E.g. our paths are all formatted for UNIX operating systems.

@baltpeter
Copy link
Member Author

baltpeter commented Apr 11, 2023

I only tested Windows, not WSL.

E.g. our paths are all formatted for UNIX operating systems.

I don't think that should be a problem:

image

@baltpeter
Copy link
Member Author

baltpeter commented Apr 11, 2023

Also works fine in Python:

image

@zner0L zner0L assigned zner0L and unassigned baltpeter Apr 11, 2023
@zner0L
Copy link

zner0L commented Apr 12, 2023

OK, so I underestimated the amount of work necessary to get libuv binding working in python. Since I don't want to write another https://github.com/saghul/pyuv, we need to make a decision. pyuv fails to build against python 3.10, but that was fixed in saghul/pyuv#275. However, this only fixes it for 3.10 for now, it breaks again on 3.11. The developer doesn't really seem so interested in it anymore, but I feel like the community is still behind using pyuv. So, we could just load the module from the master branch, as suggested here: saghul/pyuv#276

However, we would depend on a fairly unstable module there, which I don't like, so I want your opinion, @baltpeter.

@zner0L
Copy link

zner0L commented Apr 12, 2023

The other option would be to implement some kind of complicated socket logic (Thank you for nothing, Windows. Why does everything need to be so complicated with you.).

@baltpeter
Copy link
Member Author

Random thought: The reason we switched to the IPC mechanism in the first place was because we didn't receive the stdout from mitmproxy in time (maybe due to flushing problems).

Now that we use our own mitmproxy script anyway, wouldn't it make sense to check whether we can go back to stdout (with judicious flushing after every message) before implementing our own Windows layer thingy?

@zner0L
Copy link

zner0L commented Apr 12, 2023

We don't use our own mitmproxy script. I abandoned that because I realized it added no value. But I guess I could try that again, maybe that is the easiest way.

@baltpeter
Copy link
Member Author

I meant the addon, not the script.

@baltpeter
Copy link
Member Author

Should be possible to write to stdout from there as well, shouldn't it?

@zner0L
Copy link

zner0L commented Apr 12, 2023

I mean, I can just use os.write to another file descriptor.

@zner0L
Copy link

zner0L commented Apr 12, 2023

It seems like it works like that. We can just set --set ipcPipeFd=1 and the scripts send the messages via the stdout. which are flushed correctly. (It works on Linux and Windows, I just confirmed that.)

@baltpeter baltpeter assigned baltpeter and unassigned zner0L Apr 13, 2023
@baltpeter
Copy link
Member Author

The structure of a venv on Windows and *nix are annoyingly quite different.

Linux:

├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── flask
│   ├── frida
│   ├── frida-apk
│   ├── frida-compile
│   ├── frida-create
│   ├── frida-discover
│   ├── frida-join
│   ├── frida-kill
│   ├── frida-ls
│   ├── frida-ls-devices
│   ├── frida-ps
│   ├── frida-pull
│   ├── frida-push
│   ├── frida-rm
│   ├── frida-trace
│   ├── litecli
│   ├── mitmdump
│   ├── mitmproxy
│   ├── mitmweb
│   ├── normalizer
│   ├── objection
│   ├── pip
│   ├── pip3
│   ├── pip3.10
│   ├── pygmentize
│   ├── pysemver
│   ├── python -> /usr/bin/python
│   ├── python3 -> python
│   ├── python3.10 -> python
│   ├── sqlformat
│   └── tabulate
├── include
├── lib
│   └── python3.10
│       └── site-packages
│           └── …
├── lib64 -> lib
└── pyvenv.cfg

Windows (no packages installed yet):

├───Include
├───Lib
│   └───site-packages
│       ├─── …
└───Scripts
│   └───activate
│   └───activate.bat
│   └───Activate.ps1
│   └───deactivate.bat
│   └───pip.exe
│   └───pip3.11.exe
│   └───pip3.exe
│   └───python.exe
│   └───pythonw.exe

@baltpeter
Copy link
Member Author

baltpeter commented Apr 13, 2023

Oops, accidentally posted these in the wrong issue:

For some reason, Parcel breaks os.platform (it removes the import and replaces it with an undefined identifier, very helpful). But process.platform works.

Windows just really wants to annoy you, doesn't it? Of course, PATH entries are delimited by ; instead of :. -.-

@baltpeter
Copy link
Member Author

And here we have a commit just for Mr. "why don't you just grep?" @zner0L. :P

tweaselORG/appstraction@bed52e6

@baltpeter
Copy link
Member Author

Oh joy. process.kill() "unconditionally terminates" a process on Windows (why can Node do that but taskmgr.exe can't? :P). That of course means that mitmdump can't write to the HAR file anymore.

@baltpeter
Copy link
Member Author

Apparently, the solution is genuinely to call taskkill.exe manually. Wow.

There is a wrapper for that but I think we can manage on our own.

@baltpeter
Copy link
Member Author

Are you kidding me, Windows? -.-

PS C:\Users\root> taskkill /pid 7956
ERROR: The process with PID 7956 could not be terminated.
Reason: This process can only be terminated forcefully (with /F option).

@baltpeter
Copy link
Member Author

And, of course, using /f causes the same problem.

@baltpeter
Copy link
Member Author

There is also a tskill, but that can only force-stop sigh:

PS C:\Users\root> tskill /?
Ends a process.

TSKILL processid | processname [/SERVER:servername] [/ID:sessionid | /A] [/V]

  processid           Process ID for the process to be terminated.
  processname         Process name to be terminated.
  /SERVER:servername  Server containing processID (default is current).
                         /ID or /A must be specified when using processname
                         and /SERVER
  /ID:sessionid       End process running under the specified session.
  /A                  End process running under ALL sessions.
  /V                  Display information about actions being performed.

@baltpeter
Copy link
Member Author

Thank you kind strangers on the internet who and the same problem and wrote a literal friggin' native Rust module to send Ctrl + C to a process. <3

Thanks to you, it wööörks!

image

@baltpeter
Copy link
Member Author

baltpeter commented Apr 13, 2023

I am ever so slightly worried about the fact that no traffic was recorded for any of the apps I tried, though. .D

@baltpeter
Copy link
Member Author

One problem: The first time I started mitmproxy, I dismissed the Windows Firewall dialog, assuming it would come back the next time. That was a mistake. It didn't and I subsequently forgot about it.

In "Windows Defender Firewall", I clicked "Allow an app or feature through Windows Defender Firewall", "Change Settings" and then set "Python" as allowed:

image

But that isn't enough. I think the HTTPS certificate is still broken.

@baltpeter
Copy link
Member Author

Indeed HTTP traffic is recorded, but HTTPS traffic errors.

@baltpeter
Copy link
Member Author

Amazingly enough, the path to the mitmproxy CA is the same on Windows. :D

@baltpeter
Copy link
Member Author

sigh

There's a \r in the filename: adb push "C:\\Users\\root\\.mitmproxy\\mitmproxy-ca-cert.pem" "/system/etc/security/cacerts/c8750f0d\r.0"

Why? Because we are splitting on \n. :(

https://github.com/tweaselORG/appstraction/blob/c4502ba06d1600329241479400693dc7ac239981/src/android.ts#L170-L172

@baltpeter
Copy link
Member Author

HTTPS traffic! \o/

image

baltpeter added a commit to tweaselORG/appstraction that referenced this issue Apr 13, 2023
@baltpeter
Copy link
Member Author

I haven't tested the emulator, but between tweaselORG/cyanoacrylate#19 and tweaselORG/appstraction#61, Windows support should be mostly read. For Android at least. I haven't looked at iOS at all, yet.

@baltpeter
Copy link
Member Author

I sometimes get this error:

undefined:2
{"status": "done"}
^

SyntaxError: Unexpected token { in JSON at position 75
    at JSON.parse (<anonymous>)
    at Socket.listener (file:///C:/Users/root/coding/tweasel/ca-test/node_modules/cyanoacrylate/dist/index.js:117:30)
    at Socket.emit (node:events:525:35)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)

Pretty sure that is the problem I was expecting here: tweaselORG/cyanoacrylate#19 (comment)

@baltpeter
Copy link
Member Author

Here's the script I used for testing (based on the example):

import { readdir } from 'fs/promises';
import path from 'path';
import { pause, startAnalysis } from 'cyanoacrylate';

(async () => {
    const apkFolder = 'C:\\Users\\root\\Downloads\\single-apks';

    console.log('starting');
    const analysis = await startAnalysis({
        platform: 'android',
        runTarget: 'device',
        capabilities: ['frida', 'certificate-pinning-bypass'],
        targetOptions: {
        },
    });
    console.log('started');

    await analysis.ensureDevice();
    console.log('ensured.');

    const apks = await readdir(apkFolder);
    for (const apkFile of apks) {
        console.log(apkFile);
        const appAnalysis = await analysis.startAppAnalysis(path.join(apkFolder, apkFile));
        console.log('ana started');

        await appAnalysis.installApp();
        console.log('installed');
        await appAnalysis.setAppPermissions();
        console.log('perms set');
        await appAnalysis.startTrafficCollection();
        console.log('traffic started');
        await appAnalysis.startApp();
        console.log('app started');

        await pause(6_000);
        console.log('paused');

        await appAnalysis.stopTrafficCollection();
        console.log('traffic stopped');

        const result = await appAnalysis.stop();
        console.log('app stopped');
        await appAnalysis.uninstallApp();
        console.log('uninstalled');

        console.dir(result, { depth: null });
        console.log('\n\n');
    }

    await analysis.stop();
    console.log('done');
})();

@baltpeter
Copy link
Member Author

And these were my setup steps:

  • Install Node.js 18.15.0.
    • Mostly installer defaults (will already add to PATH).
    • Under "Tools for Native Modules", check "Automatically install the necessary tools."
  • This will already install Python 3.11.2.
  • Install Android platform tools and Android build tools (via), add to PATH. I needed to restart Windows Terminal for the changes to take effect. Opening a new terminal window wasn't enough.
  • Connect phone, run adb devices to test. Trust computer on phone.
  • I didn't install an OEM driver, but might be necessary.
  • Install OpenSSL, add to PATH.
mkdir ca-test
cd ca-test
npm init --yes
npm i cyanoacrylate # I used a `yarn pack`ed `.tar.gz`.
  • The first time mitmproxy is started, allow it to go through the firewall.

Honestly, that is all already covered in the README, you just need to figure out how to get those dependencies on Windows (but then, we don't provide instructions for that on Linux, either).

@baltpeter
Copy link
Member Author

baltpeter commented Apr 16, 2023

The ideviceinstaller command syntax is different between Windows and *nix. :(

PS C:\Users\root> ideviceinstaller
ERROR: Missing command.

Usage: C:\Users\root\Downloads\libimobile-suite-latest_x86_64-mingw64.tar\ideviceinstaller.exe OPTIONS

Manage apps on iOS devices.

COMMANDS:
  list                List installed apps
        -b, --bundle-identifier  Only query for given bundle identifier
            (can be passed multiple times)
        -o list_user      list user apps only (this is the default)
        -o list_system    list system apps only
        -o list_all       list all types of apps
        -o xml            print full output as xml plist
  install PATH        Install app from package file specified by PATH.
                      PATH can also be a .ipcc file for carrier bundles.
  uninstall BUNDLEID  Uninstall app specified by BUNDLEID.
  upgrade PATH        Upgrade app from package file specified by PATH.
  list-archives       List archived apps
        -o xml            print full output as xml plist
  archive BUNDLEID   Archive app specified by BUNDLEID, possible options:
        -o uninstall      uninstall the package after making an archive
        -o app_only       archive application data only
        -o docs_only      archive documents (user data) only
        -o copy=PATH      copy the app archive to directory PATH when done
        -o remove         only valid when copy=PATH is used: remove after copy
  restore BUNDLEID   Restore archived app specified by BUNDLEID
  remove-archive BUNDLEID    Remove app archive specified by BUNDLEID

OPTIONS:
  -u, --udid UDID     Target specific device by UDID
  -n, --network       Connect to network device
  -w, --notify-wait   Wait for app installed/uninstalled notification
                      to before reporting success of operation
  -h, --help          Print usage information
  -d, --debug         Enable communication debugging
  -v, --version       Print version information

Homepage:    <https://libimobiledevice.org>
Bug Reports: <https://github.com/libimobiledevice/ideviceinstaller/issues>

vs.

❯ ideviceinstaller
ERROR: No mode/command was supplied.
Usage: ideviceinstaller OPTIONS

Manage apps on iOS devices.

OPTIONS:
  -u, --udid UDID	Target specific device by UDID.
  -n, --network		Connect to network device.
  -l, --list-apps	List apps, possible options:
       -o list_user	- list user apps only (this is the default)
       -o list_system	- list system apps only
       -o list_all	- list all types of apps
       -o xml		- print full output as xml plist
  -i, --install ARCHIVE	Install app from package file specified by ARCHIVE.
                       	ARCHIVE can also be a .ipcc file for carrier bundles.
  -U, --uninstall APPID	Uninstall app specified by APPID.
  -g, --upgrade ARCHIVE	Upgrade app from package file specified by ARCHIVE.
  -L, --list-archives	List archived applications, possible options:
       -o xml		- print full output as xml plist
  -a, --archive APPID	Archive app specified by APPID, possible options:
       -o uninstall	- uninstall the package after making an archive
       -o app_only	- archive application data only
       -o docs_only	- archive documents (user data) only
       -o copy=PATH	- copy the app archive to directory PATH when done
       -o remove	- only valid when copy=PATH is used: remove after copy
  -r, --restore APPID	Restore archived app specified by APPID
  -R, --remove-archive APPID  Remove app archive specified by APPID
  -o, --options		Pass additional options to the specified command.
  -w, --notify-wait		Wait for app installed/uninstalled notification
                    		to before reporting success of operation
  -h, --help		prints usage information
  -d, --debug		enable communication debugging
  -v, --version		print version information

Homepage:    <https://libimobiledevice.org>
Bug Reports: <https://github.com/libimobiledevice/ideviceinstaller/issues>

@baltpeter
Copy link
Member Author

Looking good. That really wasn't too bad.

image

@baltpeter
Copy link
Member Author

Here are my setup steps for iOS (again, nothing that isn't in the README already):

mkdir ca-test
cd ca-test
npm init --yes
npm i cyanoacrylate # I used a `yarn pack`ed `.tar.gz`.
  • The first time mitmproxy is started, allow it to go through the firewall.

@baltpeter
Copy link
Member Author

Actually, maybe we should mention that you need the iTunes drivers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants