Scriptable Debugger using an embedded Python 3 interpreter

[thp]

This adds scripting support for the debugger. Use cases are automated tracing/debugging of games (Python code can run automatically when a special breakpoint is hit, and Python code can query the internal state of the emulator).

[hrydgard]

Cool! And nicely integrated.

Just curious, what have you used it for? I'd like to see some working useful example code, in order to merge this.

People have been asking for something like this for quite some time, I've always been partial to Lua because it's so tiny and easy to integrate (and would be more viable on mobile), while Python is a bigger thing (although mobile is changing, maybe we could include it now). Depending on use cases mobile might not be interesting anyway...

[thp]

As a simple example (requiring python3-gi and Gtk3 on Debian/Ubuntu installed), it's possible to write a script that will dump the RAM every 100 frames (while the checkbox in the window is activated).

The code example (this is ugly code with global variables all over the place, but it should get one use case across):

import ppsspp
from gi.repository import Gtk

Gtk.init()
frame_no = 0
dump_mem = False

def on_dump_ram_toggled(checkbutton):
    global dump_mem
    dump_mem = checkbutton.get_active()
    print(f'dump_mem is now {dump_mem}')

w = Gtk.Window(title='PPSSPP Python Debugger')
vbox = Gtk.VBox()
b = Gtk.CheckButton.new_with_label('Dump RAM')
b.connect('toggled', on_dump_ram_toggled)
vbox.pack_start(b, True, True, 10)
w.add(vbox)
w.show_all()

def on_frame_function():
    while Gtk.events_pending():
        Gtk.main_iteration()

    global frame_no, dump_mem
    if frame_no % 100 == 0 and dump_mem:
        filename = f'ramdump-{ppsspp.get_game_id()}-frame{frame_no}.raw'
        print('dumping ram to', filename)
        with open(filename, 'wb') as fp:
            fp.write(ppsspp.get_ram_memory_view())
    frame_no += 1

w.set_title(ppsspp.get_game_id())
ppsspp.on_frame(on_frame_function)

[thp]

Another example, for UCES00001 (WipEout Pure):

There is a function at 0x0009cad0 that takes a C string as its first argument. We want to know with which values it is called. This can be accomplished by setting a breakpoint at that address and installing a breakpoint handler from Python. When the breakpoint is hit, read the register value of A0 (first argument register), and treat this as a pointer to a C string and print it.

Note that in all cases, you need to enable the "DevMenu" top left, and select "Load Python Script" to load the script (nothing is loaded automatically at the moment, it always needs user interaction). The file must be called ppsspp_python.py in the Python folder of the memstick root.

import ppsspp 
                                                                                               
hash_filename_func = ppsspp.default_load_address + 0x0009cad0                                  

def read_cstring_at(addr):
    result = []
    while True:
        ch = ppsspp.read_memory_u8(addr)
        if ch == 0:
            break
        result.append(ch)
        addr += 1
    return bytes(result)

def on_breakpoint_function(addr):
    if addr == hash_filename_func:
        a0 = ppsspp.get_reg_value(ppsspp.MIPS_REG_A0)
        filename = read_cstring_at(a0)
        print('Filename:', filename)

if ppsspp.get_game_id() == 'UCES00001':
    ppsspp.add_breakpoint(hash_filename_func)
    ppsspp.on_breakpoint(on_breakpoint_function)

[thp]

And another example, this time just looking at which files sceIoOpen and sceIoDopen are called with (this should work with any game):

import ppsspp

def read_cstring_at(addr):
    result = []
    while True:
        ch = ppsspp.read_memory_u8(addr)
        if ch == 0:
            break
        result.append(ch)
        addr += 1
    return bytes(result)

def on_breakpoint_function(addr):
    symbol_here = addr_to_symbol.get(addr, None)
    if symbol_here == 'zz_sceIoOpen':
        print(f'sceIoOpen(filename={read_cstring_at(ppsspp.get_reg_value(ppsspp.MIPS_REG_A0))}, ...)')
    elif symbol_here == 'zz_sceIoDopen':
        print(f'sceIoDopen(dirname={read_cstring_at(ppsspp.get_reg_value(ppsspp.MIPS_REG_A0))}, ...)')

symbol_to_addr = {}
addr_to_symbol = {}
for name, address, size in ppsspp.get_all_symbols():
    print(f'{name} @ {address:08x} (size={size})')
    symbol_to_addr[name] = address
    addr_to_symbol[address] = name

ppsspp.on_breakpoint(on_breakpoint_function)
ppsspp.add_breakpoint(symbol_to_addr['zz_sceIoOpen'])
ppsspp.add_breakpoint(symbol_to_addr['zz_sceIoDopen'])

[thp]

Just curious, what have you used it for? I'd like to see some working useful example code, in order to merge this.

I have posted some simple examples what I've used it for. I also have another savegame-related example (similar to the sceIoOpen hooking example) and of course it's all "exploration style" small scripts to check something out easily.

People have been asking for something like this for quite some time, I've always been partial to Lua because it's so tiny and easy to integrate (and would be more viable on mobile), while Python is a bigger thing (although mobile is changing, maybe we could include it now). Depending on use cases mobile might not be interesting anyway...

Yeah right now I assume that this mostly targets Linux and macOS, both on the Desktop. There's no reason why it wouldn't work on Windows (but I haven't tested this, and you probably need to build with the same compiler that Python 3 was build with to avoid C runtime issues on Windows). And same for mobile, should work but untested. Building Python is still a bit more involved than building Lua, unfortunately (this links against the system-installed Python development libraries).

Personally, I just need this during development on my computer, so not useful on mobile, and I'm more used to Python, which is why I picked that for scratching that itch. This PR could probably be used as a template to implement Lua support, though.

[unknownbrackets]

In my opinion:

• If we add additional APIs, they should be as consistent as reasonable with the websocket API. Or the websocket API should change to be consistent with them. It'd be best for them to be the same one way or another unless there's a good reason. I will say the web debugger already uses the current API.
• There's non-trivial danger that this will be broken over time if it's left off by default and not intended to be used cross platform.
• The intent of the websocket API wasn't just for debugging; it was also for accessibility (i.e. text to speech, additional sound or tactile feedback), social integrations/challenges (i.e. a counter for twitch streaming), utilities (tool to generate maps as you play, track received items, etc.), enhancements (triggering different higher quality/remixed music to play at points), cheating/trainer, fan translation, launchers, etc. An embedded scripting engine (such as Python) would possibly serve these goals well too - so I don't want to preclude the idea of a cross-platform enabled API.
• If we add callbacks for breakpoints, we should think about threading. It seems like this scripting is synchronous and on the cpu thread generally, but is inited from UI. Those should be the same thread but I don't want to box ourselves in, especially if this is off by default.
• Also, for other use cases I wonder if we'd want breakpoint callbacks to return a value for whether to break. But that's for later anyway.

-[Unknown]

[hrydgard]

Nice examples!

And great points unknown!

Absolutely agree that we should strive for debugger APIs to be as similar as possible, so if this can be tweaked to match closer, that would be great.

It might also be reasonable to instead of this, just write a python library that hits the existing websocket debugger API. In that case you'll be running the python script from the "outside", though callbacks become harder and memory dumps etc might perform worse this way so there might be enough reason to have both indeed.

The danger of breakage for default-off code is indeed high - we could easily enable it in one or more of our Linux CI builds though, for example, so at least compilation will be checked.

[unknownbrackets]

Technically, should check the actual memory size (might be larger than 32 MB for some homebrew and PS3 remasters.)

-[Unknown]

[thp]

Is there a way to query the size from the PPSSPP Core?

[unknownbrackets]

I suppose Memory::g_MemorySize is the best way. You might also use PSP_GetKernelMemoryBase() and PSP_GetUserMemoryEnd() - PSP_GetKernelMemoryBase()

-[Unknown].

[unknownbrackets]

A python script from outside should be able to do all of these things. Instead of the breakpoint callback, you'd have to use the cpu.stepping event, and check the PC, then resume emulation explicitly using cpu.resume. To do that with symbols, you'd use hle.func.list.

I'd be interested to know how performance compares. The overhead isn't huge but definitely exists. It's worth noting that the websocket API is asynchronous and intentionally separate from PPSSPP. It's also intentionally thread-specific, so you could have multiple python scripts concurrently connected doing different things.

None of the above is to say that adding python scripting would be bad, just noting that the "outside" approach is possible.

-[Unknown]

[thp]

• If we add additional APIs, they should be as consistent as reasonable with the websocket API. Or the websocket API should change to be consistent with them. It'd be best for them to be the same one way or another unless there's a good reason. I will say the web debugger already uses the current API.

Is the websocket API documented anywhere outside the code? My initial try (before implementing this PR) was actually to use the websocket API from some Python websocket module, but I failed and was lazy, so this was kind of "faster" for me to achieve my goal.

I can definitely name the functions cpu_breakpoint_add(), cpu_breakpoint_update(), cpu_breakpoint_remove(), and cpu_breakpoint_list() for consistency, but some functions that I used in my scripts don't (yet?) have a websocket API equivalent.

On the other hand, the PR as is isn't as feature complete yet (e.g. no memory breakpoints at all), I just implemented the functions as I needed them.

• There's non-trivial danger that this will be broken over time if it's left off by default and not intended to be used cross platform.

Yes. If it helps, I could turn this PR into some kind of "debugger plugin API", so that instead of depending on Python at build time, it would just dlopen() a "debugger plugin" with a fixed C ABI (to avoid C++ ABI inconsistencies) and then have the same hooks available; all this would still be living in the same address space (in-process).

Not sure when I find the time, but if that's something that would be acceptable (as opposed to just going with the websockets API only), I could look into that at some point (and implemented the Python debugger support as seen here as example "debugger plugin"). Such a "debugger plugin API" would then also allow for other scripting languages such as Lua to be implemented, and no build-time dependencies would be added to the core, so the feature could be enabled unconditionally for any platforms that have some kind of shared library loading mechanism.

But then again, out-of-tree plugins could similarly break, but at least it wouldn't affect the core.

Let me know if adding this "shared library plugin layer" would help acceptance of this change or if it would add an unnecessary layer of complexity.

• The intent of the websocket API wasn't just for debugging; it was also for accessibility (i.e. text to speech, additional sound or tactile feedback), social integrations/challenges (i.e. a counter for twitch streaming), utilities (tool to generate maps as you play, track received items, etc.), enhancements (triggering different higher quality/remixed music to play at points), cheating/trainer, fan translation, launchers, etc. An embedded scripting engine (such as Python) would possibly serve these goals well too - so I don't want to preclude the idea of a cross-platform enabled API.

Yes, and that's what I looked into at first, but the websocket API wasn't documented well enough (might help if there are scripting examples). I mean, if you can "port" the above examples to some external scripting with the websocket API (in any language), I'd be a happy camper and can probably build my one-off debugging tools on top of those. But for things like dumping and searching RAM, in-process might still be preferred.

• If we add callbacks for breakpoints, we should think about threading. It seems like this scripting is synchronous and on the cpu thread generally, but is inited from UI. Those should be the same thread but I don't want to box ourselves in, especially if this is off by default.

In my tests, the loading and frame / breakpoint callbacks indeed all happen on the same thread. Apart from that, I didn't check if they are conceptually the same thread or not. Improvement suggestions there appreciated (as long as there is only a single "MIPS CPU" thread in which the CPU breakpoints happen, that should probably be the thread in which the scripting debugger lives, too).

• Also, for other use cases I wonder if we'd want breakpoint callbacks to return a value for whether to break. But that's for later anyway.

Yes. It might also make sense that instead of having a "global" callback we have a per-breakpoint callback. The current PR assumes there's always only one entity that wants to use callback-based breakpoints.

[unknownbrackets]

Is the websocket API documented anywhere outside the code? My initial try (before implementing this PR) was actually to use the websocket API from some Python websocket module, but I failed and was lazy, so this was kind of "faster" for me to achieve my goal.

Not currently. I'm too lazy to carefully maintain API documentation or setup CI to republish/etc. There are reasonably detailed comments in the code, which I'm not too lazy to accurately maintain. If anyone wants to make this better, it's very welcome.

It looks possible to use websocket or websockets to do this, although asynchronous handling in Python looks... cumbersome. I'm not a Python wizard, though. I created a dedicated Node.js sample since I know that a bit better:

https://github.com/unknownbrackets/ppsspp-api-samples/tree/master/js
(see monitor-file-open.js)

In theory, it's roughly the same idea, possibly with a bunch of ifs in an "on_message" callback for which response you're getting.

I can definitely name the functions cpu_breakpoint_add(), cpu_breakpoint_update(), cpu_breakpoint_remove(), and cpu_breakpoint_list() for consistency, but some functions that I used in my scripts don't (yet?) have a websocket API equivalent.

Well, I think naming them the same makes sense. I forget if there's an event for every frame, but we could add that to the websockets API for sure (maybe Python would be "on_" + event.) If there are special "memory view" or similar functions, I suppose those don't need to match as long as they're consistent.

If someday we do have documentation, it'd be nice if it is reasonably consistent between, with differences only where differences would make obvious sense.

Yes. If it helps, I could turn this PR into some kind of "debugger plugin API", so that instead of depending on Python at build time, it would just dlopen() a "debugger plugin" with a fixed C ABI (to avoid C++ ABI inconsistencies) and then have the same hooks available; all this would still be living in the same address space (in-process).

That could be interesting. Then people who prefer 1-indexed arrays and a language no one actually uses for anything else can use Lua, and people who want an actually useful language worth learning can use Python, V8/JS, Ruby, C, whatever.

So the feature could be enabled unconditionally for any platforms that have some kind of shared library loading mechanism.

But then again, out-of-tree plugins could similarly break, but at least it wouldn't affect the core.

Right. Maybe we could have a simple C plugin we test via CI...

Yes, and that's what I looked into at first, but the websocket API wasn't documented well enough (might help if there are scripting examples).

Well, I created some better samples at least. I didn't do all your examples but figured the sceIoOpen was a good representative one.

But for things like dumping and searching RAM, in-process might still be preferred.

I was thinking of adding an API to just grab the entire RAM as a base64 blob (or potentially non-base64 if I get around to adding non-json response support to the WebSocket API.) But I also have ideas for a "mark and set" memory API.

The idea would be:

• memory.delta.mark - create snapshot / mark point
• memory.delta.pages - report pages with differences (manageable list of e.g. 8k for UI)
• memory.delta.changes - report actual changed ranges, with page range parameter... maybe with old/new values.
• memory.delta.unmark - throw away snapshot (haven't decided if there's just one or some max.)

Along with some form of memory searching. This is in part around ideas for a better memory debugger UI experience.

In my tests, the loading and frame / breakpoint callbacks indeed all happen on the same thread.

Okay, makes sense. CPU breakpoints indeed should be on the same thread. Maybe we could add asserts later.

-[Unknown]

[hrydgard]

Then people who prefer 1-indexed arrays and a language no one actually uses for anything else can use Lua

Ow the burn :) Probably deserved though. 1-indexed arrays are the thing I like the least about the language, for sure.

Liking the sound of the rest. And the node.js sample looks great.

[thp]

Since this PR kind of fell asleep a bit, wonder if it has chances of being merged if we change the explicit Python support with a simple C API (load shared library with a single known entry point -> pass a struct of function pointers to it -> it returns a struct of function pointers) and then the Python support (or any other language) can be implemented as a shared library.

I could possibly find time to draft something and bring the PR up to date with the current master branch if there's interest?