"maintenance info sections" expected output and other bare metal woes.

[nicolasnoble]

Alright so, I've decided to try the ghidra debugger again with the 10.3 release, and I'm still not getting good results.

Context: I'm the author of PCSX-Redux, and I have written a gdb server in there that gdb or IDA can successfully connect to in general, and properly debug.

In order to try out what I'm talking about here, one can simply download PCSX-Redux. The Windows version contains a file called openbios.elf, an open source software which is a MIPS r3000 binary, able to boot the PlayStation. It can simply be loaded within ghidra to analyze.

Within PCSX-Redux, the default installation will load openbios and it can be debugged straight by enabling the debugger: in Configuration->Emulation, uncheck "Dynarec CPU", check "Enable Debugger", and check "Enable GDB Server". You can then connect to it using gdb-multiarch:

I managed to get past a few hurdles:

• ghidra doesn't recognize the mips:3000 architecture:

  ghidra/Ghidra/Processors/MIPS/data/languages/mips.ldefs

  Line 33 in ce5f6b4

  <external_name tool="gnu" name="mips:4000"/>

  Changing this line to mips:3000 makes ghidra accept an offer from the generic gdb class, which is probably the only one that should handle this properly. Note that I don't think there's a good way to handle this in upstream: correctly handling the nuances between the r3000 and r4000 and above CPUs would require doing a breaking change by further refining the processor IDs. "MIPS:LE:32:default" would neither fit r3000 nor r4000 properly anymore.

• ghidra doesn't seem to understand the "remote target" threadid. I'm sending some generic thread information to the gdb client, and the non-machine version of info threads returns this:

(gdb) info threads
  Id   Target Id                  Frame
* 1    Remote target     0xbfc00004 in ?? ()

Which ends up confusing this function:

ghidra/Ghidra/Debug/Debugger-agent-gdb/src/main/java/agent/gdb/manager/impl/GdbThreadInfo.java

Line 78 in ce5f6b4

GdbThreadInfo(String id, String targetId, String name, String state, String core,

I had to add the following:

		mappingMatcher = TARGET_ID_REMOTE_TARGET.matcher(targetId);
		if (mappingMatcher.matches()) {
			this.tid = 1;
		}

Otherwise, with this.tid being null, then down the line, there'll be multiple exceptions with various pieces of the code trying to use it as a number.

Now, with these two changes, I can properly attach and step through the code, but:

1. I can't place breakpoints. They are simply ignored. From searching around, it seems that this is because the maintenance info sections command returns nothing, but at this point, I don't have a good idea of what this is expected to do nor return. I have the feeling this is going to be the same problem as with info proc mappings, which can then be solved with another alias, but given I don't know what to send out, I'm a bit stuck here. I guess I could also try changing the server to support the command properly, but I'm similarly stuck in what this is expected to do anyway. Note that I can place breakpoints manually using the gdb console, and they'll even show up in the Dynamic view, or the Objects view. I just can't interact with breakpoints in the UI.
2. I can't inspect values in the Decompile view. Every time I try to hover a variable, instead of a value, I get an exception in the popup that says Error: java.lang.NullPointerException: Cannot invoke "ghidra.app.plugin.core.debug.stack.UnwindInfo.function()" because "this.info" is null. I haven't started investigating this one yet, aside from this exception that's firing:

   ghidra/Ghidra/Debug/Debugger/src/main/java/ghidra/app/plugin/core/debug/stack/AnalysisUnwoundFrame.java

   Lines 97 to 99 in ce5f6b4

   	if ((info == null) == (infoErr == null)) {
   	throw new AssertionError();
   	}

   I don't think this has to do with the "Modules" window being empty, but I'm not exactly sure what's going on yet.

What would be the best course of action for me to clear out these next hurdles?

[d-millar]

@nicolasnoble Hey Nicolas, with a little luck, will get a chance to look at your issues in detail tomorrow. In the meantime, if you look at Discussion #4301 and Ghidra/Debug/Debugger-agent-gdb/data/scripts/fallback_info_proc_mappings.gdb, I think you’ll see what you need for “maint info sections” and “info proc mappings”. Basically, you want to return a single line indicating all of memory is in play and is contained in one section.

On the remote target threadid, might be as simple as not returning the tid in hex, but I need to look at that in a bit more detail. The breakpoint failure most definitely sounds liked a casualty of the missing commands above. Basically, without a module list, the debugger doesn’t know how to map static Ghidra programs to dynamic memory. You can try forcing all memory in from the Regions view and mapping memory identically from the Modules view as a test, although that’s probably not the ideal long term solution.

No immediate guess on the stack unwind problem - hopefully tomorrow. Probably, better to modify the debugger opinions than the mips.ldefs unless you plan to write a language module for the 3000. Will ping the team on the best approach there and get back to you.

[nicolasnoble]

Right, so I'm already using this script to override info proc mappings, but it doesn't do anything for maintenance info sections, which currently just returns empty without ever talking to the gdb server I wrote. Since there's nothing dynamic on the platform, I am guessing this is normal.

[nicolasnoble]

About the tid, gdb only returns this one string, Remote target, without anything else in it. I'm returning a fake numerical thread id, but it seems gdb ignores it and just overrides it with this string.

[nicolasnoble]

Ah, about thread id, it seems like I only need to implement the support for the multiprocess feature extension: https://github.com/bminor/binutils-gdb/blob/56c1f748a5df6d0f7c75586204e3010fa3e164f9/gdb/remote.c#L12011-L12023

https://sourceware.org/gdb/onlinedocs/gdb/General-Query-Packets.html#multiprocess-extensions

I'll try adding this.

[d-millar]

Re "maintenance info sections", correct you have to roll you own here. I would guess the script should look something like:

define info proc mappings
echo 0x0 0xFFFFFFFF 0xFFFFFFFF 0x0 mem \n
end

define maintenance info sections ALLOBJ (or -all-objects)
`PCSX-Redux', file type elf64-mips:3000. \n
[0] 0x0->0xFFFFFFFF at 0x00000000: .interp ALLOC LOAD READWRITE DATA HAS_CONTENTS
end

but am guessing you'll need to play around with this.

[nicolasnoble]

Perfect, thanks. I had no idea what format this was expecting, so this helps a lot. I'll give this a try later today.

[nicolasnoble]

So with these changes, (and still the change to the mips.ldefs file) breakpoints now work properly, but hovering on variables now fail differently.

When trying to hover on a variable the first time, there'll be an out of bound exception during this callstack:

Any further hovering just gets a generic failure about how it can't find the stack frame.

[d-millar]

@nicolasnoble @dev747368 has a tickets in for the various hover issues - will add this to the queue. Also, if I get a chance today, will see if there's a straightforward workaround. Thanks!

[nicolasnoble]

I'll probably make a quick video to show how to get going because the setup is quite involved at this point, which I can then share here.

[nicolasnoble]

Here: https://youtu.be/yqnPm9unNJQ - probably want to watch this in 4k. I'm showing how to do the connection, how to start the debugger, but most importantly, I'm showing the various bugs there is with data inspection; I don't know if you're tracking all of them.

[d-millar]

@nicolasnoble Just following up: the video was very very helpful - am going to lobby all of customers to do this. :)

Various issues:
(1) Had you used "openbios.elf" in place of "PCSX Binary" in your ghidra_debugger_script, I think you would have avoided the need for manually mapping memory. That said, that's almost as much work, if not more, than hitting the <=> key, so.... I think your demonstration of that feature in the video does your customers a service.
(2) I am going to add a sample script for "maintenance info sections" to the distribution - not sure why I didn't do that when I added the "info proc mappings". Doesn't really obviate the need for the script, so will have to think about that some more.
(3) Have added a ticket for the problems with the hover - the fix may be a bit complex, so not sure when that will be remedied.
(4) Have added a ticket for the hover stack trace - that should be fixed soon.
(5) Have added a ticket for migrating structures from the Static listing to the Dynamic listing. Great idea - seconded by others, thanks!
(6) Have a ticket in to add r3000, so you won't have to edit the .ldefs file.

Again, thanks for the visuals. Keep us posted if you hit other bugs!