register commands require a frame unconditionally

[lf-]

Bugzilla Link	49941
Version	unspecified
OS	Linux
CC	@JDevlieghere,@jimingham

Extended Description

I've been trying out lldb as a replacement for gdb for working on my operating system, which I develop on qemu with a gdbstub. Part of this involves debugging code that is run before the initialization of the stack, in early startup code and exception handlers.

Here is an example thing that can go wrong:

(lldb) register write t0 123
error: invalid frame

The register read command is also affected by this issue. It appears that the cause of it is that a bunch of the commands in CommandObjectRegister.cpp are defined as eCommandRequiresFrame when they should handle there not being a frame present with slightly degraded functionality. It's not that they don't require a frame at all though: ExecutionContext::GetRegisterContext needs a frame to work, but it should be skipped if one is unavailable, which it is not currently.

[jimingham]

I'm not sure I understand your problem.

Presumably by the time you stop the target program it is either executing some code or about to execute some code, and you can determine the register state it will adopt when it goes to do that.

What keeps you from presenting that something to lldb as frame 0 of some thread of execution?

[lf-]

The specific trouble I have is that if I want to read registers when the system first boots (before any code has been run at all), there is no way of doing that, that I know of, without apparently needing to write some Python as I just found out. It would be useful to have a fallback mode that dumps the registers without needing to have a frame recognizer.

[jimingham]

What is the meaning of the registers you want to present?

[lf-]

What is the meaning of the registers you want to present?

I want to find the initial state of the user registers (i.e. x1-x31) and more generally, understand the initialization process of the virtual system. So I want to step through the initialization code instruction by instruction [1]. The mostly undocumented initialization code is dynamically generated by qemu at runtime, so I don't have any module name or executable file for it. After messing with it and reading source for too long, I am still unsure if it is even possible to define a frame recognizer to tell lldb I have a frame for this code.

I have tried reading the source code, in particular, https://reviews.llvm.org/D44603, to try to understand what is even meant by the module name argument to frame recognizer add, but even doing:

frame recognizer add -l mu.NullFrameRecognizer -x -s .* -x -n .*

with the following frame recognizer:

class NullFrameRecognizer:
def get_recognized_arguments(self, frame):
return []

does not get anything from frame info 0.

If you wish to try this task out for yourself, try:

    qemu-system-riscv64 -machine virt -m 128M -smp 1 -nographic -s -S

then, from another window, with `lldb -a riscv64`,

    (lldb) gdb-remote 1234

Try to disassemble this startup code that the CPU has stopped at: `disassemble --pc` doesn't work because there's no frame, register read, likewise. I can disassemble it by address, but that's not helpful, because the only reason I know the program counter address is because I've used gdb extensively on this target before. I don't know how I can get the initial register state to be able to understand the code either, given `register read` doesn't work :)

This is trivial in gdb, it will tell you the program counter as soon as you connect, and `info registers` does not require a frame.

[1]: Stepping through it is also broken for me, `si` appears to be essentially continuing instead of stepping one instruction; unsure if this is related to not having frames and I am probably going to file a bug for this as well soon

[jimingham]

How are you connecting to the qemu stub?

If you connect to a standard gdb-remote stub with something like the gdb-remote command in lldb, then the ProcessGDBRemote plugin will query for the thread info (for qemu that's usually one thread per executing virtual core), and among other things make a 0th frame to hold the register information the stub told it for that thread. That should all just happen automatically.

What is different in your setup that that is not happening?

Anyway, if you want to attach to a system and use lldb to represent threads of execution that are not currently active on any of the system cores, the lldb construct for that is the "OperatingSystem Plugin". That is a way of making "Memory backed" threads either to overlay ones already running on the core, or to make up new ones.

The "frame recognizer" is for re-interpreting a real frame, not for making one up out of whole cloth. It allows us to do things like: when stopped at a known function - but one without debug information - lldb can dig the argument values out of registers, cast those values to their known types, and fill the result of "frame var" with these reinterpreted values.

That's not what you want to do - you don't have a frame to begin with, and you aren't triggering off of the function that occupies a particular frame.

[lf-]

How are you connecting to the qemu stub?

If you connect to a standard gdb-remote stub with something like the
gdb-remote command in lldb, then the ProcessGDBRemote plugin will query
for the thread info (for qemu that's usually one thread per executing
virtual core), and among other things make a 0th frame to hold the register
information the stub told it for that thread. That should all just happen
automatically.

What is different in your setup that that is not happening?

Anyway, if you want to attach to a system and use lldb to represent threads
of execution that are not currently active on any of the system cores, the
lldb construct for that is the "OperatingSystem Plugin". That is a way of
making "Memory backed" threads either to overlay ones already running on the
core, or to make up new ones.

The "frame recognizer" is for re-interpreting a real frame, not for making
one up out of whole cloth. It allows us to do things like: when stopped at
a known function - but one without debug information - lldb can dig the
argument values out of registers, cast those values to their known types,
and fill the result of "frame var" with these reinterpreted values.

That's not what you want to do - you don't have a frame to begin with, and
you aren't triggering off of the function that occupies a particular frame.

Noted, I will probably want to write a debugger plugin for the OS work I'm doing, so thanks for the keywords.

Here is the reproduction, if you want to try it:

In one terminal:
$ qemu-system-riscv64 -machine virt -m 128M -smp 1 -nographic -s -S

In another:
$ lldb -a riscv64
(lldb) gdb-remote 1234
Process 1 stopped

• thread #​1, stop reason = signal SIGTRAP
  frame #​0: 0xffffffffffffffff
  (lldb) register read
  error: invalid frame

Reproducible on the latest llvmorg-13-init-7035-g27dfcd978edc with qemu 5.2.0 (and a <2 week old git version) on linux.

As for a gdb (11.0.50.20210122-git) session, it does indeed seem to be getting a frame 0, which lldb does not:

(gdb) set architecture riscv
The target architecture is set to "riscv".
(gdb) set disassemble-next-line on
(gdb) target remote :1234
Remote debugging using :1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x0000000000001000 in ?? ()
=> 0x0000000000001000: 97 02 00 00 auipc t0,0x0
(gdb) bt
#​0 0x0000000000001000 in ?? ()

[lf-]

Alright, an update. What's happening here is that something that, according to a comment, is not supposed to happen, is in fact happening.

      // There shouldn't be any way not to get the frame info for frame
      // 0. But if the unwinder can't make one, lets make one by hand
      // with the SP as the CFA and see if that gets any further.
      if (!success) {
        cfa = reg_ctx_sp->GetSP();
        pc = reg_ctx_sp->GetPC();
      }

Here, the call to GetSP() returns LLDB_INVALID_ADDRESS, because it fetched a garbage register number. Why did it do that? It didn't have a register with eRegisterKindGeneric and kind LLDB_REGNUM_GENERIC_SP. Oops.

Why'd that happen? Well, qemu was giving us xml that doesn't define any generic attribute for any of the registers (see ProcessGDBRemote.cpp:4400). Is it qemu's fault? Nope, they have the latest version of these target xml files from gdb (see gdb/features/riscv/64bit-cpu.xml in the gdb tree). Is it gdb's fault? Not really, it seems like this "generic" attribute is of lldb's invention to begin with, and none of their target definitions have these attributes!

I believe that given this information I probably should go write a Python file that defines the target properly. I can't fix it upstream as I doubt they would accept a patch adding stuff that only lldb can use (it's kind of a moot point anyway, it is not physically safe for me to contribute to anything under the GNU project).

[llvmbot]

There should always be a frame zero, even if there is no stack. Maybe the bug here is somehow we are failing to create a frame zero? Frame zero just gets the stock register context with the actual register values. All other frames (1 and up) must actually unwind the register values.

[llvmbot]

Are you using a custom GDB server that is built into your OS? It might help to see a packet log of what is going on and I might be able to tell more why things are failing.

If you are using a GDB server that isn't lldb-server, then LLDB needs more information about your registers. LLDB discovers the registers in ProcessGDBRemote.cpp in the function:

void ProcessGDBRemote::BuildDynamicRegisterInfo(bool force);

In there it will check in the following order

• check if there is a target definition file specified in the LLDB settings and use this file which defines all registers
• read the target.xml from the GDB server
• send a series of qRegisterInfo packets (LLDB extension packet)

Does your GDB server support the "target.xml" so it can tell LLDB about the registers that it has? I would guess the answer is yes. If this is the case, then LLDB still needs extra information about the registers:

• compiler register number for reach register (register numbers used in .eh_frame for example)
• DWARF register numbers for each register (usually the same as compiler reg numbers, but these sometimes differ)
• LLDB generic information (which register is PC, SP, FP, etc)

When we fetch registers via the target.xml packet, we can use the "lldb_private::ABI" plug-ins to augment and provide the missing register information via:

virtual void ABI::AugmentRegisterInfo(RegisterInfo &info) = 0;

We will first fill in the lldb_private::RegisterInfo struct from lldb-private-types.h with the info retrieved via XML, and the ABI plug-in is used to set all of the other LLDB specific info (compiler reg #, DWARF reg #, generic reg #, etc).

I would guess that this is the reason this is all failing.

[llvmbot]

a GDB remote packet log can be exported using:

(lldb) log enable -f /tmp/packets.txt gdb-remote packets

then run your debug session, quit, and attach the file.

[llvmbot]

There is a patch in progress that adds better support for RISCV (32 and 64), though I don't think it actually was checked in yet. I will try and find it and post the link. We had some people doing RISCV work that were able to get LLDB working by applying this patch. It adds the necessary ABI plug-in and it fixes the register info and then things worked.

[llvmbot]

Found it. Try applying this and see if this fixes your issues:

https://reviews.llvm.org/D62732

[llvmbot]

If you still have issues, then please attach a packet log from the debug session using the command I supplied.

[lf-]

Found it. Try applying this and see if this fixes your issues:

https://reviews.llvm.org/D62732

Oh my. I should have gone looking. I'll try that one next week when I have more time.

Below is a description of the (imo) unreasonably hard to debug failure that does happen when there's no generic registers registered because the plugin isn't present:

There should always be a frame zero, even if there is no stack. Maybe the
bug here is somehow we are failing to create a frame zero? Frame zero just
gets the stock register context with the actual register values. All other
frames (1 and up) must actually unwind the register values.

Yes, that's correct. There is a frame 0, it has cfa_is_valid true, but the actual cfa and pc addresses are all ff because it fails to get either one and instead set them to LLDB_INVALID_ADDRESS, which is all ones.

The reason it fails to get the stack frame is because lldb doesn't have generic register info for riscv64 (as confirmed by calling Dump() with a debugger attached to lldb).

We thus have the invalid frame error. This also explains why the unwind is failing.

We really need to improve the error reporting for this, though. Maybe a verbose log line saying that it couldn't find the generic registers and that an ABI plugin/other target info needs to be written? I'm not familiar with the logging in lldb or how we could usefully surface this information.

This will keep happening to embedded folks working on new odd targets and getting this confusing pattern of errors, so I really think it needs to be explicitly reported. I may come up with a patch. What behaviour would be acceptable? Quitting the debugger on connection? Something else?

[llvmbot]

Found it. Try applying this and see if this fixes your issues:

https://reviews.llvm.org/D62732

Oh my. I should have gone looking. I'll try that one next week when I have
more time.

Below is a description of the (imo) unreasonably hard to debug failure that
does happen when there's no generic registers registered because the plugin
isn't present:

There should always be a frame zero, even if there is no stack. Maybe the
bug here is somehow we are failing to create a frame zero? Frame zero just
gets the stock register context with the actual register values. All other
frames (1 and up) must actually unwind the register values.

Yes, that's correct. There is a frame 0, it has cfa_is_valid true, but the
actual cfa and pc addresses are all ff because it fails to get either one
and instead set them to LLDB_INVALID_ADDRESS, which is all ones.

The reason it fails to get the stack frame is because lldb doesn't have
generic register info for riscv64 (as confirmed by calling Dump() with a
debugger attached to lldb).

Indeed!

We thus have the invalid frame error. This also explains why the unwind is
failing.

We really need to improve the error reporting for this, though. Maybe a
verbose log line saying that it couldn't find the generic registers and that
an ABI plugin/other target info needs to be written? I'm not familiar with
the logging in lldb or how we could usefully surface this information.

We definitely need to improve this.

This will keep happening to embedded folks working on new odd targets and
getting this confusing pattern of errors, so I really think it needs to be
explicitly reported. I may come up with a patch. What behaviour would be
acceptable? Quitting the debugger on connection? Something else?

I think we should give a much better error and explanation of what we can do to fix the issue. For this to be correct we will probably need a new virtual function on the lldb_private::Process class that gets called when no compiler, DWARF or generic register information is set.

The ProcessGDBRemote version of this function can first check if there are any registers at all.

If there are no registers, then the GDB server might not support the "target.xml". In that case we can tell users they must create a target definition file, and point to examples that are checked into LLDB already. These are python files that can define everything that is needed for a debugging target that uses a GDB server that doesn't have dynamic register info. We have many examples in our sources:

lldb/examples/python/*_target_definition.py

Users would make a new target definition file and specify it using the "settings set" command:

(lldb) settings set plugin.process.gdb-remote.target-definition-file /path/to/riscv_target_definition.py

If there are registers, we should check if any of the register information has compiler, dwarf and generic information. If there is none, we can suggest to use the target definition files as above, or let them know they can add a lldb_private::ABI plug-in implementation by modifying LLDB code.

The other easier solution is to detect the registers in ProcessGDBRemote and then immediately following that, check to verify that the compiler, DWARF and generic registers are set for the important registers: PC, SP, FP (if there is one for the architecture). If not, dump out some text as defined above and kill the session?

I am open to any better suggestions.