Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement gdb-stub on 3rd UART #360

Merged
merged 15 commits into from Apr 8, 2023
Merged

Implement gdb-stub on 3rd UART #360

merged 15 commits into from Apr 8, 2023

Conversation

xobs
Copy link
Member

@xobs xobs commented Apr 6, 2023

This patchset implements a gdb-stub server that runs on the 3rd UART (app_uart). With this patch, enabling the gdb-stub feature will yield a server that allows for application level debugging.

Building

To build, add gdb-stub to the list of features:

cargo build \
    --release \
    --target riscv32imac-unknown-xous-elf \
    --package xous-kernel \
    --features "renode gdb-stub"

Then load the kernel into an image onto the target device.

Usage

To use, attach gdb to the serial port. For example, to debug the ticktimer, you might run:

$ riscv-none-elf-gdb \
    -ex 'tar ext \\.\COM43' \
    -ex 'attach 2' \
    target/riscv32imac-unknown-xous-elf/release/xous-ticktimer

This will load the xous-ticktimer ELF file, then attach to the GDB server running on the serial port connected to COM43, then attach to process ID 2.

You will then be able to use normal GDB commands to inspect memory, single-step, add breakpoints, and switch threads.

Useful commands

The info thr command will list current threads. You may then switch threads with e.g. thr 4 to switch to thread 4.

The x command can be used to examine arbitrary memory.

b is used to insert breakpoints. Note that breakpoints are accomplished by replacing instructions in memory with c.ebreak instructions, so debugging does not work with XIP processes.

c will continue execution, and Control-C will pause execution.

stepi will single-step assembly execution.

You may switch processes with attach [PID]

Debugging and servers

When a process is under debug, messages will queue up. Interrupts are masked, and the process is paused.

When you continue a process under debug, it is unpaused, interrupts are re-enabled, and messages are delivered.

Renode fixes

When running this under Renode, a very recent copy (from April 2023 or later) is required. This also requires adjustments to the simulation models, which are included in this patchset.

xobs added 14 commits April 6, 2023 10:42
This syscall is the opposite of ClaimInterrupt.

Signed-off-by: Sean Cross <sean@xobs.io>
This function never failed. Make it infallible so that it matches the
same calling convention as `enable_irq()`

Signed-off-by: Sean Cross <sean@xobs.io>
Previously, `disable_irq()` would unconditionally mask an interrupt in
the `sim` register. However, when interrupts are disabled this register
is zero, and we really should be updating `SIM_BACKING`.

Consult the state of the interrupt handler and update the appropriate
field as necessary.

Signed-off-by: Sean Cross <sean@xobs.io>
Add an initial reimplementation of the gdb-stub using the gdbstub
backing crate.

This supports process enumeration and basic thread investigation.
Processes will pause when the debugger is inspecting them. Memory and
registers can be investigated. Threads can be listed and selected.

It does not yet support breakpoints.

Signed-off-by: Sean Cross <sean@xobs.io>
This adds a basic gdb server. Currently, it is possible to list
processes and inspect memory and threads. It is not yet possible to
actually interact with code (e.g. no breakpoints and no single
stepping).

Signed-off-by: Sean Cross <sean@xobs.io>
These calls should be possible to make during interrupts.

Signed-off-by: Sean Cross <sean@xobs.io>
Disable IRQs during destruction, but only if constructed with
`GdbUart::new()`. This allows characters to buffer when the GDB bridge
is destroyed while avoiding the double-free error we were getting from
creating a new UART in the IRQ handler.

Signed-off-by: Sean Cross <sean@xobs.io>
When debugging an invalid process (e.g. not yet run, terminated, etc.)
return an error rather than silently continuing.

Signed-off-by: Sean Cross <sean@xobs.io>
Implement the basics required to get "extended mode" working. This is
waiting on upstream to fix issues, but should allow us to attach to
processes once it works.

Signed-off-by: Sean Cross <sean@xobs.io>
Since gdbstub is able to do this, break individual features into their
own files.

Signed-off-by: Sean Cross <sean@xobs.io>
Many of the devicse have changed from `uint` to `ulong`, so forward-port
the peripherals to work with the new format.

Signed-off-by: Sean Cross <sean@xobs.io>
Debug handling was not correct, and didn't work at all. Additionally,
interrupt masking didn't work.

Fix these problems.

Signed-off-by: Sean Cross <sean@xobs.io>
The gdb server runs on an additional serial port and provides
process-level debugging to the kernel.

Signed-off-by: Sean Cross <sean@xobs.io>
The riscv target requires stepi to be implemented, which we now do.

Signed-off-by: Sean Cross <sean@xobs.io>
@xobs
Copy link
Member Author

xobs commented Apr 6, 2023

I rebased onto main, and fixed an issue with the build where the guardrail wasn't properly implemented.

Resume the debugged process when the debugger disconnects.

Signed-off-by: Sean Cross <sean@xobs.io>
@bunnie
Copy link
Member

bunnie commented Apr 8, 2023

We should probably put the documentation on "building" and "usage" somewhere more permanent than in this issue...

@xobs
Copy link
Member Author

xobs commented Apr 8, 2023

Agreed -- I just wanted to have it tested on real hardware first.

I think there should be an xtask switch to add the gdb-stub feature flag to the kernel at some point, assuming it works.

@bunnie bunnie merged commit 9bb574c into main Apr 8, 2023
@xobs xobs deleted the gdb-stub-redux branch April 8, 2023 08:52
@bunnie
Copy link
Member

bunnie commented Apr 8, 2023

i've added a gdb-stub feature flag on my local tree for xtask, but, i'm having some technical difficulties getting a proper version of gdb installed on the raspberry pi (my SD card has worn out and is emitting write errors).

It'll take a while to re-image the disk, and then I'll be able to test this.

@xobs
Copy link
Member Author

xobs commented Apr 8, 2023

You ought to be able to use gdb-multiarch from your distro. Alternately, the arm release from https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/tag/v12.2.0-3 should work.

@bunnie
Copy link
Member

bunnie commented Apr 8, 2023

yah the problem isn't the distro. it's that the card has hit its write life limit and is hanging as i extract it.

@xobs
Copy link
Member Author

xobs commented Apr 8, 2023

There's a delightful Python program from stack overflow to hook a serial port up to a TCP socket, so you can use your build machine:

#!/usr/bin/python

import socket
import sys
import serial

#open serial port
ser = serial.Serial('/dev/ttyAMA0', 115200, timeout=0)
#create socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

#bond to the port. Don't use localhost to accept external connections
server_address = ('', 2105)
print('starting up on {} port {}'.format(*server_address))
sock.bind(server_address)

#listen
sock.listen(1)

#loop
while True:
    #waits for a new connection
    print('waiting for a connection')
    connection, client_address = sock.accept()
    try:
        print('connection from', client_address)
        #continously send from serial port to tcp and viceversa
        connection.settimeout(0.1)
        while True:
            try:
                data = connection.recv(16)
                if data == '': break
                ser.write(data)
            except KeyboardInterrupt:
                connection.close()
                sys.exit()
            except Exception as e:
                pass
            received_data = ser.read(ser.inWaiting())
            connection.sendall(received_data)
    except Exception as e:
        print e

    finally:
        #clean up connection
        connection.close()

https://stackoverflow.com/a/64394570

@bunnie
Copy link
Member

bunnie commented Apr 8, 2023

that's a neat trick, maybe it should be documented somewhere where it can be discovered by others?

@bunnie
Copy link
Member

bunnie commented Apr 8, 2023

woo more yak shaving. the card is now renewed to a new one, but my libc is too old for the xpack toolchain....it'll be another few hours while the dist-upgrade runs.

@bunnie
Copy link
Member

bunnie commented Apr 8, 2023

Alright, got things upgraded and tried running it. It sort of works, but the connection seems unstable. This is what I get on a "good" run:

pi@betrusted-dev:~/code/betrusted-scripts $ riscv-none-elf-gdb -ex 'tar ext /dev/ttyS0' -ex 'attach 27' vault
GNU gdb (xPack GNU RISC-V Embedded GCC armv7l) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=armv7l-unknown-linux-gnueabihf --target=riscv-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
vault: No such file or directory.
Remote debugging using /dev/ttyS0
Ignoring packet error, continuing...
warning: unrecognized item "
> q
> S
> u
> p
Printing processes
Process 1 state: Running(100)  TID: 0  Memory mapping: (satp: 0x804409a0, mode: 1, ASID: 1, PPN: 409a0000) conns:0/32 kernel
Process 2 state: Sleeping  TID: 2  Memory mapping: (satp: 0x80840a56, mode: 1, ASID: 2, PPN: 40a56000) conns:5/32 xous-ticktimer
Process 3 state: Sleeping  TID: 3  Memory mapping: (satp: 0x80c40a4f, mode: 1, ASID: 3, PPN: 40a4f000) conns:1/32 xous-log
Process 4 state: Sleeping  TID: 2  Memory mapping: (satp: 0x81040a48, mode: 1, ASID: 4, PPN: 40a48000) conns:2/32 xous-names
Process 5 state: Sleeping  TID: 0  Memory mapping: (satp: 0x81440a41, mode: 1, ASID: 5, PPN: 40a41000) conns:21/32 xous-susres
Process 6 state: Sleeping  TID: 2  Memory mapping: (satp: 0x81840a3a, mode: 1, ASID: 6, PPN: 40a3a000) conns:7/32 graphics-server
Process 7 state: Sleeping  TID: 2  Memory mapping: (satp: 0x81c40a33, mode: 1, ASID: 7, PPN: 40a33000) conns:10/32 keyboard
Process 8 state: Sleeping  TID: 2  Memory mapping: (satp: 0x82040a2c, mode: 1, ASID: 8, PPN: 40a2c000) conns:8/32 spinor
Process 9 state: Sleeping  TID: 2  Memory mapping: (satp: 0x82440a25, mode: 1, ASID: 9, PPN: 40a25000) conns:10/32 llio
Process 10 state: Sleeping  TID: 2  Memory mapping: (satp: 0x82840a1e, mode: 1, ASID: 10, PPN: 40a1e000) conns:9/32 com
Process 11 state: Sleeping  TID: 4  Memory mapping: (satp: 0x82c40a17, mode: 1, ASID: 11, PPN: 40a17000) conns:17/32 dns
Process 12 state: Sleeping  TID: 2  Memory mapping: (satp: 0x83040a10, mode: 1, ASID: 12, PPN: 40a10000) conns:21/32 gam
Process 13 state: Sleeping  TID: 2  Memory mapping: (satp: 0x83440a09, mode: 1, ASID: 13, PPN: 40a09000) conns:7/32 ime-frontend
Process 14 state: Sleeping  TID: 2  Memory mapping: (satp: 0x83840a02, mode: 1, ASID: 14, PPN: 40a02000) conns:8/32 codec
Process 15 state: Sleeping  TID: 2  Memory mapping: (satp: 0x83c409fb, mode: 1, ASID: 15, PPN: 409fb000) conns:6/32 modals
Process 16 state: Sleeping  TID: 2  Memory mapping: (satp: 0x840409f4, mode: 1, ASID: 16, PPN: 409f4000) conns:16/32 root-keys
Process 17 state: Sleeping  TID: 2  Memory mapping: (satp: 0x844409ed, mode: 1, ASID: 17, PPN: 409ed000) conns:5/32 trng
Process 18 state: Sleeping  TID: 2  Memory mapping: (satp: 0x848409e6, mode: 1, ASID: 18, PPN: 409e6000) conns:6/32 sha2
Process 19 state: Sleeping  TID: 3  Memory mapping: (satp: 0x84c409df, mode: 1, ASID: 19, PPN: 409df000) conns:6/32 engine-25519
Process 20 state: Sleeping  TID: 2  Memory mapping: (satp: 0x850409d8, mode: 1, ASID: 20, PPN: 409d8000) conns:5/32 jtag
Process 21 state: Sleeping  TID: 2  Memory mapping: (satp: 0x854409d1, mode: 1, ASID: 21, PPN: 409d1000) conns:29/32 status
Process 22 state: Sleeping  TID: 5  Memory mapping: (satp: 0x858409ca, mode: 1, ASID: 22, PPN: 409ca000) conns:16/32 pddb
Process 23 state: Sleeping  TID: 2  Memory mapping: (satp: 0x85c409c3, mode: 1, ASID: 23, PPN: 409c3000) conns:10/32 usb-device-xous
Process 24 state: Sleeping  TID: 14  Memory mapping: (satp: 0x860409bc, mode: 1, ASID: 24, PPN: 409bc000) conns:17/32 net
Process 25 state: Sleeping  TID: 2  Memory mapping: (satp: 0x864409b5, mode: 1, ASID: 25, PPN: 409b5000) conns:3/32 ime-plugin-shell
Process 26 state: Sleeping  TID: 2  Memory mapping: (satp: 0x868409ae, mode: 1, ASID: 26, PPN: 409ae000) conns:18/32 shellchat
Process 27 state: Sleeping  TID: 2  Memory mapping: (satp: 0x86c409a7, mode: 1, ASID: 27, PPN: 409a7000) conns:18/32 vault
" in "qSupported" response
^CQuit
Don't know how to attach.  Try "help target".
(gdb) quit

Of course in the above trial I had the path of the executable wrong, but at least it did something. More typically, I get something like this:

pi@betrusted-dev:~/code/betrusted-scripts $ riscv-none-elf-gdb -ex 'tar ext /dev/ttyS0' -ex 'attach 27' ~/vault
GNU gdb (xPack GNU RISC-V Embedded GCC armv7l) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=armv7l-unknown-linux-gnueabihf --target=riscv-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pi/vault...
Remote debugging using /dev/ttyS0
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout
Don't know how to attach.  Try "help target".
(gdb)

I wonder if there are flow control issues possibly? I guess there are no RTS/CTS pins on this interface, but basically if the kernel is held up you're going to lose data. The FIFO is only 16 bytes deep on the Tx/Rx queues on the UARTs, so if you write more than that and the kernel is busy with something else, you'll end up dropping bytes.

@bunnie bunnie mentioned this pull request Apr 8, 2023
@xobs
Copy link
Member Author

xobs commented Apr 8, 2023

That looks like you might be connecting to the kernel debug port? Isn't there a command to switch the serial port to be the app_uart?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants