SOLUTION FOUND - support remote debug from x86_64 based PC gdb client with gdbgui, to ARM arch gdbserver (multiarch)

[adowdy]

TLDR - I found a way to make gdbgui work for my usecase described! I'll put the fix at the bottom of the post

Is your feature request related to a problem? Please describe.
I really want to be able to use gdbgui frontend for a multiarch remote debugging usecase. My target system is running gdbserver on an ARM based device. If I try to run gdbgui then connect to gdbserver by any means (target remote 192.168.0.212:9091) I get the following issue:

target remote 192.168.0.212:9091

Remote debugging using 192.168.0.212:9091

warning: Architecture rejected target-supplied description

Reading /home/root/myRemoteTestApplicationExecutable from remote target...

warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Architecture of file not recognized.

So I believe this issue is that my desktop machine's gdb binary is not sufficient for multiarch debugging, so I did:
sudo apt-get install gdb-multiarch on my Ubuntu system. I tried it out just with commandline first, and it works like a charm:
gdb-multiarch --eval-command="target remote 192.168.0.212:9091"

Now though, I want to use gdb-multiarch as the debugger binary for gdbgui
gdbgui --gdb gdb-multiarch --gdb-args='--eval-command="target remote 192.168.0.212:9091"'
OR
gdbgui --gdb gdb-multiarch

• then use GUI to connect to gdbserver...

This method sadly also doesn't work, giving me a different error though:

gdbgui spawned subprocess with pid 28321 from executable gdb-multiarch.

gdb process 28321 is running for this tab

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

.... editing out some boilerplate GDB output...

set breakpoint pending on

warning: Architecture rejected target-supplied description

Reading /home/root/myRemoteTestApplicationExecutable from remote target...

warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Reading /home/root/myRemoteTestApplicationExecutable from remote target...

Reading symbols from target:/home/root/myRemoteTestApplicationExecutable...

done.

Reading /lib/ld-linux-armhf.so.3 from remote target...

Reading /lib/ld-linux-armhf.so.3 from remote target...

Reading /lib/ld-2.23.so from remote target...

Reading /lib/.debug/ld-2.23.so from remote target...

0x76fcfac0 in ?? () from target:/lib/ld-linux-armhf.so.3

Connected to remote target! Adding breakpoint to main, then continuing target execution.


Program

 received signal SIGSEGV, Segmentation fault.

0x00000000 in ?? ()

gdbgui noticed a signal was recieved (Segmentation fault, SIGSEGV).

If the program exited due to a fault, you can attempt to re-enter the state of the program when the fault 

occurred by clicking the below button.


Program

 received signal SIGSEGV, Segmentation fault.

0x00000000 in ?? ()

gdbgui noticed a signal was recieved (Segmentation fault, SIGSEGV).

If the program exited due to a fault, you can attempt to re-enter the state of the program when the fault 

occurred by clicking the below button.


Program terminated with signal 

SIGSEGV, Segmentation fault.

The program no longer exists.

/home/root/myRemoteTestApplicationExecutable: No such file or directory.

Cannot access memory at address 0x0

Describe the solution you'd like
I am not picky, but so long as there's a way to use gdbgui to debug a remote gdbserver that's actually running ARM arch instead of matching the desktop architecture would be an absolute dream! (I'd even take a solution from any similar tool to gdbgui, but this one is really quite nice).

Describe alternatives you've considered
The closest I've gotten (aside from just using gdb-multiarch commandline) is using ddd but it's ancient and buggy on my modern system.
ddd --eval-command="target remote $ADOLOCAL:9091" --debugger gdb-multiarch
This gives some sort of minimal GUI and I've been able to set breakpoints and step through (so it gets further than gdbgui at the moment) -- but it's really clunky, and freezes up constantly.

Additional context

TLDR Solution:

I was trying to rely on a prebuild gdb-multiarch from ubuntu apt repos, which didn't work. When I decided to download gdb and rebuild from source while configuring for arm-linux-gnuabi target arch.

Build method:

1. downloaded latest gdb source code
2. unzip it, go into folder

./configure  --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --target=arm-linux-gnuabi &&
make -j8 &&
sudo make install

4. now arm-linux-gnuabi-gdb is installed by default to /usr/local/bin ... but you can instead provide prefix=<path> to where you want it to install in ./configure script above

Using this, I was able to build a secondary copy of gdb called arm-linux-gnuabi-gdb which i could feed to gdbgui like this:

gdbgui -g arm-linux-gnuabi-gdb

Works great! This is leaps and bounds better than running gdb on commandline on my remote target.

Relevant link for building gdb:
https://sourceware.org/gdb/wiki/BuildingCrossGDBandGDBserver
http://www.brain-dump.org/blog/entry/138/Cross_Arch_Remote_Debugging_with_gdb_and_gdbserver

[E3V3A]

@nerael
Thanks for sharing this very helpful and useful info.

[cs01]

I updated gdbgui's docs to be easier to modify for the community. They are all hosted in this repository under docs/, and are just markdown files. This should be added to give it this workflow exposure. Thanks for writing it up!

[seanybaggins]

This solution is somewhat out of date. The API for gdbgui has changed.

gdbgui --version

0.14.0.2

trying to modify the tutorial here to match the current API.

Here is the command I ultimately end up running.

gdbgui --gdb-cmd 'arm-none-eabi-gdb -command=debug.gdb' target/thumbv7em-none-eabihf/debug/blinky-st

Here is the content of my debug.gdb file

# This connects to the GDB server running locally.
#   - for openocd, use port :3333
#   - for JLinkGDBServer, use port :2331
target remote :3333

# Due to https://github.com/rust-embedded/cortex-m-rt/issues/139,
#   we will get an infinite backtrace on a panic!(). Set a finite
#   limit to the backtrace to prevent the debugger falling into
#   an endless loop trying to read the backtrace
set backtrace limit 32

# Load the specified firmware onto the device
load

# Reset the target device before running (using JLinkGDBServer)
# monitor reset

# Reset the target device before running (using openocd)
monitor reset halt

# Begin running the program
continue

The gui launches but gdb does not load the microprocessor. I think it is because the script does not have the binary being passed to it.

GDB input terminal

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=arm-none-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.


debug.gdb:13: Error in sourced command file:
No executable file specified.
Use the "file" or "exec-file" command.
New UI allocated

[CalleOlsen]

If you want to make a target with cmake that runs the gdbgui
When doing make gui_debug for example.
Here is how i did it.
Create a cmake file e.g
debug_target.cmake

function(make_openocd_target)
  cmake_parse_arguments(TARGET "" "NAME;OPENOCD_PATH;OPENOCD_INTERFACE;OPENOCD_TARGET" "" ${ARGN})

  get_target_property(BIN_DIR ${TARGET_NAME} BINARY_DIR)
  add_custom_target(gui_debug
    COMMAND bash -c "${CMAKE_SOURCE_DIR}/tools/openocd/gdbgui.sh \
${CMAKE_SOURCE_DIR}/tools/openocd/run_openocd.sh \
${TARGET_OPENOCD_PATH} \
${TARGET_OPENOCD_INTERFACE} \
${TARGET_OPENOCD_TARGET} \
${BIN_DIR}/${TARGET_NAME} \
${CMAKE_SOURCE_DIR}/tools/gdb/gdbgui.gdb \
${CMAKE_SOURCE_DIR}"
 VERBATIM
)
endfunction(make_openocd_target)

The above code will call this small script
Which create a tmux-session and split it in 2.
One part is starting openocd, and the other one gdbgui.

This is the gdbgui.sh script

#!/usr/bin/env bash


RUN_EXEC_CMD=$1
TARGET_OPENOCD_PATH=$2
TARGET_OPENOCD_INTERFACE=$3
TARGET_OPENOCD_TARGET=$4
TARGET_BIN=$5
GDB_CFG=$6
PRJ_DIR=$7

cmd="gdbgui -g \"arm-none-eabi-gdb ${TARGET_BIN} -ix ${GDB_CFG}\" --project ${PRJ_DIR}"

echo "${cmd}"

tmux new-session \
"${RUN_EXEC_CMD} -o \
 ${TARGET_OPENOCD_PATH} -i\
 ${TARGET_OPENOCD_INTERFACE} -t \
 ${TARGET_OPENOCD_TARGET}" \; \
 split-window -h -p 50 " ${cmd} " \;

The above script will call run_openocd.sh
Which is abit more:

#!/bin/bash
function showHelp {
    cat << EOF
Helper program to run openocd
 -h -- help (this)
 -o -- openocd path , where openocd is compiled, just the path
 -t -- target (e.g rp2040)
 -i -- interface (e.g picoprobe)
 -e -- Elf file to be added to the target, using the interface
EOF

}

function with_elf {

    (
        echo "${OPENOCD_BIN} \
-s ${OPENOCD_TCL} \
-f ${OPENOCD_TCL}/interface/${INTERFACE}.cfg \
-f ${OPENOCD_TCL}/target/${TARGET}.cfg \
-c \"program ${ELF} verify reset init exit\"
"
        ${OPENOCD_BIN} \
            -s ${OPENOCD_TCL} \
            -f ${OPENOCD_TCL}/interface/${INTERFACE}.cfg \
            -f ${OPENOCD_TCL}/target/${TARGET}.cfg \
            -c "program ${ELF} reset  exit"

    )


}

function without_elf {

    (
        cd ${OPENOCD_PATH}
        echo "${OPENOCD_BIN} \
-s ${OPENOCD_TCL} \
-f ${OPENOCD_TCL}/interface/${INTERFACE}.cfg \
-f ${OPENOCD_TCL}/target/${TARGET}.cfg"
        out=$(${OPENOCD_BIN} -s ${OPENOCD_TCL} -f ${OPENOCD_TCL}/interface/${INTERFACE}.cfg -f ${OPENOCD_TCL}/target/${TARGET}.cfg)
    )
}

function exec_openocd {

    if [[ -z "${ELF}" ]]; then
        without_elf
    else
        with_elf
    fi



}


INTERFACE="picoprobe"
TARGET="rp2040"
ELF=
OPENOCD_BIN=

while getopts "ho:t:i:e:" args
do
    case $args in
        h)
            showHelp
            echo "FSISI"
            exit 0
            ;;
        o)
            echo "Setting ocdpath=${OPTARG}"
            export OPENOCD_PATH=${OPTARG}
            ;;
        i)
            echo "Setting inteface=${OPTARG}"
            INTERFACE=${OPTARG%\.cfg}
            echo ${INTERFACE}
            ;;
        t)
            echo "Setting target=${OPTARG}"
            TARGET=${OPTARG%\.cfg}
            ;;
        e)
            echo "Flashing elf=${OPTARG}"
            ELF=${OPTARG}
            ;;
    esac

done



if [[ -z "${ELF}" ]]; then

    echo "Not using elf"
else
    echo "Using ${ELF}"
fi
#Setting the openOCD binary path
OPENOCD_SRC=${OPENOCD_PATH}/src
OPENOCD_BIN=${OPENOCD_SRC}/openocd
OPENOCD_TCL=${OPENOCD_PATH}/tcl



exec_openocd

The last task is to actually create the target that is related to you binary in Cmake.
For example:

set(TARGET_BIN "MYELF")
target_link_libraries(${TARGET_BIN} 
.
.
)

make_openocd_target(NAME ${TARGET_BIN}
  OPENOCD_PATH ${OPENOCD_PATH}
  OPENOCD_INTERFACE ${OPENOCD_INTERFACE}
  OPENOCD_TARGET stm32f4x
)

I don't know if that made any sense.. But at least it might give you an idea of how to do it ... The above code is a bit of a (late afternoon) hack , but it works...