Problem of debugging rust applications

[raboserg]

Hello!
I have problem of debugging rust applications.

The problem with rust was noticed after the build and upgrade of GDB to 14.2.0 version.

The stack trace:

pwndbg> b 2
Breakpoint 1 at 0x8934: file src/main.rs, line 2.
pwndbg> r
Starting program: /target/debug/asm
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, asm::main () at src/main.rs:2
19 println!("Hello, world!");

Warning: the current language does not match this frame.
Traceback (most recent call last):
  File "/GDB/pwndbg/pwndbg/commands/__init__.py", line 196, in __call__
    return self.function(*args, **kwargs)
  File "/GDB/pwndbg/pwndbg/commands/__init__.py", line 356, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/GDB/pwndbg/pwndbg/commands/context.py", line 406, in context
    func(
  File "/GDB/pwndbg/pwndbg/commands/context.py", line 514, in context_regs
    regs = get_regs()
  File "/GDB/pwndbg/pwndbg/commands/context.py", line 606, in get_regs
    desc = pwndbg.chain.format(value)
  File "/GDB/pwndbg/pwndbg/chain.py", line 134, in format
    symbol = pwndbg.gdblib.symbol.get(link) or None
  File "/GDB/pwndbg/pwndbg/lib/cache.py", line 141, in decorator
    value = func(*a, **kw)
  File "/GDB/pwndbg/pwndbg/gdblib/symbol.py", line 97, in get
    result = gdb.execute("info symbol %#x" % int(address), to_string=True, from_tty=False)
gdb.error: That operation is not available on integers of more than 8 bytes.

If that is an issue, you can report it on https://github.com/pwndbg/pwndbg/issues
(Please don't forget to search if it hasn't been reported before)
To generate the report and open a browser, you may run bugreport --run-browser
PS: Pull requests are welcome

Steps to reproduce

1. touch main.rs
2. insert this code into main.rs

fn main() {
    println!("Hello, world!");
}

3. rustc -g ./main.rs
4. gdb ./main
5. l
6. b 2
7. r

Gdb session history:

l
b 2
r

My setup

Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.29
OS: Ubuntu 20.04.6 LTS
OS ABI: #111~20.04.1-Ubuntu SMP Mon Mar 11 15:44:43 UTC 2024
Architecture: x86_64
Endian: little
Charset: utf-8
Width: 211
Height: 51
Gdb:      14.2
Python:   3.8.10 (default, Nov 22 2023, 10:22:35)  [GCC 9.4.0]
Pwndbg:   2024.02.14 build: a75b1502
Capstone: 5.0.1280
Unicorn:  2.0.1
This GDB was configured as follows:
   configure --host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu
	     --with-auto-load-dir=$debugdir:$datadir/auto-load
	     --with-auto-load-safe-path=$debugdir:$datadir/auto-load
	     --with-expat
	     --with-gdb-datadir=/usr/local/share/gdb (relocatable)
	     --with-jit-reader-dir=/usr/local/lib/gdb (relocatable)
	     --without-libunwind-ia64
	     --without-lzma
	     --without-babeltrace
	     --without-intel-pt
	     --without-xxhash
	     --with-python=/usr
	     --with-python-libdir=/usr/lib
	     --without-debuginfod
	     --with-curses
	     --without-guile
	     --without-amd-dbgapi
	     --disable-source-highlight
	     --enable-threading
	     --enable-tui
	     --without-system-readline
	     --with-separate-debug-dir=/usr/local/lib/debug (relocatable)

("Relocatable" means the directory can be moved with the GDB installation
tree, and GDB will still find it.)

Such problems do not exist with C.
Maybe no support for rust, but before everything was fine.

[disconnect3d]

Hi, thanks for reporting this.

I believe we do set language C or sth like that somewhere in the code and that is probably causing the issue.

Also this is a weird error, how da hell is this int bigger than 8 bytes? :)

    result = gdb.execute("info symbol %#x" % int(address), to_string=True, from_tty=False)
gdb.error: That operation is not available on integers of more than 8 bytes.

Need to investigate it :)

[disconnect3d]

Haven't debugged it yet, need to run on GDB 14.x... perhaps our release package has this, heh.

[dmur1]

i've just run into this also 🙃

Description

crash when b on hello_cargo::main then r

Steps to reproduce

$ cargo new hello_cargo
$ cat src/main.rs
fn main() {
println!("Hello, world!");
}
$ cargo build
$ gdb ./target/debug/hello_cargo

b hello_cargo::main
r

Gdb session history:

b hello_cargo::main
r
set execption-verbose on
set exception-verbose on
r
bugreport help

My setup

Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.38
OS: Ubuntu 23.10
OS ABI: #26-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar  5 21:19:28 UTC 2024
Architecture: x86_64
Endian: little
Charset: utf-8
Width: 142
Height: 68
Gdb:      14.0.50.20230907-git
Python:   3.11.6 (main, Oct  8 2023, 05:06:43) [GCC 13.2.0]
Pwndbg:   2024.02.14 build: 7828ec5
Capstone: 5.0.1280
Unicorn:  2.0.1
This GDB was configured as follows:
   configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu
	    --with-auto-load-dir=$debugdir:$datadir/auto-load
	    --with-auto-load-safe-path=$debugdir:$datadir/auto-load
	    --with-expat
	    --with-gdb-datadir=/usr/share/gdb (relocatable)
	    --with-jit-reader-dir=/usr/lib/gdb (relocatable)
	    --without-libunwind-ia64
	    --with-lzma
	    --with-babeltrace
	    --with-intel-pt
	    --with-xxhash
	    --with-python=/usr (relocatable)
	    --with-python-libdir=/usr/lib (relocatable)
	    --with-debuginfod
	    --with-curses
	    --without-guile
	    --without-amd-dbgapi
	    --enable-source-highlight
	    --enable-threading
	    --enable-tui
	    --with-system-readline
	    --with-separate-debug-dir=/usr/lib/debug (relocatable)
	    --with-system-gdbinit=/etc/gdb/gdbinit
	    --with-system-gdbinit-dir=/etc/gdb/gdbinit.d

("Relocatable" means the directory can be moved with the GDB installation
tree, and GDB will still find it.)

pwndbg> version
Gdb: 14.0.50.20230907-git
Python: 3.11.6 (main, Oct 8 2023, 05:06:43) [GCC 13.2.0]
Pwndbg: 2024.02.14 build: 7828ec5
Capstone: 5.0.1280
Unicorn: 2.0.1

Breakpoint 1, hello_cargo::main () at src/main.rs:2
2 println!("Hello, world!");
Warning: the current language does not match this frame.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/user/pwndbg/pwndbg/commands/init.py:196 in call │
│ │
│ 193 │ │
│ 194 │ def call(self, *args: Any, **kwargs: Any) -> str | None: │
│ 195 │ │ try: │
│ ❱ 196 │ │ │ return self.function(*args, **kwargs) │
│ 197 │ │ except TypeError as te: │
│ 198 │ │ │ print(f"{self.function.name.strip()!r}: {self.function.doc.strip()}" │
│ 199 │ │ │ pwndbg.exception.handle(self.function.name) │
│ │
│ /home/user/pwndbg/pwndbg/commands/init.py:356 in _OnlyWhenRunning │
│ │
│ 353 │ @functools.wraps(function) │
│ 354 │ def _OnlyWhenRunning(*a: Any, **kw: Any) -> Optional[T]: │
│ 355 │ │ if pwndbg.gdblib.proc.alive: │
│ ❱ 356 │ │ │ return function(*a, **kw) │
│ 357 │ │ else: │
│ 358 │ │ │ print(f"{function.name}: The program is not being run.") │
│ 359 │ │ │ return None │
│ │
│ /home/user/pwndbg/pwndbg/commands/context.py:406 in context │
│ │
│ 403 │ │ │ result_settings[target].update(settings) │
│ 404 │ │ │ with target as out: │
│ 405 │ │ │ │ result[target].extend( │
│ ❱ 406 │ │ │ │ │ func( │
│ 407 │ │ │ │ │ │ target=out, │
│ 408 │ │ │ │ │ │ width=settings.get("width", None), │
│ 409 │ │ │ │ │ │ with_banner=settings.get("banner_top", True), │
│ │
│ /home/user/pwndbg/pwndbg/commands/context.py:514 in context_regs │
│ │
│ 511 │
│ 512 │
│ 513 def context_regs(target=sys.stdout, with_banner=True, width=None): │
│ ❱ 514 │ regs = get_regs() │
│ 515 │ if pwndbg.gdblib.config.show_compact_regs: │
│ 516 │ │ regs = compact_regs(regs, target=target, width=width) │
│ 517 │
│ │
│ /home/user/pwndbg/pwndbg/commands/context.py:606 in get_regs │
│ │
│ 603 │ │ │ desc = C.format_flags(value, bit_flags, pwndbg.gdblib.regs.last.get(reg, 0)) │
│ 604 │ │ │
│ 605 │ │ else: │
│ ❱ 606 │ │ │ desc = pwndbg.chain.format(value) │
│ 607 │ │ │
│ 608 │ │ result.append(f"{m}{regname} {desc}") │
│ 609 │ return result │
│ │
│ /home/user/pwndbg/pwndbg/chain.py:134 in format │
│ │
│ 131 │ # Colorize the chain │
│ 132 │ rest = [] │
│ 133 │ for link in chain: │
│ ❱ 134 │ │ symbol = pwndbg.gdblib.symbol.get(link) or None │
│ 135 │ │ if symbol: │
│ 136 │ │ │ symbol = f"{link:#x} ({symbol})" │
│ 137 │ │ rest.append(M.get(link, symbol)) │
│ │
│ /home/user/pwndbg/pwndbg/lib/cache.py:141 in decorator │
│ │
│ 138 │ │ │ │ if value is not _NOT_FOUND_IN_CACHE: │
│ 139 │ │ │ │ │ return value │
│ 140 │ │ │ │ │
│ ❱ 141 │ │ │ │ value = func(*a, **kw) │
│ 142 │ │ │ │ │
│ 143 │ │ │ │ # Sanity check: its not perfect and won't cover all cases like ([],) │
│ 144 │ │ │ │ # but it should be good enough │
│ │
│ /home/user/pwndbg/pwndbg/gdblib/symbol.py:97 in get │
│ │
│ 94 │ │ return "" │
│ 95 │ │
│ 96 │ # This sucks, but there's not a GDB API for this. │
│ ❱ 97 │ result = gdb.execute("info symbol %#x" % int(address), to_string=True, from_tty=Fals │
│ 98 │ │
│ 99 │ if not gdb_only and result.startswith("No symbol"): │
│ 100 │ │ address = int(address) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
error: That operation is not available on integers of more than 8 bytes.

[dmur1]

pwndbg> info symbol 0xffffffffffffff70
That operation is not available on integers of more than 8 bytes.

this is what is being attempted

[dmur1]

i think #2094 fixes things but i'm not really sure about the root cause - for some reason we try to query the symbol associated with some whacky addresses like 0xffffffffffffff70 when debugging rust bins and that breaks the regs in the context @disconnect3d thoughts?

[intrigus-lgtm]

This is indirectly caused by bminor/binutils-gdb@d760ae2 I believe.
gdb now parses 0xffffffffffffffff as i128 when in rust mode, while previously it would parse that as i64.
#2094 doesn't really fix the problem, only hides it.

When in rust mode, ptype 0xffffffffffffff70i64 would work, but this doesn't work in c mode, because i64 is not valid there.
I'm going to file a bug to the gdb developers.

I don't know how/whether we want to workaround this.
Those addresses can probably legitimately occur when debugging the kernel.

[disconnect3d]

@intrigus-lgtm thanks a lot for investigating this! Maybe we can detect those addresses in bugged GDB versions and add i64 at the end when we try to ask GDB for the symbol?

[intrigus-lgtm]

If gdb exposes whether we are in rust mode, then this should be possible by just always adding i64. No need to detect whether we have a problematic address.
Afaik there is no universal suffix that works across languages and the problem is rust specific as well.

[intrigus-lgtm]

Gdb bug: https://sourceware.org/bugzilla/show_bug.cgi?id=31565