Skip to content

Use addr2line for backtraces#557

Merged
mkeeter merged 5 commits intomasterfrom
mkeeter/better-backtrace
Aug 12, 2025
Merged

Use addr2line for backtraces#557
mkeeter merged 5 commits intomasterfrom
mkeeter/better-backtrace

Conversation

@mkeeter
Copy link
Copy Markdown
Contributor

@mkeeter mkeeter commented Aug 8, 2025

(staged on top of #556)

This PR switches stack printing to use addr2line. Stack unwinding remains the same, using our own code, but now we pass each PC in the stack to addr2line to get a more precise backtrace.

Before

ID TASK                       GEN PRI STATE
24 hf                           0   5 wait: send to spartan7_loader/gen0
   |
   +--->  0x24056d28 0x0801ec7e userlib::sys_send_stub
                     @ /hubris/sys/userlib/src/lib.rs:157
          0x24056fa0 0x0801bc60 userlib::sys_send
                     @ /hubris/sys/userlib/src/lib.rs:121
          0x24056fa0 0x0801bc78 main
                     @ /hubris/drv/cosmo-hf/src/main.rs:47

After

ID TASK                       GEN PRI STATE
24 hf                           0   5 wait: send to spartan7_loader/gen0
   |
   +--->  0x24056d28 0x0801ec7e userlib::sys_send_stub
                     @ /hubris/sys/userlib/src/lib.rs:197:13
          0x24056fa0 0x0801bc78 userlib::sys_send
                     @ /hubris/sys/userlib/src/lib.rs:138:14
          0x24056fa0 0x0801bc78 drv_spartan7_loader_api::Spartan7Loader::ping
                     @ /build/drv-spartan7-loader-api-81382492354c304f/out/client_stub.rs:50:29
          0x24056fa0 0x0801bc78 drv_spartan7_loader_api::Spartan7Loader::get_token
                     @ /hubris/drv/spartan7-loader-api/src/lib.rs:19:9
          0x24056fa0 0x0801bc78 main
                     @ /hubris/drv/cosmo-hf/src/main.rs:54:38

There are three noticeable improvements:

  • We get both lines and columns
  • The lines represent the actual call site, not the beginning of the function!
  • We get a more accurate set of calls for inlined functions (note that main, get_token, ping, and sys_send are all at the same stack frame!)

This changes a zillion files in the test suite, because we now use addr2line data instead of unwind frames when possible.

One noticeable change is that we now show the same PC address (starting with 0x080... in the example above) for all frames in an inlined stack. Previously, we used the addr of each symbol, which is a slightly different behavior; I think it represents the start of the inlined function.

I did a spot check, and the new results seem reasonable.

Here's an example diff:

    SP = 0x20002b70 <- i2c_driver: 0x20002800+0x370
         |
         +--->  0x20002b80 0x08015ac8 userlib::sys_irq_control_stub
-               0x20002c00 0x080148e2 drv_stm32h7_i2c::I2cController::write_read
-               0x20002c00 0x08014658 drv_stm32h7_i2c_server::main::{{closure}}
-               0x20002c00 0x08014658 userlib::hl::recv_without_notification::{{closure}}
-               0x20002c00 0x080142d4 userlib::hl::recv
-               0x20002c00 0x080142d4 userlib::hl::recv_without_notification
+               0x20002c00 0x08014920 drv_stm32h7_i2c::I2cController::write_read
+               0x20002c00 0x08014920 drv_stm32h7_i2c_server::main::{{closure}}
+               0x20002c00 0x08014920 userlib::hl::recv_without_notification::{{closure}}
+               0x20002c00 0x08014920 userlib::hl::recv
+               0x20002c00 0x08014920 userlib::hl::recv_without_notification
                0x20002c00 0x08014920 main
                0x20002c00 0x0801404e _start

I extracted the I2C driver code, then disassembled it to see what these actually addresses represent.

humility -d humility-bin/tests/cmd/cores/hubris.core.kiowa.4 extract elf/task/i2c_driver > i2c
arm-none-eabi-objdump -d i2c | bat -p -ls

The new dissassembly puts everything at 0x8014920, which is

 801491c:   f7ff fba8   bl  8014070 <_ZN4core3ops8function6FnOnce9call_once17hb574483a6f289883E>
 8014920:   9908        ldr r1, [sp, #32]

That certainly is a function call! (to be pedantic, it's the return address after the function call)

The old trace includes the following addresses:

 80142cc:	f10d 0b58 	add.w	fp, sp, #88	; 0x58
 80142d0:	f64f 74ff 	movw	r4, #65535	; 0xffff
 80142d4:	4630      	mov	r0, r6 ; <-- here
 80142d6:	2104      	movs	r1, #4
 8014654:	2400      	movs	r4, #0
 8014656:	9805      	ldr	r0, [sp, #20]
 8014658:	6885      	ldr	r5, [r0, #8] <-- here
 801465a:	6841      	ldr	r1, [r0, #4]
 80148da:	f040 0001 	orr.w	r0, r0, #1
 80148de:	6020      	str	r0, [r4, #0]
 80148e0:	e51f      	b.n	8014322 <main+0x202>
 80148e2:	2010      	movs	r0, #16 <-- here
 80148e4:	2603      	movs	r6, #3
 80148e6:	e7df      	b.n	80148a8 <main+0x788>
 80148e8:	2010      	movs	r0, #16

None of these are function calls or values of the PC on the stack. They may be the beginning of inlined functions, but it's awfully hard to tell what's going on, so I think it's fine to discard this information.

@mkeeter mkeeter force-pushed the mkeeter/better-backtrace branch from 62cfdde to 98490b4 Compare August 8, 2025 20:29
Copy link
Copy Markdown
Contributor

@labbott labbott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for doing this

@mkeeter mkeeter force-pushed the mkeeter/bump-deps branch from 4f42dfa to a00701b Compare August 12, 2025 17:54
@mkeeter mkeeter force-pushed the mkeeter/better-backtrace branch from 98490b4 to 3f9d1d2 Compare August 12, 2025 17:54
@mkeeter mkeeter force-pushed the mkeeter/bump-deps branch from a00701b to b6ee1b8 Compare August 12, 2025 18:11
@mkeeter mkeeter force-pushed the mkeeter/better-backtrace branch from 3f9d1d2 to 6a17ab8 Compare August 12, 2025 18:11
Base automatically changed from mkeeter/bump-deps to master August 12, 2025 18:38
@mkeeter mkeeter force-pushed the mkeeter/better-backtrace branch from 6a17ab8 to c8eba88 Compare August 12, 2025 18:38
@mkeeter mkeeter force-pushed the mkeeter/better-backtrace branch from c8eba88 to 387b862 Compare August 12, 2025 19:07
@mkeeter mkeeter merged commit 9cf9d76 into master Aug 12, 2025
11 checks passed
@mkeeter mkeeter deleted the mkeeter/better-backtrace branch August 12, 2025 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants