DWARF: linkage_name does not include hash, does not match any symbol

[m4b]

So this is going to be a long issue, but the gist is, to put it semi dramatically, is that I think that all of rust debugging info might be slightly broken, but workable enough for say gdb, that it has gone unnoticed. At the very least, I think:

1. no_mangle statics are definitely broken no_mangle static symbols have improper debug info #33172
2. there are some major discrepancies between rust and c++ dwarf output that should be resolved, w.r.t. the DWARF linkage_name

/cc @philipc @fitzgen @tromey @rkruppe @michaelwoerister

Discussion

I've created some test files in rust and c++, and also grepped the binaries for dwarf dies and also symbol table values, which I explain below.

Repro / Test files

#[derive(Debug)]
pub struct Foo {
    x: u64,
    y: i32,
}

#[no_mangle]
pub static TEST: Foo = Foo { x: 0xdeadbeef, y: -55 };

pub static TEST2: Foo = Foo { x: 0xbeefdead, y: -55 };

fn deadbeef() {
    println!("TEST: {:?} - {:?}", TEST, TEST2);
}

pub fn main() {
    deadbeef()
}

and an approximating C++ file:

#include<cstdio>
#include<cstdint>

namespace test {

  struct Foo {
    uint64_t x;
    int64_t y;
  };

  Foo TEST = { .x = 0xdeadbeef, .y = -55 };
}

test::Foo TEST = { .x = 0xbeefdead, .y = -55 };

namespace test {
  void deadbeef() {
    printf("test::TEST: {0x%lx, %ld}\n", test::TEST.x, test::TEST.y);
    printf("TEST: {0x%lx, %ld}\n", TEST.x, TEST.y);
  }
}

int main() {
  test::deadbeef();
  return 0;
}

I've compiled the rust and c++ versions as follows:

rustc -g test.rs -o test
g++ -g -std=c++11 test.cpp -o test_cpp
clang++ -g -std=c++11 test.cpp -o test_cpp_clang

I will use the clang output for the c++ examples below, since it shares the same backend infrastructure, though the g++ does output the same.

Analysis

First I will show the dwarf values for TEST, TEST2, and the function deadbeef for the test binary, then the test_cpp_clang binary.

I would like to direct the reader's attention to the DW_AT_linkage_name field, and the corresponding linkage name it shows for each binary.

DW_TAG_variable [3]  
  DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000061] = "TEST")
  DW_AT_type [DW_FORM_ref4]	(cu + 0x0049 => {0x00000049})
  DW_AT_external [DW_FORM_flag_present]	(true)
  DW_AT_decl_file [DW_FORM_data1]	("/home/m4b/tmp/bad_debug/test.rs")
  DW_AT_decl_line [DW_FORM_data1]	(8)
  DW_AT_alignment [DW_FORM_udata]	(1)
  DW_AT_location [DW_FORM_exprloc]	(<0x9> 03 c0 65 05 00 00 00 00 00 )
  DW_AT_linkage_name [DW_FORM_strp]	( .debug_str[0x00000066] = "_ZN4test4TESTE")

DW_TAG_variable [6]  
  DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000075] = "TEST2")
  DW_AT_type [DW_FORM_ref4]	(cu + 0x0049 => {0x00000049})
  DW_AT_decl_file [DW_FORM_data1]	("/home/m4b/tmp/bad_debug/test.rs")
  DW_AT_decl_line [DW_FORM_data1]	(10)
  DW_AT_alignment [DW_FORM_udata]	(1)
  DW_AT_location [DW_FORM_exprloc]	(<0x9> 03 d0 65 05 00 00 00 00 00 )
  DW_AT_linkage_name [DW_FORM_strp]	( .debug_str[0x0000007b] = "_ZN4test5TEST2E")

DW_TAG_subprogram [7] *
  DW_AT_low_pc [DW_FORM_addr]	(0x00000000000073b0)
  DW_AT_high_pc [DW_FORM_data4]	(0x000000f7)
  DW_AT_frame_base [DW_FORM_exprloc]	(<0x1> 56 )
  DW_AT_linkage_name [DW_FORM_strp]	( .debug_str[0x000000d9] = "_ZN4test8deadbeefE")
  DW_AT_name [DW_FORM_strp]	( .debug_str[0x000000ec] = "deadbeef")
  DW_AT_decl_file [DW_FORM_data1]	("/home/m4b/tmp/bad_debug/test.rs")
  DW_AT_decl_line [DW_FORM_data1]	(12)

And now for the cpp version:

DW_TAG_variable [3]  
  DW_AT_name [DW_FORM_strp]     ( .debug_str[0x00000053] = "TEST")
  DW_AT_type [DW_FORM_ref4]     (cu + 0x0048 => {0x00000048})
  DW_AT_external [DW_FORM_flag_present] (true)
  DW_AT_decl_file [DW_FORM_data1]       ("/home/m4b/tmp/bad_debug/test.cpp")
  DW_AT_decl_line [DW_FORM_data1]       (11)
  DW_AT_location [DW_FORM_exprloc]      (<0x9> 03 30 10 20 00 00 00 00 00 )
  DW_AT_linkage_name [DW_FORM_strp]     ( .debug_str[0x0000008e] = "_ZN4test4TESTE")

DW_TAG_subprogram [6]  
  DW_AT_low_pc [DW_FORM_addr]   (0x0000000000000670)
  DW_AT_high_pc [DW_FORM_data4] (0x0000004c)
  DW_AT_frame_base [DW_FORM_exprloc]    (<0x1> 56 )
  DW_AT_linkage_name [DW_FORM_strp]     ( .debug_str[0x000002be] = "_ZN4test8deadbeefEv")
  DW_AT_name [DW_FORM_strp]     ( .debug_str[0x000002d2] = "deadbeef")
  DW_AT_decl_file [DW_FORM_data1]       ("/home/m4b/tmp/bad_debug/test.cpp")
  DW_AT_decl_line [DW_FORM_data1]       (17)
  DW_AT_external [DW_FORM_flag_present] (true)

DW_TAG_variable [9]  
  DW_AT_name [DW_FORM_strp]       ( .debug_str[0x00000053] = "TEST")
  DW_AT_type [DW_FORM_ref4]       (cu + 0x0048 => {0x00000048})
  DW_AT_external [DW_FORM_flag_present]   (true)
  DW_AT_decl_file [DW_FORM_data1] ("/home/m4b/tmp/bad_debug/test.cpp")
  DW_AT_decl_line [DW_FORM_data1] (14)
  DW_AT_location [DW_FORM_exprloc]        (<0x9> 03 40 10 20 00 00 00 00 00 )

The first thing to note is that for the rust, no_mangle static, TEST, it is given a linkage name:

DW_AT_linkage_name [DW_FORM_strp]	( .debug_str[0x00000066] = "_ZN4test4TESTE")

In contrast to the cpp version, which (correctly) has none. I believe the rust DIE that is emitted is outright incorrect, and is the cause of the issue in #33172

Note, although this issue was closed in favor of #32574 that issue does not no_mangle the static.

Unfortunately, that issue also noted (but did not seem to pursue further):

Now, this variable is not actually emitted. There's no ELF symbol for it.

which I believe may be the crux of the major problem at large here: the linkage_name on all non-mangled Rust DWARF DIEs references a non-existent symbol - I think this is at best highly unusual, and at worst problematic.

Missing Symbols

Considering only ELF at the moment, we can verify that for TEST, TEST2, and deadbeef, there is no symbol referenced by the linkage_name on the DIE:

565d0    LOCAL      OBJECT      _ZN4test5TEST217h8314c5b1b9028ef4E     0x10      .rodata(16)
 73b0    LOCAL      FUNC        _ZN4test8deadbeef17hc8e13bcb4738fc41E  0xf7      .text(14)
565c0    GLOBAL     OBJECT      TEST                                   0x10      .rodata(16)

You will note that the symbols include the symbol hash.

In contrast with the cpp version, the symbol name (including parameter types, see the v (for void) in _ZN4test8deadbeefEv) is identical to the linkage_name, as I think, expected:

201040    GLOBAL     OBJECT      TEST                                  0x10    .data(22)
   670    GLOBAL     FUNC        _ZN4test8deadbeefEv                   0x4c    .text(12)
201030    GLOBAL     OBJECT      _ZN4test4TESTE                        0x10    .data(22)

Debuggers

I would now like to present a debugging session (primarily in gdb) for the two binaries to attempt to illustrate some of the oddities that occur, and motivate why I think there is something wrong here, at the very least.

There are general ergonomic issues and other oddities that I think are surfacing because of the current debug info situation. I have used gdb to illustrate, as lldb is essentially non-functioning for me, and when it doesn't segfault, I cannot break on un-mangled names for the rust binaries.

GDB

First the rust binary:

(gdb) ptype TEST
No symbol 'TEST' in current context
(gdb) ptype test::TEST
No symbol 'test::TEST' in current context
(gdb) ptype test::TEST2
Reading in symbols for test.rs...done.
type = struct test::Foo {
  x: u64,
  y: i32,
}
(gdb) ptype test::deadbeef
type = fn ()
(gdb) whatis TEST
No symbol 'TEST' in current context
(gdb) whatis test::TEST
No symbol 'test::TEST' in current context
(gdb) whatis test::TEST2
type = test::Foo
(gdb) whatis test::deadbeef
type = fn ()
(gdb) info addr TEST
Symbol "TEST" is at 0x565c0 in a file compiled without debugging.
(gdb) info addr test::TEST
No symbol "test::TEST" in current context.
(gdb) info addr test::TEST2
Symbol "test::TEST2" is static storage at address 0x565d0.
(gdb) info addr test::deadbeef
Symbol "test::deadbeef" is a function at address 0x73b0.
(gdb) info addr _ZN4test5TEST217h8314c5b1b9028ef4E
Symbol "test::TEST2" is at 0x565d0 in a file compiled without debugging.
(gdb) info addr _ZN4test8deadbeef17hc8e13bcb4738fc41E
Symbol "test::deadbeef" is at 0x73b0 in a file compiled without debugging.

The last two are particularly troubling. This certainly looks like a direct consequence of mapping the unmangled, symbol name + hash, in the ELF symbol table to the mangled-without-hash linkage name in the DWARF DIE.

If we compare this to the exact same c++ debugging sequence, it is exactly what one would expect:

(gdb) ptype TEST
type = struct test::Foo {
    uint64_t x;
    int64_t y;
}
(gdb) ptype test::TEST
type = struct test::Foo {
    uint64_t x;
    int64_t y;
}
(gdb) ptype test::deadbeef
type = void (void)
(gdb) whatis TEST
type = test::Foo
(gdb) whatis test::TEST
type = test::Foo
(gdb) whatis test::deadbeef
type = void (void)
(gdb) info addr TEST
Symbol "TEST" is static storage at address 0x201040.
(gdb) info addr test::TEST
Symbol "test::TEST" is static storage at address 0x201030.
(gdb) info addr test::deadbeef
Symbol "test::deadbeef()" is a function at address 0x670.
(gdb) info addr _ZN4test4TESTE
Symbol "test::TEST" is static storage at address 0x201030.
(gdb) info addr _ZN4test8deadbeefEv
Symbol "test::deadbeef()" is a function at address 0x670.

LLDB

As of lldb with llvm 5.0.0 I cannot even auto-complete or break on non-mangled names, which further increases my suspicion something is wrong, and the fact that it segfaults occasionally when I auto-complete using test:: as a seed and the stack trace indicates its in the dwarf DIE parsing logic is equally suspicious:

Process 11803 (lldb) of user 1000 dumped core.
Stack trace of thread 11841:
#0  0x00007f4d2e9a4295 _ZN16DWARFCompileUnit6GetDIEEj (liblldb.so.3.8.0)
#1  0x00007f4d2e9a9f60 _ZN14DWARFDebugInfo6GetDIEERK6DIERef (liblldb.so.3.8.0)
#2  0x00007f4d2e9a4265 _ZN16DWARFCompileUnit6GetDIEEj (liblldb.so.3.8.0)
#3  0x00007f4d2e9ac6b0 _ZNK19DWARFDebugInfoEntry13GetAttributesEPK16DWARFCompileUnitN14DWARFFormValue14
#4  0x00007f4d2e9a4f89 _ZN16DWARFCompileUnit12IndexPrivateEPS_N4lldb12LanguageTypeERKN14DWARFFormValue1
#5  0x00007f4d2e9a454c _ZN16DWARFCompileUnit5IndexER9NameToDIES1_S1_S1_S1_S1_S1_S1_ (liblldb.so.3.8.0)
#6  0x00007f4d2e9cab60 _ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_
#7  0x00007f4d2e9caa1a _ZNSt13__future_base13_State_baseV29_M_do_setEPSt8functionIFSt10unique_ptrINS_12
#8  0x00007f4d31feedbf __pthread_once_slow (libpthread.so.0)
#9  0x00007f4d2e9ca5dc _ZNSt13__future_base11_Task_stateISt5_BindIFZN10TaskRunnerIjE7AddTaskIRZN15Symbo
#10 0x00007f4d2ec2fd15 _ZN12_GLOBAL__N_112TaskPoolImpl6WorkerEPS0_ (liblldb.so.3.8.0)
#11 0x00007f4d2c970a6f execute_native_thread_routine (libstdc++.so.6)
#12 0x00007f4d31fe708a start_thread (libpthread.so.0)
#13 0x00007f4d2c3de47f __clone (libc.so.6)

Solutions

I believe 1. is the first that should be attempted, but I am not an expert in the compiler internals.

1. Have the dwarf name mangler include the symbol hash, specifically I think it needs to be appended right about here: https://github.com//m4b/rust/blob/b15a8eafcd7e50af116d398c1ac3c5a0504c0270/src/librustc_trans/debuginfo/namespace.rs#L47
2. Output non-hash versions of symbols into the symbol table that point to the same address, etc., but which reference the non-hash name in the string table. This is a hack imho, and I've already tried this via binary editing and it did not seem to have any noticeable effect.

Eventually, I think that the final, correct solution imho is to exactly mirror the dwarf output of both the gcc and clang backends, which means:

1. Making the linkage_name = the symbol table name
2. Adding parameters, etc., into the symbol table name?

Final Considerations

I believe I've also found some weirdness w.r.t. other symbol names, specifically:

1. locally scoped statics,
2. impls of traits with certain annotations

examples of which are:

_ZN3std10sys_common9backtrace11log_enabled7ENABLED17hc187c5b3618ccb2eE.0.0
_ZN61_$LT$alloc..heap..Heap$u20$as$u20$alloc..allocator..Alloc$GT$3oom17h28fb525969c57bd8E

at lines:

1. https://github.com//m4b/rust/blob/b15a8eafcd7e50af116d398c1ac3c5a0504c0270/src/libstd/sys_common/backtrace.rs#L148 (0.0 appended to a mangled name is not a valid mangled name afaik)
2. https://github.com//m4b/rust/blob/b15a8eafcd7e50af116d398c1ac3c5a0504c0270/src/liballoc/heap.rs#L94-L100 (the symbol name appears to differ widly from the debug name)

, respectively.

But I think this is another story for another time ;)

[m4b]

Oh, I would also like to give a shout-out to @philipc's ddbug, it's a super useful tool (already!!) and greatly aided me when perusing the dwarf output, and independently verifying details :)

[philipc]

Making the linkage_name = the symbol table name

I agree that this should done. It needs more than just including the hash though, since the DWARF linkage name seems to be missing other stuff (eg sanitize).

Adding parameters, etc., into the symbol table name?

Is this necessary if rust doesn't have overloading? The DWARF contains the parameters in other DIEs, so it's still available to the debugger.

[m4b]

I agree that this should done. It needs more than just including the hash though, since the DWARF linkage name seems to be missing other stuff (eg sanitize).

Sure, I didn't even investigate more complicated examples; as I hinted at end, I've definitely seen pretty big divergence between symbol name and debug name. A proper solution unites them fully.

Is this necessary if rust doesn't have overloading? The DWARF contains the parameters in other DIEs, so it's still available to the debugger.

Yea you're right this probably isn't as important for rust/needed at all. I just wanted to illustrate it should be the unique symbol name, whatever that means for rust, whether its hash + other stuff, I'm not quite sure. For c++ it'll be the parameters and other stuff, etc., but the point, as above, is that its the same value in the dwarf linkage name.

[tromey]

See also bug #32925, which is about the same problem.

Making the linkage_name = the symbol table name

I agree with this plan; I think that's the intended purpose of the linkage name.

[m4b]

Someone mentioned it somewhere, maybe in irc, but I'll post it here, the dwarf name mangler also doesn't sanitize, and is basically out-of-sync with the actual name mangler.

Really it seems like there are two implementations of the name mangler at this point, and to illustrate, here is the linkage_name for https://github.com//m4b/rust/blob/590d1fa7b28776f4f7bba9b759a76f1ab5240f63/src/libcore/fmt/mod.rs#L305 when specialized to new<u64>:

DW_TAG_subprogram [15] *
  DW_AT_low_pc [DW_FORM_addr]       (0x0000000000000000)
  DW_AT_high_pc [DW_FORM_addr]      (0x0000000000000068)
  DW_AT_frame_base [DW_FORM_block1] (<0x01> 56 )
  DW_AT_MIPS_linkage_name [DW_FORM_strp]    ( .debug_str[0x000002f5] = "_ZN4core3fmt8{{impl}}8new<u64>E")
  DW_AT_name [DW_FORM_strp] ( .debug_str[0x000002ec] = "new<u64>")
  DW_AT_decl_file [DW_FORM_data1]   (0x02)
  DW_AT_decl_line [DW_FORM_data2]   (0x0129)
  DW_AT_type [DW_FORM_ref4] (cu + 0x0333 => {0x00000333})
  DW_AT_external [DW_FORM_flag]     (0x01)

or more pretty printed using ddbug:

fn core::fmt::ArgumentV1::new<u64>
        linkage name: _ZN4core3fmt8{{impl}}8new<u64>E
        symbol name: _ZN4core3fmt10ArgumentV13new17hc12d934f3b4cc6f4E
        address: 0x7230-0x7297
        size: 104
        return type:
                [16]    struct core::fmt::ArgumentV1
        parameters:
                [8]     x: &u64
                [8]     f: fn(&u64, &mut core::fmt::Formatter) -> core::result::Result<(), core::fmt::Error>

I.e., it's not sanitized (the linkage_name contains <) and also the {{impl}} is present.

So incidentally, and I've mentioned this before in
#37646, but I'd really, really, really like to see stuff like {{impl}} go away completely from the rust symbol name mangler.

But now I'm realizing, once this issue is fixed though I'm thinking in some cases it actually will disappear, and it is was just an artifact of the separate dwarf name mangler. I'm realizing now it pops up as an auto-complete item because its in the dwarf information, but I don't believe can {{ occur anywhere in the actual symbol binary names.

[michaelwoerister]

To give some historic context: Back in 2013, when I implemented the initial version of debuginfo support, DW_AT_linkage_name was the only way to make the Rust-unaware GDB print qualified names (e.g. foo::bar::baz). There is no deeper reason to linkage name attributes being the way they are. I agree that we should get rid of this hack now that we should not need it anymore.

[tromey]

like to see stuff like {{impl}} go away completely from the rust symbol name mangler

Me too. Actually I'd go farther and say I think rust's mangling scheme should change completely, and not be C++-like at all.

FWIW gdb's DWARF reader just ignores the mangled name in these cases now:

  if (cu->language == language_rust && mangled != NULL
      && strchr (mangled, '{') != NULL)
    mangled = NULL;

[sfackler]

Is there an issue about removing {{impl}} in particular in favor of the version that actually includes the impl type? It makes backtraces really confusing to look at.

[fitzgen]

Great bug, thanks for all the context.

+1 for the proposed symbol table name == DW_AT_linkage_name plan.

@tromey says:

I'd go farther and say I think rust's mangling scheme should change completely, and not be C++-like at all.

Interesting -- what were you imagining instead? Ignoring formatting nested function pointers and arrays in a C/C++ style (which obviously isn't an issue here), my experience is that all of the complexity with C++ symbols is the substitutions table, which prevents symbols from taking up an inordinate amount of space in artifacts. How would you approach this problem differently?

I realize this is slightly off topic, so feel free to reach out to me off-thread.

[m4b]

@sfackler agreed. I think this will actually be "automagically" fixed once the issue is resolved, since for example:

struct Foo(u64);

impl From<()> for Foo {
     fn from(_: ()) -> Self {
       Foo (0xdeadbeef)
     }
}

Is mangled into the actual symbol table as:

<test::Foo as core::convert::From<()>>::from::hb10e290c9fb6fd69

which is probably what you wanted to see (modulo the symbol hash), yes?

I'd go farther and say I think rust's mangling scheme should change completely, and not be C++-like at all.

I am interested in different name mangling schemes too and am interested in what you had in mind as well, @tromey @fitzgen please /cc me if you open another thread/whatever :)

[sfackler]

which is probably what you wanted to see (modulo the symbol hash), yes?

Yep!

[tromey]

Interesting -- what were you imagining instead?

I don't have a concrete plan; but I think (1) starting Rust symbols with _Z is a bit ugly in that it means adding special cases to existing C++ demanglers to deal with Rust; (2) it seems to me that there are already many Rust constructs that are emitted "strangely" (relative to C++ norms I guess); (3) there are things in the C++ mangling scheme that aren't relevant to Rust. So, I think it would be better all around to come up with something Rust-specific.

[steveklabnik]

Triage: https://github.com/rust-lang/rfcs/blob/master/text/2603-symbol-name-mangling-v2.md was implemented, changing the mangling scheme completely. I'm not sure about the other stuff here though; can anyone give a summary of what would be needed to close this issue?