Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf record --call-graph dwarf does not support ld.lld's default --rosegment -z noseparate-code layout #1646

Closed
MaskRay opened this issue May 27, 2022 · 4 comments
Closed

perf record --call-graph dwarf does not support ld.lld's default --rosegment -z noseparate-code layout #1646

MaskRay opened this issue May 27, 2022 · 4 comments
Assignees
@MaskRay
Labels
[BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 5.19 This bug was fixed in Linux 5.19 [TOOL] lld The issue is relevant to LLD linker

Comments

@MaskRay
Copy link
Member

MaskRay commented May 27, 2022 •
edited
Loading

Move llvm/llvm-project#53156 here since it is a perf bug.

@Kha reported that ld.lld linked objects do not symbolize correctly:

$ cat foo.c
#include <stdio.h>

void foo(int d) { int s = 0; for (int i = 0; i < 1000000; i++) { printf(""); s += i / d; } }
int main() { foo(100000); }

% clang -no-pie -fno-pie -fuse-ld=bfd foo.c && perf record --call-graph dwarf ./a.out && perf report | head -n 20
[ perf record: Woken up 6 times to write data ]
[ perf record: Captured and wrote 1.385 MB perf.data (169 samples) ]
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 169  of event 'cycles'
# Event count (approx.): 75073906
#
# Children      Self  Command  Shared Object      Symbol                                
# ........  ........  .......  .................  ......................................
#
    85.57%     0.00%  a.out    a.out              [.] _start
            |
            ---_start
               __libc_start_main
               main
               foo
               |          
               |--67.08%--__printf (inlined)
               |          |     
% clang -no-pie -fno-pie -fuse-ld=lld foo.c && perf record --call-graph dwarf ./a.out && perf report | head -n 20
[ perf record: Woken up 6 times to write data ]
[ perf record: Captured and wrote 1.417 MB perf.data (173 samples) ]
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 173  of event 'cycles'
# Event count (approx.): 74585658
#
# Children      Self  Command  Shared Object      Symbol                                
# ........  ........  .......  .................  ......................................
#
    93.53%     0.00%  a.out    [unknown]          [.] 0xffffffffffffffff
            |
            ---0xffffffffffffffff
               |          
               |--88.60%--__libc_start_main
               |          main
               |          foo
               |          |          
               |          |--72.70%--__printf (inlined)

Tips: to build perf with debug info:

make O=/tmp/linux/perf ARCH=x86_64 -j60 -C tools/perf DEBUG=1 EXTRA_CFLAGS='-O0 -g'

It is due to perf not supporting ld.lld's default --rosegment -z noseparate-code layout (since 10.0)

    ld.lld (default) => fail
    ld.lld --no-rosegment => ok
    ld.lld -z separate-code => ok

    ld.bfd -z noseparate-code => ok
    ld.bfd -z separate-code (default for Linux/x86) => ok but by luck:
    there are two PT_LOAD but their p_vaddr difference equals p_offset
    difference
intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue May 27, 2022
@MaskRay @intel-lab-lkp
perf: Fix segbase for ld.lld linked objects
207e920 
segbase is the address of .eh_frame_hdr and table_data is segbase plus
the header size. find_proc_info computes segbase as `map->start +
segbase - map->pgoff` which is wrong when

* .eh_frame_hdr and .text are in different PT_LOAD program headers
* and their p_vaddr difference does not equal their p_offset difference

Since 10.0, ld.lld's default --rosegment -z noseparate-code layout has
such R and RX PT_LOAD program headers.

    ld.lld => fail
    ld.lld --no-rosegment => ok
    ld.lld -z separate-code => ok

    ld.bfd -z separate-code (default for Linux/x86) => ok
    ld.bfd -z noseparate-code => ok

To fix the issue, compute segbase as dso's base address plus
PT_GNU_EH_FRAME's p_vaddr.

Reported-by: Sebastian Ullrich <sebasti@nullri.ch>
Link: ClangBuiltLinux#1646
Signed-off-by: Fangrui Song <maskray@google.com>
Cc: Ian Rogers <irogers@google.com>
@nickdesaulniers
Copy link
Member

@MaskRay sent https://lore.kernel.org/llvm/20220527055217.452307-1-maskray@google.com/.

@nickdesaulniers nickdesaulniers added [BUG] linux A bug that should be fixed in the mainline kernel. [PATCH] Submitted A patch has been submitted for review [TOOL] lld The issue is relevant to LLD linker labels May 27, 2022
@nickdesaulniers
Copy link
Member

cc @captain5050

intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue May 27, 2022
@MaskRay @intel-lab-lkp
perf: Fix segbase for ld.lld linked objects
02affb0 
segbase is the address of .eh_frame_hdr and table_data is segbase plus
the header size. find_proc_info computes segbase as `map->start +
segbase - map->pgoff` which is wrong when

* .eh_frame_hdr and .text are in different PT_LOAD program headers
* and their p_vaddr difference does not equal their p_offset difference

Since 10.0, ld.lld's default --rosegment -z noseparate-code layout has
such R and RX PT_LOAD program headers.

    ld.lld (default) => perf report fails to unwind `perf record
    --call-graph dwarf` recorded data
    ld.lld --no-rosegment => ok (trivial, no R PT_LOAD)
    ld.lld -z separate-code => ok but by luck: there are two PT_LOAD but
    their p_vaddr difference equals p_offset difference

    ld.bfd -z noseparate-code => ok (trivial, no R PT_LOAD)
    ld.bfd -z separate-code (default for Linux/x86) => ok but by luck:
    there are two PT_LOAD but their p_vaddr difference equals p_offset
    difference

To fix the issue, compute segbase as dso's base address plus
PT_GNU_EH_FRAME's p_vaddr. The base address is computed by iterating
over all dso-associated maps and then subtract the first PT_LOAD p_vaddr
(the minimum guaranteed by generic ABI) from the minimum address.

In libunwind, find_proc_info transitively called by unw_step is cached,
so the iteration overhead is acceptable.

Reported-by: Sebastian Ullrich <sebasti@nullri.ch>
Link: ClangBuiltLinux#1646
Signed-off-by: Fangrui Song <maskray@google.com>
Cc: Ian Rogers <irogers@google.com>

--
Changes from v1:
* Fix elf_base_address to use the first PT_LOAD
* Use dso::elf_base_addr which is a constant even if the dso is loaded into multiple processes
@nathanchance
Copy link
Member

Patch accepted: https://git.kernel.org/acme/c/dc2cf4ca866f571590b65f5e4de06b8c9a302808

No idea if the SHA is stable though.

@nathanchance nathanchance added [PATCH] Accepted A submitted patch has been accepted upstream and removed [PATCH] Submitted A patch has been submitted for review labels Jun 3, 2022
@MaskRay
Copy link
Member Author

MaskRay commented Jun 5, 2022

A message [GIT PULL] perf tools changes for v5.19: 3rd batch tells me that this has been merged.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc2cf4ca866f571590b65f5e4de06b8c9a302808

@MaskRay MaskRay closed this as completed Jun 5, 2022
@MaskRay MaskRay added the [FIXED][LINUX] 5.19 This bug was fixed in Linux 5.19 label Jun 5, 2022
@MaskRay MaskRay mentioned this issue Jun 5, 2022
Closed
@nickdesaulniers nickdesaulniers removed the [PATCH] Accepted A submitted patch has been accepted upstream label Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MaskRay MaskRay

Labels
[BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 5.19 This bug was fixed in Linux 5.19 [TOOL] lld The issue is relevant to LLD linker
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MaskRay @nickdesaulniers @nathanchance