Skip to content

If gem5 is built with gold linker, I cannot see the source code for certain lines when debugging the simulator in GDB despite using the debug build #109

Open
@cirosantilli

Description

@cirosantilli

TL;DR

Remove gold linker:

--- a/build-gem5
+++ b/build-gem5
@@ -125,7 +125,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat-regression#gem5-unit-t
             gold_linker_cmd = []
         else:
             extra_env = {}
-            gold_linker_cmd = ['--gold-linker', LF,]
+            gold_linker_cmd = []
         exit_status = self.sh.run_cmd(
             (
                 [

Upstream bug report: https://sourceware.org/bugzilla/show_bug.cgi?id=25309

Workaround patch at: 44bf382

original investigation path

I've hit this many times in different locations of the code, time to open a precise bug report.
Ubuntu 18.04, GCC 7.4, GDB 8.1 at LKMC 5e39c1b gem5 0e8ce8b12e614925fdbbf306522e24aa8b05ac91

./build-gem5 --arch aarch64 --gem5-worktree master --gem5-build-type debug --verbose
./run --arch aarch64 --emulator gem5 --gem5-worktree master --gem5-build-type debug \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --debug-vm-args='-nh -ex "b FullO3CPU<O3CPUImpl>::tick()" -ex run' \
  -- --cpu-type DerivO3CPU --caches

When the breakpoint is hit, we cannot see the current source line. bt starts with:

#0  0x00005555592335e0 in FullO3CPU<O3CPUImpl>::tick() ()
#1  0x000055555923a05b in FullO3CPU<O3CPUImpl>::FullO3CPU(DerivO3CPUParams*)::{lambda()#1}::operator()() const ()
#2  0x000055555923ebe0 in std::_Function_handler<void (), FullO3CPU<O3CPUImpl>::FullO3CPU(DerivO3CPUParams*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#3  0x00005555571e7e50 in std::function<void ()>::operator()() const (this=0x55555dcb0400) at /usr/include/c++/7/bits/std_function.h:706
#4  0x00005555571e4260 in EventFunctionWrapper::process (this=0x55555dcb03c8) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/eventq.hh:839
#5  0x00005555594ae489 in EventQueue::serviceOne (this=0x55555da5dc00) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/eventq.cc:225
#6  0x00005555594ba2f6 in doSimLoop (eventq=0x55555da5dc00) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/simulate.cc:219
#7  0x00005555594b9f21 in simulate (num_cycles=18446744073709551615) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/simulate.cc:132
#8  0x0000555559611f3c in GlobalSimLoopExitEvent* pybind11::detail::argument_loader<unsigned long>::call_impl<GlobalSimLoopExitEvent*, GlobalSimLoopExitEvent* (*&)(unsigned long), 0ul, pybind11::detail::void_type>(GlobalSimLoopExitEvent* (*&)(unsigned long), pybind11::detail::index_sequence<0ul>, pybind11::detail::void_type&&) ()

The first frame inside bt that has source is #3:

#3  0x00005555571e7e50 in std::function<void ()>::operator()() const (this=0x55555dcb0400) at /usr/include/c++/7/bits/std_function.h:706

i b shows:

(gdb) i b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00005555592335e0 <FullO3CPU<O3CPUImpl>::tick()+4>
        breakpoint already hit 1 time
Warning: the current language does not match this frame.

and l shows the start of the file that contains the main entry point:

32
33      #include "sim/init.hh"
34      #include "sim/init_signals.hh"
35
36      // main() is now pretty stripped down and just sets up python and then
37      // calls initM5Python which loads the various embedded python modules
38      // into the python environment and then starts things running by
39      // calling m5Main.
40      int
41      main(int argc, char **argv)

Non-debug build cannot find the source code either.

Running:

nm -Cs out/gem5/master/build/ARM/gem5.debug | grep 'FullO3CPU<O3CPUImpl>::tick'

gives:

0000000003cdf5dc W FullO3CPU<O3CPUImpl>::tick()
0000000004808460 r FullO3CPU<O3CPUImpl>::tick()::__PRETTY_FUNCTION__

And since PIE is used, we confirm https://stackoverflow.com/questions/2463150/what-is-the-fpie-option-for-position-independent-executables-in-gcc-and-ld/51308031#51308031

(gdb) p/x 0x00005555592335e0 - 0x555555554000
$6 = 0x3cdf5e0

which is 0000000003cdf5dc + 4, so looks about right.

Let's see if addr2line sees the source:

addr2line -e out/gem5/master/build/ARM/gem5.debug 0x3cdf5dc

and nope!

??:?

OK, so no need for GDB to reproduce, yay! Trying it for other symbols does work, e.g. the first source line we see in EventFunctionWrapper::process:

addr2line -e out/gem5/master/build/ARM/gem5.debug 0x1c90244

gives:

out/gem5/master/build/ARM/sim/eventq.hh:839

The tick event is defined as a lambda as follows:

      tickEvent([this]{ tick(); }, "FullO3CPU tick",
                false, Event::CPU_Tick_Pri),

which then gets called from an std::function.

And the break point itself is a member method of a C++ template class:

template <class Impl>
void
FullO3CPU<Impl>::tick()
{

The full compilation command of src/cpu/o3/cpu.cc can be seen from the ./build command, and does have -ggdb3, but no -O0!

-O0 was not used, use it from now on

However hacking the build to add -O0 still did not solve the problem ()

diff --git a/src/SConscript b/src/SConscript
index 76bbb9e65..bb9ebbb44 100644
--- a/src/SConscript
+++ b/src/SConscript
@@ -1317,7 +1317,7 @@ if env['GCC']:
     if sys.platform == 'sunos5':
         ccflags['debug'] += ['-gstabs+']
     else:
-        ccflags['debug'] += ['-ggdb3']
+        ccflags['debug'] += ['-ggdb3', '-O0']
     ldflags['debug'] += ['-O0']
     # opt, fast, prof and perf all share the same cc flags, also add
     # the optimization to the ldflags as LTO defers the optimization

Now we have:

$ nm -Cs out/gem5/master/build/ARM/gem5.debug | grep 'FullO3CPU<O3CPUImpl>::tick'
0000000003cdf5dc W FullO3CPU<O3CPUImpl>::tick()
0000000004808460 r FullO3CPU<O3CPUImpl>::tick()::__PRETTY_FUNCTION__
$ addr2line -e out/gem5/master/build/ARM/gem5.debug 0x3cdf5dc
??:?

and according to:

readelf --debug-dump=decodedline out/gem5/upstream-master/build/ARM/gem5.debug > decodeline.tmp

the closest addresses we have are on an unrelated file:

ns_gige.cc                                   652           0x3cdf5db
ns_gige.cc                                   651           0x3cdf5e3

Everything from now on is after that patch:

compile.tmp.sh

g++ \
  -o out/gem5/upstream-master/build/ARM/cpu/o3/cpu.do \
  -c \
  -std=c++11 \
  -pipe \
  -fno-strict-aliasing \
  -Wall \
  -Wundef \
  -Wextra \
  -Wno-sign-compare \
  -Wno-unused-parameter \
  -Werror \
  -Wno-error=deprecated-declarations \
  -Wno-error=deprecated \
  -pthread \
  -fno-builtin-malloc \
  -fno-builtin-calloc \
  -fno-builtin-realloc \
  -fno-builtin-free \
  -DPROTOBUF_INLINE_NOT_IN_HEADERS=0 \
  -ggdb3 \
  -O0 \
  -DNUMBER_BITS_PER_SET=64 \
  -DDEBUG \
  -DTRACING_ON=1 \
  -Iext/pybind11/include \
  -Iout/gem5/upstream-master/build/drampower/src \
  -Iout/gem5/upstream-master/build/nomali/include \
  -Iout/gem5/upstream-master/build/iostream3 \
  -Iout/gem5/upstream-master/build/libfdt \
  -Iout/gem5/upstream-master/build/libelf \
  -Iout/gem5/upstream-master/build/fputils/include \
  -Iinclude \
  -Iext \
  -I/usr/include/python2.7 \
  -I/usr/include/x86_64-linux-gnu/python2.7 \
  -I/usr/include/hdf5/serial \
  -Iext/googletest/googletest/include \
  -Iext/googletest/googlemock/include \
  -Iout/gem5/upstream-master/build/ARM \
  -Iout/gem5/upstream-master/build/ARM/systemc/ext \
  out/gem5/upstream-master/build/ARM/cpu/o3/cpu.cc

intermediate-link.tmp.sh

g++ \
  -o out/gem5/upstream-master/build/ARM/cpu/o3/lib.do.partial \
  -fuse-ld=gold \
  -r \
  -nostdlib \
  -flinker-output=rel \
  out/gem5/upstream-master/build/ARM/cpu/o3/base_dyn_inst.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/commit.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/cpu.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/deriv.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/decode.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/dyn_inst.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/fetch.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/free_list.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/fu_pool.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/iew.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/inst_queue.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/lsq.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/lsq_unit.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/mem_dep_unit.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/regfile.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/rename.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/rename_map.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/rob.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/scoreboard.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/store_set.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/thread_context.do \
  out/gem5/upstream-master/build/ARM/cpu/o3/checker.do \
  -Lout/gem5/upstream-master/build/drampower \
  -Lout/gem5/upstream-master/build/nomali \
  -Lout/gem5/upstream-master/build/iostream3 \
  -Lout/gem5/upstream-master/build/libfdt \
  -Lout/gem5/upstream-master/build/libelf \
  -Lout/gem5/upstream-master/build/fputils \
  -L/usr/lib/x86_64-linux-gnu/hdf5/serial \
  -Lout/gem5/upstream-master/build/googletest

link.tmp.sh

g++ \
  -o out/gem5/upstream-master/build/ARM/gem5.debug \
  -fuse-ld=gold \
  -pthread \
  -L/usr/lib/python2.7/config-x86_64-linux-gnu \
  -L/usr/lib \
  -Xlinker \
  -export-dynamic \
  -Wl,-O1 \
  -Wl,-Bsymbolic-functions \
  -z origin \
  -O0 \
  out/gem5/upstream-master/build/ARM/sim/main.do \
  out/gem5/upstream-master/build/ARM/dev/pci/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/network/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/ps2/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/o3/probe/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/dt/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/protocol/lib.do.partial \
  out/gem5/upstream-master/build/ARM/kern/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/minor/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/network/fault_model/lib.do.partial \
  out/gem5/upstream-master/build/ARM/sim/probe/lib.do.partial \
  out/gem5/upstream-master/build/ARM/base/vnc/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/tlm_core/2/generic_payload/lib.do.partial \
  out/gem5/upstream-master/build/ARM/arch/arm/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/i2c/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/network/garnet2.0/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/common/lib.do.partial \
  out/gem5/upstream-master/build/ARM/sim/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/testers/directedtest/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/dt/misc/lib.do.partial \
  out/gem5/upstream-master/build/ARM/learning_gem5/part2/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/o3/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/dt/fx/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/qos/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/slicc_interface/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/cache/compressors/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/system/lib.do.partial \
  out/gem5/upstream-master/build/ARM/arch/generic/lib.do.partial \
  out/gem5/upstream-master/build/ARM/python/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/cache/prefetch/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/cache/tags/indexing_policies/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/profiler/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/virtio/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/net/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/protocol/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/tlm_core/2/quantum/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/testers/memtest/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/structures/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/core/lib.do.partial \
  out/gem5/upstream-master/build/ARM/arch/arm/tracers/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/dt/int/lib.do.partial \
  out/gem5/upstream-master/build/ARM/proto/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/dt/bit/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/tlm_bridge/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/testers/garnet_synthetic_traffic/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/testers/traffic_gen/lib.do.partial \
  out/gem5/upstream-master/build/ARM/sim/power/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/utils/lib.do.partial \
  out/gem5/upstream-master/build/ARM/base/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/cache/tags/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/channel/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/arm/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/storage/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/serial/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/pred/lib.do.partial \
  out/gem5/upstream-master/build/ARM/unittest/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/probes/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/trace/lib.do.partial \
  out/gem5/upstream-master/build/ARM/base/filters/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/testers/rubytest/lib.do.partial \
  out/gem5/upstream-master/build/ARM/systemc/tlm_utils/lib.do.partial \
  out/gem5/upstream-master/build/ARM/dev/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/cache/replacement_policies/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/simple/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/cache/lib.do.partial \
  out/gem5/upstream-master/build/ARM/mem/ruby/network/simple/lib.do.partial \
  out/gem5/upstream-master/build/ARM/cpu/simple/probes/lib.do.partial \
  out/gem5/upstream-master/build/ARM/base/date.do \
  -Lout/gem5/upstream-master/build/drampower \
  -Lout/gem5/upstream-master/build/nomali \
  -Lout/gem5/upstream-master/build/iostream3 \
  -Lout/gem5/upstream-master/build/libfdt \
  -Lout/gem5/upstream-master/build/libelf \
  -Lout/gem5/upstream-master/build/fputils \
  -L/usr/lib/x86_64-linux-gnu/hdf5/serial \
  -Lout/gem5/upstream-master/build/googletest \
  -lpython2.7 \
  -lpthread \
  -ldl \
  -lutil \
  -lm \
  -lz \
  -lprotobuf \
  -lrt \
  -ltcmalloc \
  -lhdf5 \
  -lhdf5_cpp \
  -lfputils \
  -lelf \
  -lfdt \
  -liostream3 \
  -lnomali \
  -ldrampower \
  -lpng

TODO where do:

  -Wl,-O1 \
  -Wl,-Bsymbolic-functions \

come from? Cannot find in gem5 source code.

-O0 and don't use gold linker

It works!!!

--- a/build-gem5
+++ b/build-gem5
@@ -125,7 +125,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat-regression#gem5-unit-t
             gold_linker_cmd = []
         else:
             extra_env = {}
-            gold_linker_cmd = ['--gold-linker', LF,]
+            gold_linker_cmd = []
         exit_status = self.sh.run_cmd(
             (
                 [

Don't use gold linker

Remove the -O0 patch, don't use gold, it works.

Ubuntu 19.10 reproduction

Also reproduced in Ubuntu 19.10 with gcc 8, Binutils 2.33. Here EventFunctionWrapper::process() also did not have symbols, so template functions are not a requirement

Failed minimal reproducible example attempts

Partial linking + gold was not enough on a minimal example: https://github.com/cirosantilli/cpp-cheat/tree/b4aaea05a02f5953a484aefde4c34216ddec9574/linker/incremental_link hacked with -use-gold on all link steps:


 main.out: f12.o main.o
-   $(CC) $(CFLAGS) -o '$@' $^
+ $(CC) -use-gold $(CFLAGS) -o '$@' $^

 main_r.out: f12_r.o main.o
-   $(CC) $(CFLAGS) -o '$@' $^
+ $(CC) -use-gold $(CFLAGS) -o '$@' $^

 f12_r.o: f1.o f2.o
-   ld -o $@ -r $^
+ ld -use-gold -o $@ -r $^

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions