Description
TL;DR
Remove gold linker:
--- a/build-gem5
+++ b/build-gem5
@@ -125,7 +125,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat-regression#gem5-unit-t
gold_linker_cmd = []
else:
extra_env = {}
- gold_linker_cmd = ['--gold-linker', LF,]
+ gold_linker_cmd = []
exit_status = self.sh.run_cmd(
(
[
Upstream bug report: https://sourceware.org/bugzilla/show_bug.cgi?id=25309
Workaround patch at: 44bf382
original investigation path
I've hit this many times in different locations of the code, time to open a precise bug report.
Ubuntu 18.04, GCC 7.4, GDB 8.1 at LKMC 5e39c1b gem5 0e8ce8b12e614925fdbbf306522e24aa8b05ac91
./build-gem5 --arch aarch64 --gem5-worktree master --gem5-build-type debug --verbose
./run --arch aarch64 --emulator gem5 --gem5-worktree master --gem5-build-type debug \
--userland userland/arch/aarch64/freestanding/linux/hello.S \
--debug-vm-args='-nh -ex "b FullO3CPU<O3CPUImpl>::tick()" -ex run' \
-- --cpu-type DerivO3CPU --caches
When the breakpoint is hit, we cannot see the current source line. bt
starts with:
#0 0x00005555592335e0 in FullO3CPU<O3CPUImpl>::tick() ()
#1 0x000055555923a05b in FullO3CPU<O3CPUImpl>::FullO3CPU(DerivO3CPUParams*)::{lambda()#1}::operator()() const ()
#2 0x000055555923ebe0 in std::_Function_handler<void (), FullO3CPU<O3CPUImpl>::FullO3CPU(DerivO3CPUParams*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#3 0x00005555571e7e50 in std::function<void ()>::operator()() const (this=0x55555dcb0400) at /usr/include/c++/7/bits/std_function.h:706
#4 0x00005555571e4260 in EventFunctionWrapper::process (this=0x55555dcb03c8) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/eventq.hh:839
#5 0x00005555594ae489 in EventQueue::serviceOne (this=0x55555da5dc00) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/eventq.cc:225
#6 0x00005555594ba2f6 in doSimLoop (eventq=0x55555da5dc00) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/simulate.cc:219
#7 0x00005555594b9f21 in simulate (num_cycles=18446744073709551615) at /work/linux-kernel-module-cheat/out/gem5/master/build/ARM/sim/simulate.cc:132
#8 0x0000555559611f3c in GlobalSimLoopExitEvent* pybind11::detail::argument_loader<unsigned long>::call_impl<GlobalSimLoopExitEvent*, GlobalSimLoopExitEvent* (*&)(unsigned long), 0ul, pybind11::detail::void_type>(GlobalSimLoopExitEvent* (*&)(unsigned long), pybind11::detail::index_sequence<0ul>, pybind11::detail::void_type&&) ()
The first frame inside bt
that has source is #3
:
#3 0x00005555571e7e50 in std::function<void ()>::operator()() const (this=0x55555dcb0400) at /usr/include/c++/7/bits/std_function.h:706
i b
shows:
(gdb) i b
Num Type Disp Enb Address What
1 breakpoint keep y 0x00005555592335e0 <FullO3CPU<O3CPUImpl>::tick()+4>
breakpoint already hit 1 time
Warning: the current language does not match this frame.
and l
shows the start of the file that contains the main
entry point:
32
33 #include "sim/init.hh"
34 #include "sim/init_signals.hh"
35
36 // main() is now pretty stripped down and just sets up python and then
37 // calls initM5Python which loads the various embedded python modules
38 // into the python environment and then starts things running by
39 // calling m5Main.
40 int
41 main(int argc, char **argv)
Non-debug build cannot find the source code either.
Running:
nm -Cs out/gem5/master/build/ARM/gem5.debug | grep 'FullO3CPU<O3CPUImpl>::tick'
gives:
0000000003cdf5dc W FullO3CPU<O3CPUImpl>::tick()
0000000004808460 r FullO3CPU<O3CPUImpl>::tick()::__PRETTY_FUNCTION__
And since PIE is used, we confirm https://stackoverflow.com/questions/2463150/what-is-the-fpie-option-for-position-independent-executables-in-gcc-and-ld/51308031#51308031
(gdb) p/x 0x00005555592335e0 - 0x555555554000
$6 = 0x3cdf5e0
which is 0000000003cdf5dc + 4, so looks about right.
Let's see if addr2line
sees the source:
addr2line -e out/gem5/master/build/ARM/gem5.debug 0x3cdf5dc
and nope!
??:?
OK, so no need for GDB to reproduce, yay! Trying it for other symbols does work, e.g. the first source line we see in EventFunctionWrapper::process
:
addr2line -e out/gem5/master/build/ARM/gem5.debug 0x1c90244
gives:
out/gem5/master/build/ARM/sim/eventq.hh:839
The tick event is defined as a lambda as follows:
tickEvent([this]{ tick(); }, "FullO3CPU tick",
false, Event::CPU_Tick_Pri),
which then gets called from an std::function
.
And the break point itself is a member method of a C++ template class:
template <class Impl>
void
FullO3CPU<Impl>::tick()
{
The full compilation command of src/cpu/o3/cpu.cc
can be seen from the ./build
command, and does have -ggdb3
, but no -O0
!
-O0 was not used, use it from now on
However hacking the build to add -O0
still did not solve the problem ()
diff --git a/src/SConscript b/src/SConscript
index 76bbb9e65..bb9ebbb44 100644
--- a/src/SConscript
+++ b/src/SConscript
@@ -1317,7 +1317,7 @@ if env['GCC']:
if sys.platform == 'sunos5':
ccflags['debug'] += ['-gstabs+']
else:
- ccflags['debug'] += ['-ggdb3']
+ ccflags['debug'] += ['-ggdb3', '-O0']
ldflags['debug'] += ['-O0']
# opt, fast, prof and perf all share the same cc flags, also add
# the optimization to the ldflags as LTO defers the optimization
Now we have:
$ nm -Cs out/gem5/master/build/ARM/gem5.debug | grep 'FullO3CPU<O3CPUImpl>::tick'
0000000003cdf5dc W FullO3CPU<O3CPUImpl>::tick()
0000000004808460 r FullO3CPU<O3CPUImpl>::tick()::__PRETTY_FUNCTION__
$ addr2line -e out/gem5/master/build/ARM/gem5.debug 0x3cdf5dc
??:?
and according to:
readelf --debug-dump=decodedline out/gem5/upstream-master/build/ARM/gem5.debug > decodeline.tmp
the closest addresses we have are on an unrelated file:
ns_gige.cc 652 0x3cdf5db
ns_gige.cc 651 0x3cdf5e3
Everything from now on is after that patch:
compile.tmp.sh
g++ \
-o out/gem5/upstream-master/build/ARM/cpu/o3/cpu.do \
-c \
-std=c++11 \
-pipe \
-fno-strict-aliasing \
-Wall \
-Wundef \
-Wextra \
-Wno-sign-compare \
-Wno-unused-parameter \
-Werror \
-Wno-error=deprecated-declarations \
-Wno-error=deprecated \
-pthread \
-fno-builtin-malloc \
-fno-builtin-calloc \
-fno-builtin-realloc \
-fno-builtin-free \
-DPROTOBUF_INLINE_NOT_IN_HEADERS=0 \
-ggdb3 \
-O0 \
-DNUMBER_BITS_PER_SET=64 \
-DDEBUG \
-DTRACING_ON=1 \
-Iext/pybind11/include \
-Iout/gem5/upstream-master/build/drampower/src \
-Iout/gem5/upstream-master/build/nomali/include \
-Iout/gem5/upstream-master/build/iostream3 \
-Iout/gem5/upstream-master/build/libfdt \
-Iout/gem5/upstream-master/build/libelf \
-Iout/gem5/upstream-master/build/fputils/include \
-Iinclude \
-Iext \
-I/usr/include/python2.7 \
-I/usr/include/x86_64-linux-gnu/python2.7 \
-I/usr/include/hdf5/serial \
-Iext/googletest/googletest/include \
-Iext/googletest/googlemock/include \
-Iout/gem5/upstream-master/build/ARM \
-Iout/gem5/upstream-master/build/ARM/systemc/ext \
out/gem5/upstream-master/build/ARM/cpu/o3/cpu.cc
intermediate-link.tmp.sh
g++ \
-o out/gem5/upstream-master/build/ARM/cpu/o3/lib.do.partial \
-fuse-ld=gold \
-r \
-nostdlib \
-flinker-output=rel \
out/gem5/upstream-master/build/ARM/cpu/o3/base_dyn_inst.do \
out/gem5/upstream-master/build/ARM/cpu/o3/commit.do \
out/gem5/upstream-master/build/ARM/cpu/o3/cpu.do \
out/gem5/upstream-master/build/ARM/cpu/o3/deriv.do \
out/gem5/upstream-master/build/ARM/cpu/o3/decode.do \
out/gem5/upstream-master/build/ARM/cpu/o3/dyn_inst.do \
out/gem5/upstream-master/build/ARM/cpu/o3/fetch.do \
out/gem5/upstream-master/build/ARM/cpu/o3/free_list.do \
out/gem5/upstream-master/build/ARM/cpu/o3/fu_pool.do \
out/gem5/upstream-master/build/ARM/cpu/o3/iew.do \
out/gem5/upstream-master/build/ARM/cpu/o3/inst_queue.do \
out/gem5/upstream-master/build/ARM/cpu/o3/lsq.do \
out/gem5/upstream-master/build/ARM/cpu/o3/lsq_unit.do \
out/gem5/upstream-master/build/ARM/cpu/o3/mem_dep_unit.do \
out/gem5/upstream-master/build/ARM/cpu/o3/regfile.do \
out/gem5/upstream-master/build/ARM/cpu/o3/rename.do \
out/gem5/upstream-master/build/ARM/cpu/o3/rename_map.do \
out/gem5/upstream-master/build/ARM/cpu/o3/rob.do \
out/gem5/upstream-master/build/ARM/cpu/o3/scoreboard.do \
out/gem5/upstream-master/build/ARM/cpu/o3/store_set.do \
out/gem5/upstream-master/build/ARM/cpu/o3/thread_context.do \
out/gem5/upstream-master/build/ARM/cpu/o3/checker.do \
-Lout/gem5/upstream-master/build/drampower \
-Lout/gem5/upstream-master/build/nomali \
-Lout/gem5/upstream-master/build/iostream3 \
-Lout/gem5/upstream-master/build/libfdt \
-Lout/gem5/upstream-master/build/libelf \
-Lout/gem5/upstream-master/build/fputils \
-L/usr/lib/x86_64-linux-gnu/hdf5/serial \
-Lout/gem5/upstream-master/build/googletest
link.tmp.sh
g++ \
-o out/gem5/upstream-master/build/ARM/gem5.debug \
-fuse-ld=gold \
-pthread \
-L/usr/lib/python2.7/config-x86_64-linux-gnu \
-L/usr/lib \
-Xlinker \
-export-dynamic \
-Wl,-O1 \
-Wl,-Bsymbolic-functions \
-z origin \
-O0 \
out/gem5/upstream-master/build/ARM/sim/main.do \
out/gem5/upstream-master/build/ARM/dev/pci/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/network/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/ps2/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/o3/probe/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/dt/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/protocol/lib.do.partial \
out/gem5/upstream-master/build/ARM/kern/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/minor/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/network/fault_model/lib.do.partial \
out/gem5/upstream-master/build/ARM/sim/probe/lib.do.partial \
out/gem5/upstream-master/build/ARM/base/vnc/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/tlm_core/2/generic_payload/lib.do.partial \
out/gem5/upstream-master/build/ARM/arch/arm/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/i2c/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/network/garnet2.0/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/common/lib.do.partial \
out/gem5/upstream-master/build/ARM/sim/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/testers/directedtest/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/dt/misc/lib.do.partial \
out/gem5/upstream-master/build/ARM/learning_gem5/part2/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/o3/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/dt/fx/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/qos/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/slicc_interface/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/cache/compressors/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/system/lib.do.partial \
out/gem5/upstream-master/build/ARM/arch/generic/lib.do.partial \
out/gem5/upstream-master/build/ARM/python/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/cache/prefetch/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/cache/tags/indexing_policies/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/profiler/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/virtio/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/net/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/protocol/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/tlm_core/2/quantum/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/testers/memtest/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/structures/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/core/lib.do.partial \
out/gem5/upstream-master/build/ARM/arch/arm/tracers/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/dt/int/lib.do.partial \
out/gem5/upstream-master/build/ARM/proto/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/dt/bit/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/tlm_bridge/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/testers/garnet_synthetic_traffic/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/testers/traffic_gen/lib.do.partial \
out/gem5/upstream-master/build/ARM/sim/power/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/utils/lib.do.partial \
out/gem5/upstream-master/build/ARM/base/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/cache/tags/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/channel/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/arm/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/storage/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/serial/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/pred/lib.do.partial \
out/gem5/upstream-master/build/ARM/unittest/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/probes/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/trace/lib.do.partial \
out/gem5/upstream-master/build/ARM/base/filters/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/testers/rubytest/lib.do.partial \
out/gem5/upstream-master/build/ARM/systemc/tlm_utils/lib.do.partial \
out/gem5/upstream-master/build/ARM/dev/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/cache/replacement_policies/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/simple/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/cache/lib.do.partial \
out/gem5/upstream-master/build/ARM/mem/ruby/network/simple/lib.do.partial \
out/gem5/upstream-master/build/ARM/cpu/simple/probes/lib.do.partial \
out/gem5/upstream-master/build/ARM/base/date.do \
-Lout/gem5/upstream-master/build/drampower \
-Lout/gem5/upstream-master/build/nomali \
-Lout/gem5/upstream-master/build/iostream3 \
-Lout/gem5/upstream-master/build/libfdt \
-Lout/gem5/upstream-master/build/libelf \
-Lout/gem5/upstream-master/build/fputils \
-L/usr/lib/x86_64-linux-gnu/hdf5/serial \
-Lout/gem5/upstream-master/build/googletest \
-lpython2.7 \
-lpthread \
-ldl \
-lutil \
-lm \
-lz \
-lprotobuf \
-lrt \
-ltcmalloc \
-lhdf5 \
-lhdf5_cpp \
-lfputils \
-lelf \
-lfdt \
-liostream3 \
-lnomali \
-ldrampower \
-lpng
TODO where do:
-Wl,-O1 \
-Wl,-Bsymbolic-functions \
come from? Cannot find in gem5 source code.
-O0 and don't use gold linker
It works!!!
--- a/build-gem5
+++ b/build-gem5
@@ -125,7 +125,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat-regression#gem5-unit-t
gold_linker_cmd = []
else:
extra_env = {}
- gold_linker_cmd = ['--gold-linker', LF,]
+ gold_linker_cmd = []
exit_status = self.sh.run_cmd(
(
[
Don't use gold linker
Remove the -O0
patch, don't use gold, it works.
Ubuntu 19.10 reproduction
Also reproduced in Ubuntu 19.10 with gcc 8, Binutils 2.33. Here EventFunctionWrapper::process()
also did not have symbols, so template functions are not a requirement
Failed minimal reproducible example attempts
Partial linking + gold was not enough on a minimal example: https://github.com/cirosantilli/cpp-cheat/tree/b4aaea05a02f5953a484aefde4c34216ddec9574/linker/incremental_link hacked with -use-gold
on all link steps:
main.out: f12.o main.o
- $(CC) $(CFLAGS) -o '$@' $^
+ $(CC) -use-gold $(CFLAGS) -o '$@' $^
main_r.out: f12_r.o main.o
- $(CC) $(CFLAGS) -o '$@' $^
+ $(CC) -use-gold $(CFLAGS) -o '$@' $^
f12_r.o: f1.o f2.o
- ld -o $@ -r $^
+ ld -use-gold -o $@ -r $^