(Request for Feedback) Very slow STEP (step into) and NEXT (stepping over) operation in GDB

[RoyBellingan]

This is funny, and I have yet no idea how to track and debug but in short machine is super high spec (ryzen 5950 16 core / 32gb ram 3600)
compiler gcc11.2
linker bfd / gold
gdb 11.1
The stepping over / into is very fast, overall experience is super snappy and reactive

When I link with mold, for... I have no idea why but the first time "certain" function are encountered the next is super slow, and I mean 2/3 second, instead if I put a breakpoint on the next line and do c (continue so restart execution), super fast.

No idea how to dig and how to debug this one, if you can please point me in some direction.

I noticed this tend to happen MORE when using function that involves call with QtLibraries object, even something like
if (QString("x") == "x") {
Take a good 0.7second to be stepped over...
I will cross post this on qt libraries but tracker and try to mix and match compiler and debugger too.

Edit forge to add I am using the mold version 1.0.1, I will recheck asap with the latest 1.0.3 and master

[rui314]

Please try again with mold 1.1.

Does running gdb-add-index on your executable fix the problem? Please just run gdb-add-index <your-executable> and run gdb on that file to see if it makes any difference.

[RoyBellingan]

Thank you for the feedback, I tried with and without updating, but sorry if there is a bit of improvement is less than 10% and overall debug xp is still not ideal. (using mold 1.1 89612b7 )

I tried to add this gdb-add-index to the various qt so, but it fails as they do not have debug symbol.
I would also to point out the overall start speed of the debugged program felt a bit slower than previous mold, using gold and bfd instead is almost instant.

[RoyBellingan]

Sorry to add even more uncertainty, but I tried another program and here MOLD is running flawlessly -.-
So I would say this bug is related to something on my side, I will look on some way to debug and track it.

[d3m3t3r]

I'm experiencing very slow behavior of gdb with mold-linked binary too.
My binary is 513MB large, with debug info, C++ compiled by gcc-7.5. With ld, it takes 7 seconds for gdb to load symbols, there is no delay when setting a breakpoint (br main) and stepping (step).

With mold-1.1.1 and gdb-8.1.1, it takes more than a minute to read symbols (actually the delay is mostly after "Reading symbols from my-binary...done." is printed and before the prompt appears). Setting a breakpoint is instantaneous. Stepping, again, takes more than a minute. gdb-add-index doesn't significantly change it.

With mold-1.1.1 and gdb-11.2.90.20220303-git, it takes about 7 seconds to read symbols. Setting a breakpoint is instantaneous. Stepping, again, takes more than a minute. gdb-add-index actually make loading symbols much longer again.

[rui314]

I wonder if this issue is related to #402. Can you guys check if the link command line contains the --compress-debug-sections option? If so, does removing the flag fix the issue?

[RoyBellingan]

Ciao rui314, sorry but this switch is not present.
If I have time I will try to remove piece by piece this "slow" one, because as said same system another program which has basically same dependencies and settings is going fine.

[d3m3t3r]

Not using --compress-debug-sections, actually I can reproduce the issue even when there are just -L and -l linker options.

[rui314]

Thank you guys for the input. It's unfortunate that that wasn't related.

Have you see this to happen in any open-source program? I want to reproduce the issue locally to debug.

[rui314]

I committed another change as you can see above for this issue. I'm not sure if this is the right fix because I haven't succeeded to reproduce the issue locally. Do you mind if I ask you to try again with this change?

[d3m3t3r]

It's still reproducible with mold 1.1.1 (d0e4eee). Unfortunately, the issue didn't occur with any open-source projects I tried to build.

[tromey]

Did you compile with -g3? I encountered a gdb slowdown when using that option, see #438.

[d3m3t3r]

Yes, I compile all with -ggdb3 -Og, tried with -g -O0 and -g2 -Og and the problem disappeared so I guess it's a duplicate of #438 then.

[rui314]

@d3m3t3r Thank you very much for confirming! That's a huge step towards fixing this mysterious issue.

[RoyBellingan]

Excellent guys! I totally overlooked that parameter! Thank you very much!

[rui314]

@RoyBellingan I submitted a change to workaround this issue. Do you mind if I ask you to rebuild mold and try again? I'm curious about

1. whether or not it resolves the gdb slowdown issue, and
2. the size of the debug sections.

For (2), can you share the output of readelf --sections <your-binary> | fgrep .debug?

[d3m3t3r]

Mold 1.2.0 (f9768c6) fixed the issue for me.
My .debug sections:

  [42] .debug_abbrev     PROGBITS         0000000000000000  02aa164c
  [43] .debug_aranges    PROGBITS         0000000000000000  02f12630
  [44] .debug_info       PROGBITS         0000000000000000  02fd3be0
  [45] .debug_line       PROGBITS         0000000000000000  15eebc91
  [46] .debug_loc        PROGBITS         0000000000000000  187c78ab
  [47] .debug_macro      PROGBITS         0000000000000000  2373a84f
  [48] .debug_ranges     PROGBITS         0000000000000000  263dd5b0
  [49] .debug_str        PROGBITS         0000000000000000  277c2ab0

[rui314]

Great. The .debug_macro section in your executable is large, it's about 600 MiB. But other debug sections are large as well, so it may not be too worrisome. I'll go ahead with this change until someone say that the size of .debug_macro matters. Thanks again for testing!