CRASH Segfault with DrMemory

[Desperado17]

Received SIGSEGV at client library pc 0x00007f36e6555fa9 in thread 15974
Base: 0x00007f36ff0c5000
Registers:eax=0x0000000000000033 ebx=0x00007f34e72ed298 ecx=0x0000000000000050 edx=0x0000000000000000
esi=0x0000000000000050 edi=0x00007f34e72e5e50 esp=0x00007f34e72e5d30 ebp=0x00007f34e72e5e50
r8 =0x00007f35beeaf5b0 r9 =0x0000000000000020 r10=0x00007f36ff2a3000 r11=0x000000000000008f
r12=0x0000000000000033 r13=0x00007f350d36f054 r14=0x0000000000000074 r15=0x0000000000000082
eflags=0x0000000000010203
2.5.19327-0-(Dec 1 2022 15:41:46)
-no_dynamic_options -logdir '/home/user/Downloads/DrMemory-Linux-2.5.19327/drmemory/logs/dynamorio' -client_lib '/home/user/Downloads/DrMemory-Linux-2.5.19327/bin64/debug/libdrmemorylib.so;0;-light -no_check_uninitialized -logdir /home/user/Downloads/DrMemory-Linux-2.5.19327/drmemory/logs -symcache_dir `/home/user/
0x00007f34e72e5e50 0x0000000000000008
/home/user/Downloads/DrMemory-Linux-2.5.19327/dynamorio/lib64/release/libdynamorio.so=0x00007f36ff0c5000
/home/user/Downloads/DrMemory-Linux-2.5.19327/bin64/debug/libdrmemorylib.so=0x00007f36e6302000
/usr/local/lib/libunwind.so.8=0x00007f36fec7e000
/lib/x86_64-linux-gnu/liblzma.so.5=0x00007f36fe438000
/lib/x86_64-linux-gnu/libpthread.so.0=0x00007f36fe013000
/lib/x86_64-linux-gnu/libdl.so.2=0x00007f36fe233000
/lib/x86_64-linux-gnu/libc.so.6=0x000>

~/Downloads/DrMemory-Linux-2.5.19327/bin64/drmemory -debug -light -no_check_uninitialized -- ./myapp

or

drrun -debug -t drmemory -light -no_check_uninitialized -- ./myapp

Only with

-leaks_only -no_count_leaks -no_track_allocs

program will start correctly, even one missing will trigger a fatal sigsegv.

[Desperado17]

See this discussion:

https://groups.google.com/g/dynamorio-users/c/IAFjO9gBPJ0

Ok,the nullptr in rdx seems to come via rbx from the rax register which is last written here:

#0 0x00007ffff7611374 in __ctype_b_loc () at ../include/ctype.h:42
#1 0x00007fffd992573f in cpp_demangle_read_uqname (ddata=0x7ffddaaee140) at libelftc_dem_gnu3.c:3109

https://sourceforge.net/p/elftoolchain/code/3530/tree//trunk/libelftc/libelftc_dem_gnu3.c#l3109

/* source name */
if (ELFTC_ISDIGIT(*ddata->cur) != 0)

I suspect that the lookup table inside the isdigit function is not initialized similar to isspace in this ticket:

DynamoRIO/drmemory#2425

[derekbruening]

This doesn't have enough information for reproducing: no platform or OS or application information. There are bug templates that ask all these questions (https://github.com/DynamoRIO/dynamorio/issues/new?assignees=&labels=&template=bug_report.md&title= and
https://github.com/DynamoRIO/drmemory/issues/new?assignees=&labels=&template=bug_report.md&title=) ; looks like they've been bypassed here which doesn't help. Given that it doesn't say it happens with hello,world and every application tried, it sounds like it is limited to "myapp".

If it is an issue with the private loader, it is becoming impractical to maintain it in the face of hidden dependencies in ld.so, libc, and pthreads. If you are not able to figure out what libc internal weirdness is going on here probably the thing to do is link everything the tool needs statically and bail on the private loader.

[Desperado17]

Ok, some infos about the system I currently use:

ii libc-bin 2.35-0ubuntu3.1 amd64 GNU C Library: Binaries
ii libc-dev-bin 2.35-0ubuntu3.1 amd64 GNU C Library: Development binaries
ii libc-devtools 2.35-0ubuntu3.1 amd64 GNU C Library: Development tools
ii libc6:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: Shared libraries
ii libc6:i386 2.35-0ubuntu3.1 i386 GNU C Library: Shared libraries
ii libc6-dbg:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: detached debugging symbols
ii libc6-dev:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: Development Libraries and Header Files
ii libc6-i386 2.35-0ubuntu3.1 amd64 GNU C Library: 32-bit shared libraries for AMD64

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

I cannot say a lot about the target application except that it is multithreaded (150+ threads) and proprietary. I tried to reproduce the problem by building elftools locally then put the string in addr2line but to no avail.

I suspect that the tables where the locales are stored in disappear for some reason:
https://elixir.bootlin.com/glibc/glibc-2.35/source/locale/C-ctype.c

[Desperado17]

Addendum:

• The app uses dlopen, dlsym to open and address an external library. Could this be a problem?
• Is there a way to avoid this codepath with decreased functionality? All I need are call stacks, they don't have to be symbolized.

[Desperado17]

Greetings,
can you tell me which parameters you use for building elftools for the nightlies? I would like to try defining isdigit locally with the default implementation which should suffice for my x64 environment.

Regards

Btw:
Do these dwarf strings really need the locale capable version of isdigit? Wouldn't the standard version ala c - '0' <= 9 work for most cases?

[derekbruening]

I think you're looking for: https://github.com/DynamoRIO/dynamorio/blob/master/ext/drsyms/libelftc/rebuild_elftc.sh

[Desperado17]

I'll try. Btw is there a simple method to build dynamorio with drmemory from dynamorio master?

[derekbruening]

It is built in the automated packages so mirroring its steps should do it: https://github.com/DynamoRIO/dynamorio/blob/master/.github/workflows/ci-package.yml#L68

[Desperado17]

Okay, so the source directory has a drmemory folder now. Whats missing is the cmake / build command I tried calling
./suite/runsuite_wrapper.pl automated_ci
but it finished with errors. Just calling cmake .. on the existing dynamorio build folder doesn't suffice.

[Desperado17]

Ok, I built drmemory with default cmake ../ and modified elftoolchain that replaces the libc isdigit call with a simple internal ascii definition:

int isdigit_custom ( int c )
{
return c >= '0' && c <= '9';
}

drmemory did not crash.
drmemory wasn't very fast though. Will cmake .. build in release mode? x64 Ubuntu 22.04.

[Desperado17]

Would this be an acceptable quickfix for you?