client interactions with cross-arch execve

[derekbruening]

From derek.br...@gmail.com on May 23, 2009 10:55:03

With debug build and a 32-bit client, if an app does execve to a 64-bit
app, we'll kill the process with a usage error due to the 32-bit client
mismatch:

<-- parent 16846 forked child 16903 -->
<-- execve /bin/sh -->
<Application sh (16903) DynamoRIO usage error :
Error opening instrumentation library
/work/dr/tot/exports_drmemory/libdrmemory.so:
/work/dr/tot/exports_drmemory/libdrmemory.so: wrong ELF class:
ELFCLASS32>

We should at least provide an option to turn this into a non-fatal error,
to enable using debug build in such situations. In my usage case the execve
of 64-bit sh is followed by execve of 32-bit perlbmk, and I'm fine with my
client not being loaded into sh, but I do want to be loaded into perlbmk.
This is what happens today with a release build, with a non-fatal message:

<Application sh (18109). Unable to load client library:
Error opening instrumentation library
/work/dr/tot/exports_drmemory/libdrmemory.so:
/work/dr/tot/exports_drmemory/libdrmemory.so: wrong ELF class:
ELFCLASS32.>
<Starting application sh (18109)>
<Starting application perlbmk (18110)>

We should consider what model we want to support for clients in such scenarios:
specify separate 32-bit and 64-bit clients?
Xref all the similar Windows issues: PR 240257, PR 254193, issue #49.
Those Windows cases also focus on mixed-mode processes.

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=147

[derekbruening]

From derek.br...@gmail.com on May 25, 2009 14:34:25

in r162 :

• eliminated debug-build assert for 32-vs-64 errors to allow
  debug build to continue across execve chains
• will leave case open for more substantial changes depending on
  the model we want

Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-Low

[derekbruening]

This was blocking usage in https://groups.google.com/g/dynamorio-users/c/2cAuAhIBR8Y

[derekbruening]

I think there are two possible solutions and that we should do both?

• Add a new option to specify the other-arch path to the client(s). -client_lib32, -client_lib64, and shortcuts -c32 and -c64? So you could do bin64/drrun -c bin64/client64.so -c32 bin32/client32.so or bin32/drrun -c bin/client32.so -c64 bin64/client64.so or bin64/drrun -c64 bin64/client64.so -c32 bin32/client32.so.

• Have the pre-execve code try some standard substitutions for the existing client paths like s/bin32/bin64/;s/bin/bin64/;s/lib32/lib64;s/lib/lib64;.

@johnfxgalea wondering if you've ever hit this? It seems surprising that nobody has really hit this in all these years.

[derekbruening]

Note that using drconfig to create named config files for other-arch setup has always been an option, but is not always easy to use if child executable names are not known.

[derekbruening]

Hmm for the -c <path> <options> syntax where there's no delimiter to end the options exception the -- for the app, maybe it's better to have -c3264 <path32> <path64> <options-for-both>?

[johnfxgalea]

Hmm, no, I don't recall ever encountering this issue. At the moment, my experimental setup does not mix archs - I use a 32-bit OS when working on 32-bit apps.

[derekbruening]

For drmem, since not using drrun: I guess it just calls
dr_register_client() for the other arch which will create a 1-time config
file.

And for a -t tool file like drcov with no dedicated front-end: can have
CLIENT{32,64}{_REL}?

[derekbruening]

For drmem, since not using drrun: I guess it just calls
dr_register_client() for the other arch which will create a 1-time config
file.

No that would only help if the top-level process did execve to the
other arch: won't help w/ children who will have different pids.

It has to set the -client_libXX option right? No current top-level
solution via drconfiglib: has to set the option string? Should we
add an other-arch-client-path to dr_register_client()?

[derekbruening]

I'm adding an other-arch-client-path to drconfiglib via dr_register_client_ex() which takes in a new struct (so we can add more fields more easily in the future), plus new routines dr_get_client_info_ex() and dr_client_iterator_next_ex() to support querying other-bitwidth client registration via the same struct.

I decided not to add the substitution attempts above: it seems risky. If the user did not specify both client paths, DR will just fail to load the wrong-bitwidth client in the child which is a fatal error. Seem ok?

[hunterbr72]

I would vote for a -c32 -c64 solution. Who knows if it is necessary for certain clients to have dedicated options in certain cases.

[hunterbr72]

I mean separate options for the 32 and the 64 bit world.

[derekbruening]

I would vote for a -c32 -c64 solution. Who knows if it is necessary for certain clients to have dedicated options in certain cases.

So my current prototype branch for this does have the underlying capability for separate options, if you use the -client_lib32 and -client_lib64 raw DR options, or if you use the drinjectlib interface. The reason I moved toward the -c3264 is for easy option parsing. If you had separate -c32 and -c64 what you want the syntax to look like? Have a -- in between as well as separating the app args?

$ bin64/drrun -c32 /my/lib32.so -op1 for_32 -op2 for_32 -- \
  -c64 /my/lib64.so -op3 for_64 -op4 -- \
  /my/app app_arg1

[johnfxgalea]

I actually prefer the separation, but if its implementation is an overkill I don't mind -c3264.

[derekbruening]

You will always be able to do the raw -client_lib* options though they are awkward:

$ ../exports/bin64/drrun \
  -client_lib32 ~/dr/releases/DynamoRIO-Linux-8.0.0-1/samples/bin32/libbbcount.so\;0\; \
  -client_lib64 ~/dr/releases/DynamoRIO-Linux-8.0.0-1/samples/bin64/libbbcount.so\;0\; \
  -- ~/dr/test/execve64 ~/dr/test/hello32

Maybe we should just do both: -c3264 and separate -c32 and -c64, with the extra --.

[hunterbr72]

yup, agree, both would be nice, but as John said, depends on the effort.

[derekbruening]

Note that none of the drrun proposals here allow setting separate DR options: only separate client options. You can set separate DR options with drinjectlib, or the tool file interface.

[derekbruening]

#4324 has my proposal to solve this. It uses separate -c32 -c64 for drrun, -client_lib{32,64} for core DR, new interfaces for drconfiblib, and CLIENT{32,64}_{REL,ABS} for tool files.