Box32: x86 LD_PRELOAD dropped as ELFCLASS32 inside a system() call - expected?

[shakeheartbreak]

Question about Box32 + LD_PRELOAD inside a system() call (not a bug report, trying to understand expected behaviour)

Setup: Retroid Pocket 6 (Snapdragon 8 Gen 2, arm64), Android 13, no root, Termux + proot Debian Trixie. box64 v0.4.3 built from source with BOX32 enabled. I'm running a 32-bit x86 program through a 32-bit x86 loader that injects a hook library via LD_PRELOAD (it's an arcade loader, but I don't think the specifics matter for the question). Everything is launched through box64.

Both the hook library and the target binary are confirmed 32-bit x86 - I checked the ELF headers directly, both are 7f 45 4c 46 01 ... with e_machine = EM_386.

What happens: the loader launches fine under box64, reads its config, then spawns the target. At that point I get this in the log (BOX64_LOG=2):

[BOX32] 7589|0x40004761: Calling system (3081E09C, 3081EC9C, 00000000...) =>ERROR: ld.so: object 'mylib.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored

then the target dies with a bare:

Illegal instruction

I had BOX64_SHOWSEGV=1 set and it produced no unhandled-opcode message or signal dump, just the env echo and then the bare "Illegal instruction".

What I'm trying to understand:

When a process running under Box32 makes a system() call with LD_PRELOAD set to a 32-bit x86 .so, is Box32 expected to handle that preload, or is it expected to be passed to the host (arm64) linker, which then rejects it as ELFCLASS32? The [BOX32] tag on the line makes me think Box is at least aware of the call, but the ELFCLASS32 rejection looks like the host linker talking.
If that preload is being dropped, is there a way to make Box32 own it - BOX64_EMULATED_LIBS, an rc file, LD_PRELOAD set a particular way - so the x86 hook library actually loads into the x86 child? Or is x86 LD_PRELOAD across a system()/exec boundary just not something Box32 does right now?
Does the bare SIGILL with no "unhandled opcode" message tell me anything useful about whether the crash is the recompiler hitting something it can't translate, versus the child just running unpatched (because the preload never took) and wandering into a bad path?

I can attach the full BOX64_LOG=2 log and BOX64_DLSYM_ERROR output. Mainly I want to know whether this is expected Box32 behaviour and a known limitation, or whether I've set something up wrong. Thanks for any pointers.

[ptitSeb]

While the execv familly of library calls are wrapped and will inject a call to box64 if the target program is x86/x86_64, the system function is not, and will just call programs as-is, relying on binfmt to inject box64 for x86/x86_64 type binaries.

[ptitSeb]

Also, using an x86 or x86_64 library as a preload libraries to load an native program is also not supported (and not planned to be supported).

[shakeheartbreak]

Thank you, that's the missing piece - it explains the bare SIGILL perfectly. The loader launches the game via system(), so box64 never wraps it, and since this is rootless Termux/proot there's no binfmt_misc registered to catch the x86 binary either. So the native CPU was just being handed x86 code directly, which is the illegal instruction. The game was never actually running under box64 - that's why SHOWSEGV had nothing to say. I'd been chasing the preload and the recompiler when neither was ever in the picture.

That also makes it clear why this works on a normal Linux install (binfmt routes the system() call through box64) and why native Linux on the device would be the clean fix.

For anyone who finds this later: the two structural blockers on rootless Android are (1) system() isn't wrapped and there's no binfmt to fall back on, and (2) x86 preload into a native program isn't supported. A loader using execve instead of system() would sidestep the first, since that path is wrapped - might be worth raising with the loader project as an Android-compatibility option rather than anything box64 needs to change.

Out of interest, is wrapping system() something that's ever been considered, or is the binfmt-fallback assumption deliberate? Not asking for a feature - just trying to understand whether the execve route is the only realistic angle for rootless setups. Either way, thanks for the clear answer, it's saved me a lot of guessing.

[ptitSeb]

I have considered wrapping the system function yes, but it's more complex than execv because system is just one big command string, insteads of an array of args, which makes analysis the command string and injecting box64 much more complex. Also, most program & loader use execv (and a spawn the spawn familly), wich makes correctly wrapping system very low priority.

[shakeheartbreak]

That makes sense, thank you so much for explaining it - I hadn't appreciated that system() hands over one big string to untangle whereas execv comes pre-split, and that almost everything uses execv anyway. Completely fair that it's low priority.

For context on why I was hopeful: I've been trying to get a Sega Lindbergh arcade title (OutRun 2 SP SDX, via the Linux lindbergh-loader) running on a Retroid Pocket 6 handheld, entirely on Android with no root - the dream being a single device you just pick up, no dual-boot. Your answer's actually solved the mystery even though it's not the answer I was hoping for! The loader uses system() to launch the game, so it slips straight past box64. It works on native Linux because binfmt catches it, but on rootless Android there's nothing to. So it sounds like the realistic path is getting the loader patched to use execv, rather than anything box64 should bend for.
I'd be lying if I said I wouldn't love system() wrapping to exist one day - it'd quietly unlock this whole niche of Linux arcade loaders on Android handhelds - but I completely get why it sits where it does on the list.

Genuinely grateful for the time; you've turned a wall I'd been banging my head on into a clearer understanding of why it's there and where to go next. Have an awesome day.