Doorstop segmentation fault with Rust (Linux)

[jbrazio]

Hi I've tried to make Doorstop work with Rust (ds), on Windows all works but on Linux I always get a segmentation fault.
Host system is a Debian 11, I'm using the supplied start script with Doorstop for Linux.
The game uses engine version 2019.4.7f1 (e992b1a16e65).
Any help would be appreciated, thanks.

==11338== Invalid read of size 8
==11338==    at 0x484A4C7: ??? (in /srv/rust/carbon/data/Steam/steamapps/common/rust_dedicated/libdoorstop.so)
==11338==    by 0x484A65D: ??? (in /srv/rust/carbon/data/Steam/steamapps/common/rust_dedicated/libdoorstop.so)
==11338==    by 0x484ABBE: plthook_open_by_handle (in /srv/rust/carbon/data/Steam/steamapps/common/rust_dedicated/libdoorstop.so)
==11338==    by 0x484998F: doorstop_ctor (in /srv/rust/carbon/data/Steam/steamapps/common/rust_dedicated/libdoorstop.so)
==11338==    by 0x400FFE1: call_init.part.0 (dl-init.c:72)
==11338==    by 0x40100E8: call_init (dl-init.c:30)
==11338==    by 0x40100E8: _dl_init (dl-init.c:119)
==11338==    by 0x40010C9: ??? (in /lib/x86_64-linux-gnu/ld-2.31.so)
==11338==    by 0x2: ???
==11338==    by 0x1FFEFFFEE6: ???
==11338==    by 0x1FFEFFFEF6: ???
==11338==    by 0x1FFEFFFF01: ???
==11338==  Address 0x1003e0003 is not stack'd, malloc'd or (recently) free'd

[mo10]

Try doorstop_linux_debug_4.0.0.zip to see if there is more detailed output.

[jbrazio]

Failed to hook dlsym, ignoring it. Error: no such function: dlsym
Failed to hook fclose, ignoring it. Error: no such function: fclose
Failed to hook dlsym, ignoring it. Error: no such function: dlsym
Failed to hook fclose, ignoring it. Error: no such function: fclose
Failed to hook dup2, ignoring it. Error: no such function: dup2

Other than those errors.. only the Valgrind output I just pasted earlier.

==18915== Invalid read of size 8
==18915==    at 0x484AE8B: find_dyn_by_tag (plthook_elf.c:407)
==18915==    by 0x484B29E: plthook_open_real (plthook_elf.c:627)
==18915==    by 0x484ACAD: plthook_open_by_handle (plthook_elf.c:310)
==18915==    by 0x484A9DC: doorstop_ctor (entrypoint.c:77)
==18915==    by 0x400FFE1: call_init.part.0 (dl-init.c:72)
==18915==    by 0x40100E8: call_init (dl-init.c:30)
==18915==    by 0x40100E8: _dl_init (dl-init.c:119)
==18915==    by 0x40010C9: ??? (in /lib/x86_64-linux-gnu/ld-2.31.so)
==18915==    by 0x2: ???
==18915==    by 0x1FFEFFFE6E: ???
==18915==    by 0x1FFEFFFE7E: ???
==18915==    by 0x1FFEFFFE89: ???
==18915==  Address 0x1003e0003 is not stack'd, malloc'd or (recently) free'd
==18915==
==18915==
==18915== Process terminating with default action of signal 11 (SIGSEGV)
==18915==  Access not within mapped region at address 0x1003E0003
==18915==    at 0x484AE8B: find_dyn_by_tag (plthook_elf.c:407)
==18915==    by 0x484B29E: plthook_open_real (plthook_elf.c:627)
==18915==    by 0x484ACAD: plthook_open_by_handle (plthook_elf.c:310)
==18915==    by 0x484A9DC: doorstop_ctor (entrypoint.c:77)
==18915==    by 0x400FFE1: call_init.part.0 (dl-init.c:72)
==18915==    by 0x40100E8: call_init (dl-init.c:30)
==18915==    by 0x40100E8: _dl_init (dl-init.c:119)
==18915==    by 0x40010C9: ??? (in /lib/x86_64-linux-gnu/ld-2.31.so)
==18915==    by 0x2: ???
==18915==    by 0x1FFEFFFE6E: ???
==18915==    by 0x1FFEFFFE7E: ???
==18915==    by 0x1FFEFFFE89: ???
==18915==  If you believe this happened as a result of a stack
==18915==  overflow in your program's main thread (unlikely but
==18915==  possible), you can try to increase the size of the
==18915==  main thread stack using the --main-stacksize= flag.
==18915==  The main thread stack size used in this run was 8388608.
==18915==
==18915== HEAP SUMMARY:
==18915==     in use at exit: 55,186 bytes in 31 blocks
==18915==   total heap usage: 34 allocs, 3 frees, 132,018 bytes allocated
==18915==
==18915== Searching for pointers to 31 not-freed blocks
==18915== Checked 2,485,480 bytes
==18915==
==18915== LEAK SUMMARY:
==18915==    definitely lost: 0 bytes in 0 blocks
==18915==    indirectly lost: 0 bytes in 0 blocks
==18915==      possibly lost: 0 bytes in 0 blocks
==18915==    still reachable: 55,186 bytes in 31 blocks
==18915==                       of which reachable via heuristic:
==18915==                         stdstring          : 89 bytes in 1 blocks
==18915==         suppressed: 0 bytes in 0 blocks
==18915== Rerun with --leak-check=full to see details of leaked memory
==18915==
==18915== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==18915==
==18915== 1 errors in context 1 of 1:
==18915== Invalid read of size 8
==18915==    at 0x484AE8B: find_dyn_by_tag (plthook_elf.c:407)
==18915==    by 0x484B29E: plthook_open_real (plthook_elf.c:627)
==18915==    by 0x484ACAD: plthook_open_by_handle (plthook_elf.c:310)
==18915==    by 0x484A9DC: doorstop_ctor (entrypoint.c:77)
==18915==    by 0x400FFE1: call_init.part.0 (dl-init.c:72)
==18915==    by 0x40100E8: call_init (dl-init.c:30)
==18915==    by 0x40100E8: _dl_init (dl-init.c:119)
==18915==    by 0x40010C9: ??? (in /lib/x86_64-linux-gnu/ld-2.31.so)
==18915==    by 0x2: ???
==18915==    by 0x1FFEFFFE6E: ???
==18915==    by 0x1FFEFFFE7E: ???
==18915==    by 0x1FFEFFFE89: ???
==18915==  Address 0x1003e0003 is not stack'd, malloc'd or (recently) free'd

[mo10]

It seems to be accessing the wrong address. Can you add some output here?

UnityDoorstop/src/nix/plthook/plthook_elf.c

Lines 406 to 414 in 06620de

	static const Elf_Dyn *find_dyn_by_tag(const Elf_Dyn *dyn, Elf_Sxword tag) {
	while (dyn->d_tag != DT_NULL) {
	if (dyn->d_tag == tag) {
	return dyn;
	}
	dyn++;
	}
	return NULL;
	}

There may be some compilation errors. I didn't check it in detail.

static const Elf_Dyn *find_dyn_by_tag(const Elf_Dyn *dyn, Elf_Sxword tag) {
    printf("[find_dyn_by_tag] Called dyn=%p tag=%" ELF_SXWORD_FMT "\n", dyn, tag);
    while (dyn->d_tag != DT_NULL) {
        printf("[find_dyn_by_tag] Current dyn=%p dyn->d_tag=%" ELF_SXWORD_FMT "(%p)\n", dyn, dyn->d_tag, &(dyn->d_tag));
        if (dyn->d_tag == tag) {
            printf("[find_dyn_by_tag] Expect dyn=%p dyn->d_tag=%" ELF_SXWORD_FMT "(%p)\n", dyn, dyn->d_tag, &(dyn->d_tag));
            return dyn;
        }
        dyn++;
        printf("[find_dyn_by_tag] Next dyn=%p dyn->d_tag=(%p)\n", dyn, &(dyn->d_tag));
    }
    printf("[find_dyn_by_tag] No expect\n");
    return NULL;
}

[sanjay900]

I found another game, "Clone Hero" that also has this issue. However, i found that changing the following was enough to get it working for me, but i am not really sure why.

diff --git a/src/nix/entrypoint.c b/src/nix/entrypoint.c
index 1bf4556..99a645e 100644
--- a/src/nix/entrypoint.c
+++ b/src/nix/entrypoint.c
@@ -74,7 +74,7 @@ __attribute__((constructor)) void doorstop_ctor() {
 
     void *unity_player = plthook_handle_by_name("UnityPlayer");
 
-    if (unity_player && plthook_open_by_handle(&hook, unity_player) == 0) {
+    if (unity_player && plthook_open_by_address(&hook, unity_player) == 0) {
         LOG("Found UnityPlayer, hooking into it instead");
     } else if (plthook_open(&hook, NULL) != 0) {
         LOG("Failed to open current process PLT! Cannot run Doorstop! "

[mo10]

Hmmm. I changed it here with #27

https://github.com/NeighTools/UnityDoorstop/pull/27/files#diff-6e90b97801a37aecb0ca80baea71f9845e5bd8372944bd357f0d38791cc830bcR77

I tried it on steamos(Arch based) and macos(intel & m1).
Let me double check. May be some platform difference here?

[sanjay900]

Interesting, i am using Arch myself, are there any specific details you would like about my config?

[mo10]

No, I tried it on steam deck.
I remember plthook_handle_by_name called dlopen on macos. Its return should not be passed into plthook_open_by_address (dladdr).

UnityDoorstop/src/nix/plthook/plthook_osx.c

Lines 110 to 121 in a6d6d61

	void *plthook_handle_by_name(const char *name) {
	void *mono_handle = NULL;
	uint32_t cnt = _dyld_image_count();
	for (uint32_t idx = 0; idx < cnt; idx++) {
	const char *image_name = idx ? _dyld_get_image_name(idx) : NULL;
	if (image_name && strstr(image_name, name)) {
	mono_handle = dlopen(image_name, RTLD_LAZY | RTLD_NOLOAD);
	return mono_handle;
	}
	}
	return NULL;
	}

Linux uses dl_iterate_phdr return dlpi_addr. These two addresses may not be the same thing....

UnityDoorstop/src/nix/plthook/plthook_elf.c

Lines 254 to 271 in a6d6d61

	int proc_handles(struct dl_phdr_info *info, size_t size, void *data) {
	struct handle_by_name_helper *result = (struct handle_by_name_helper *)data;
	if (result->result)
	return 0;
	if (info->dlpi_name && strstr(info->dlpi_name, result->find_name)) {
	result->result = (void *)info->dlpi_addr;
	return 0;
	}
	return 1;
	}
	void *plthook_handle_by_name(const char *name) {
	struct handle_by_name_helper result = {name, NULL};
	dl_iterate_phdr(&proc_handles, &result);
	return result.result;
	}

struct dl_phdr_info {
    ElfW(Addr)        dlpi_addr;  /* Base address of object */
    const char       *dlpi_name;  /* (Null-terminated) name of
                                     object */
    const ElfW(Phdr) *dlpi_phdr;  /* Pointer to array of
                                     ELF program headers
                                     for this object */
    ElfW(Half)        dlpi_phnum; /* # of items in dlpi_phdr */
};

[sanjay900]

When i was trying things out, i printed both dlpi_addr and the handle returned by dlopen(dlpi_name, RTLD_LAZY | RTLD_NOLOAD), and found that they were both similar but different addresses.

[mo10]

So plthook_handle_by_name returns handle on macos, but return address on linux? I'm not clear about this.

Can you check lmap value in plthook_open_by_handle?

UnityDoorstop/src/nix/plthook/plthook_elf.c

Line 306 in 06620de

if (dlinfo(hndl, RTLD_DI_LINKMAP, &lmap) != 0) {

[mo10]

My fault. The tests I built before used unpatched code.

There may need a macro to determine whether to select 'plthook_open_by_handle' or 'plthook_open_by_address'

[mo10]

Can you try this? It should work fine on every platform.

[sanjay900]

Yep, that seems to be working just fine on Linux now

[jbrazio]

Hi guys, I've been following the evolution of the topic but I can only test it on the origin system later today (UTC timezone).

[jbrazio]

#41 fixes it. 🚀
Thanks for the support and time to fix it.