'memalign' is unimplemented! panic

[harlanc]

The following problem occurred today, it didn't exist before:

thread '' panicked at 'not implemented: 'memalign' is unimplemented!', preload/src/api.rs:831:5

[koute]

Fixed on master.

[mwk088]

I am seeing the same error:
thread '' panicked at 'not implemented: 'memalign' is unimplemented!', preload/src/api.rs:902:5

I am using the prebuilt binary, tag 0.9. Is the fix not implemented in that version?

I did try building from source with following the instructions, however, I get the following error:
error: could not find Cargo.toml in /xxx/xxxxxxxx/xxxx or any parent directory

I assumed there would be a git pull, and building from inside the downloaded folder, but I never saw that in the instructions.

Thank you.

[koute]

The fix's not yet in any released versions; you need to compile it yourself for now.

I assumed there would be a git pull, and building from inside the downloaded folder, but I never saw that in the instructions.

Yes. That's what you're supposed to do.

[mwk088]

Thank you, doing a git clone and following the instructions seemed like it has worked. I have another question now....

Essentially, I have a python script that builds a GNURADIO flowgraph and runs it. All of the blocks are custom c++ code that is ran. Some of the c++ code is creating a memory leak and I am attempting to use bytehound to track down the culprit. Is bytehound the proper tool to use? I have to use it as shown below:

LD_PRELOAD=./libbytehound.so python3 top_block.py

I ran like you see above, but now an getting a crazy amount of errors, with the backtrace error below.

thread '' panicked at 'index out of bounds: the len is 512 but the index is 18446744073709551615', /root/.cargo/git/checkouts/not-perf-af1a46759dd83df9/18bd8d3/nwind/src/local_unwinding.rs:455:26
stack backtrace:

[koute]

That should not happen. My guess is that it's probably some longjmp-related shenanigans inside of the Python interpreter.

You can try setting MEMORY_PROFILER_USE_SHADOW_STACK=0 and try running it again; that will make allocations slower, but should prevent it from crashing.

Is bytehound the proper tool to use?

If the leak's on the C++ side then yes.

[mwk088]

Thank you for the quick response. Yes, the leak is absolutely in the c++ code and bytehound seems like it would help tremendously, with the leak being so small. I tried the above suggestion, and got the following output. Sorry about the format below.

MEMORY_PROFILER_USE_SHADOW_STACK=0 LD_PRELOAD=./libbytehound.so python3 top_block.py bytehound: 76cd 76cd WRN PR_SET_VMA_ANON_NAME is not supported (Linux 5.17+ required); will try to emulate in userspace bytehound: 76cd 76cd WRN Shadow stack overflow: capacity = 256, used = 197 + 29, available = 30, required = 31
bytehound: 76cd 76da ERR Failed to find a matching trampoline entry for stack pointer = 0x00007FFDAC066F10 bytehound: 76cd 76da ERR Shadow stack #0: return address = 0x0000558C19BD378E, slot = 0x00007FFDAC072168, stack pointer = 0x00007FFDAC072170 bytehound: 76cd 76da ERR Shadow stack #1: return address = 0x00007F5F42F12CF3, slot = 0x00007FFDAC0720A8, stack . . . thread 'bytehound: 76cd 76da ERR Shadow stack #22: return address = 0x00007F5F43FA5DB2, slot = 0x00007FFDAC0712D8, stack pointer = 0x00007FFDAC0712E0 <unnamed>bytehound: 76cd 76da ERR Shadow stack #23: return address = 0x00007F5F44001A78, slot = 0x00007FFDAC0711C8, stack pointer = 0x00007FFDAC0711D0 ' panicked at 'bytehound: 76cd 76da ERR Shadow stack #24: return address = 0x00007F5F43FE8480, slot = 0x00007FFDAC071138, stack pointer = 0x00007FFDAC071140 index out of bounds: the len is 512 but the index is 18446744073709551615bytehound: 76cd 76da ERR Shadow stack #25: return address = 0x00007F5F4400F7AF, slot = 0x00007FFDAC0710C8, stack pointer = 0x00007FFDAC0710D0 ', bytehound: 76cd 76da ERR Shadow stack #26: return address = 0x00007F5F4405106A, slot = 0x00007FFDAC070F98, stack pointer = 0x00007FFDAC070FA0 /root/.cargo/git/checkouts/not-perf-af1a46759dd83df9/18bd8d3/nwind/src/local_unwinding.rsbytehound: 76cd 76da ERR Shadow stack #27: return address = 0x00007F5F4400F7AF, slot = 0x00007FFDAC070F28, stack pointer = 0x00007FFDAC070F30 :455bytehound: 76cd 76da ERR Shadow stack #28: return address = 0x00007F5F4405106A, slot = 0x00007FFDAC070DF8, stack pointer = 0x00007FFDAC070E00 :bytehound: 76cd 76da ERR Shadow stack #29: return address = 0x00007F5F4400F7AF, slot = 0x00007FFDAC070D88, stack pointer = 0x00007FFDAC070D90 26 . . . bytehound: 76cd 76da ERR Shadow stack #234: return address = 0x00007F5F4400E3C7, slot = 0x00007FFDAC0673D8, stack pointer = 0x00007FFDAC0673E0 bytehound: 76cd 76da ERR Aborting 0: 0x7f5f449d9785 - std::backtrace_rs::backtrace::libunwind::trace::hf5399c1603f092ca at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 1: 0x7f5f449d9785 - std::backtrace_rs::backtrace::trace_unsynchronized::hc6fc34f5dc1c74f7 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x7f5f449d9785 - std::sys_common::backtrace::_print_fmt::h7d798828e39235a8 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/sys_common/backtrace.rs:65:5 3: 0x7f5f449d9785 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h12c08aa7a7a9a2af at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/sys_common/backtrace.rs:44:22 4: 0x7f5f44983eae - core::fmt::write::h221fbc267beb0404 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/core/src/fmt/mod.rs:1209:17 5: 0x7f5f449b9734 - std::io::Write::write_fmt::hf386ea1c52540785 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/io/mod.rs:1682:15 6: 0x7f5f449da5a7 - std::sys_common::backtrace::_print::h10c29fc0c250a606 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/sys_common/backtrace.rs:47:5 7: 0x7f5f449da5a7 - std::sys_common::backtrace::print::h37193727d9081e19 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/sys_common/backtrace.rs:34:9 8: 0x7f5f449da345 - std::panicking::default_hook::{{closure}}::h2fbe4b072bd77d72 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/panicking.rs:267:22 9: 0x7f5f449daab7 - std::panicking::default_hook::h47c73d4904e75e7d at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/panicking.rs:286:9 10: 0x7f5f449daab7 - std::panicking::rust_panic_with_hook::h1d0c42788e12b152 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/panicking.rs:688:13 11: 0x7f5f449da762 - std::panicking::begin_panic_handler::{{closure}}::h0792a0845aaba9b3 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/panicking.rs:579:13 12: 0x7f5f449da6cc - std::sys_common::backtrace::__rust_end_short_backtrace::haa99739b7e05bfed at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/sys_common/backtrace.rs:137:18 13: 0x7f5f449da6a1 - rust_begin_unwind at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/std/src/panicking.rs:575:5 14: 0x7f5f449143d2 - core::panicking::panic_fmt::h85ad9577e85cebac at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/core/src/panicking.rs:65:14 15: 0x7f5f44914451 - core::panicking::panic_bounds_check::h3ce4da2eb5b92015 at /rustc/06f049a35535d26d5c8426d4f782f78277b41aa1/library/core/src/panicking.rs:151:5 16: 0x7f5f4494d57f - nwind::local_unwinding::ShadowStack::push::h8d671622c33bcc05 at /root/.cargo/git/checkouts/not-perf-af1a46759dd83df9/18bd8d3/nwind/src/local_unwinding.rs:455:26 17: 0x7f5f4494d57f - nwind::local_unwinding::LocalAddressSpace::unwind_through_fresh_frames::h7fbdb4c61c9f246a at /opt/truearrow/data/bytehound/preload/src/unwind.rs:393:31 18: 0x7f5f4494d57f - bytehound::unwind::grab_with_unwind_state::hb67f09dc8a185472 at /opt/truearrow/data/bytehound/preload/src/unwind.rs:393:31 19: 0x7f5f44a50c81 - nwind_ret_trampoline_start at /root/.cargo/git/checkouts/not-perf-af1a46759dd83df9/18bd8d3/nwind/src/arch/amd64_trampoline.s:17 20: 0x0 - <unknown> Aborted (core dumped)

[koute]

That should not happen. The unwinding might be triggering before the options are initialized.

Can you try applying the following one-line patch, recompile and try again? (That will permanently hardcode the environment variable that I've told you to set.)

diff --git a/preload/src/opt.rs b/preload/src/opt.rs
index 9f4b1c1..6956ec3 100644
--- a/preload/src/opt.rs
+++ b/preload/src/opt.rs
@@ -34,7 +34,7 @@ static mut OPTS: Opts = Opts {
     disabled_by_default: false,
     enable_broadcasts: false,
     enable_server: false,
-    enable_shadow_stack: true,
+    enable_shadow_stack: false,
     grab_backtraces_on_free: true,
     include_file: None,
     output_path_pattern: Buffer::from_fixed_slice( b"memory-profiling_%e_%t_%p.dat" ),

[mwk088]

So, I updated the the file, reran the cargo commands and when trying to run my python script I got the same output. I did not officially remove bytehound then build again...I simply ran the cargo build commands.

update
I used the 'cargo clean' and rebuilt bytehound with your one-line patch and still get the same error.

[koute]

Are you sure you're using the right binary? After applying this patch it should be impossible to get the error you're getting. (Unless you're setting MEMORY_PROFILER_USE_SHADOW_STACK to true again.)

Anyway, I've pushed a few changes to master. Can you pull those changes in, recompile, and try running it again with MEMORY_PROFILER_USE_SHADOW_STACK=0?

[mwk088]

Thank you so much. I pulled the changes, recompiled and reset the MEMORY_PROFILER_USE_SHADOW_STACK. Now everything runs correctly. I must have been overriding the env variable you had me change in the code.

I am now trying to go through the results to analyze where the leak is coming from.
I have quite a few backtraces to review ~30, how does bytehound handle multi-threading. Any simplifying suggestions?
Here is what the graph looks like when looking at the backtraces

Is this a known bug?
ERROR: Function not found: only_not_deallocated_until_at_most (cli_core::script::AllocationList, cli_core::filter::Duration) (line 28, position 6)

Any chance bytehound could work with a glibc version at 2.17?

Thanks again.