Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Valgrind doesn't report allocations with jemalloc on nightly #28224

Closed
apasel422 opened this issue Sep 4, 2015 · 27 comments
Closed

Valgrind doesn't report allocations with jemalloc on nightly #28224

apasel422 opened this issue Sep 4, 2015 · 27 comments
Labels

Comments

@apasel422
Copy link
Contributor

@apasel422 apasel422 commented Sep 4, 2015

Compiling the following code with opt-level=0, Valgrind reports 26 allocs for rustc 1.2.0 (082e47636 2015-08-03), but 5 allocs for rustc 1.4.0-nightly (cd138dc44 2015-09-02):

fn main() {
    let mut t = 0;

    for i in 0..10 {
        t += *Box::new(i);
    }

    assert_eq!(t, 45);
}

Perhaps this has something to do with #27400? In any case, it's harder to use Valgrind to debug Rust programs because of this.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Sep 4, 2015

This is actually an optimization that LLVM is doing, it realizes that the allocation isn't necessary so it's elided entirely. As a result I believe this is working as intended, so closing.

@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Sep 4, 2015

I'm not sure that's the case. Consider this program:

#![feature(test)]

extern crate test;

fn main() {
    for i in 0..100 {
        let foo = Box::new(i);
        test::black_box(foo);
    }
}

The assembly clearly contains calls to exchange_malloc, and running the code under gdb indicates that exchange_malloc is called 100 times:

(gdb) break _ZN4heap15exchange_malloc20he8623aa1084055ffJfaE
Breakpoint 1 at 0x4a90: file ../src/liballoc/heap.rs, line 123.
(gdb) ignore 1 1000
Will ignore next 1000 crossings of breakpoint 1.
(gdb) run
Starting program: /home/andrew/foo 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Inferior 1 (process 1890) exited normally]
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000555555558a90 in foo::heap::exchange_malloc at ../src/liballoc/heap.rs:123
    breakpoint already hit 100 times
    ignore next 900 hits
@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Sep 4, 2015

But Valgrind reports:

==1899== Memcheck, a memory error detector
==1899== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==1899== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==1899== Command: ./foo
==1899== 
==1899== 
==1899== HEAP SUMMARY:
==1899==     in use at exit: 0 bytes in 0 blocks
==1899==   total heap usage: 5 allocs, 5 frees, 976 bytes allocated
==1899== 
==1899== All heap blocks were freed -- no leaks are possible
==1899== 
==1899== For counts of detected and suppressed errors, rerun with: -v
==1899== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
@alexcrichton alexcrichton changed the title Valgrind doesn't report allocations with nightly Valgrind doesn't report allocations with jemalloc on nightly Sep 4, 2015
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Sep 4, 2015

Hm something fishy may indeed be going on here, it looks like valgrind is no longer tracking jemalloc? If you use alloc_system for now you'll get accurate tracking via valgrind, I'll reopen for more investigation into jemalloc though.

(also tweaking the title a bit)

@alexcrichton alexcrichton reopened this Sep 4, 2015
@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Sep 4, 2015

Using alloc_system indeed causes valgrind to report the correct information.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Sep 5, 2015

Sorry to jump in here, but how did you ever get valgrind to report jemalloc stats in the first place? On OS X, I've never successfully gotten any useful stats.

@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Sep 8, 2015

I think the issue is that Valgrind can't track calls to jemalloc's mallocx-family of functions. There is a way to get Valgrind to track them, but it involves some custom headers: http://valgrind.org/docs/manual/mc-manual.html#mc-manual.mempools

@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Sep 8, 2015

jemalloc is apparently not being built with --enable-valgrind (which makes sense for performance reasons):

foo.c:

#include <stdio.h>
#include <stdlib.h>

extern void je_mallctl(const char *, void *, size_t *, void *, size_t);

int main() {
    int valgrind = 6;
    size_t len = 1;
    je_mallctl("config.valgrind", &valgrind, &len, NULL, 0);
    printf("%d\n", valgrind);
}
> gcc foo.c -L/usr/local/lib/rustlib/x86_64-unknown-linux-gnu/lib/ -l:liballoc_jemalloc-35017696.rlib -lpthread
> ./a.out 
0
@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Sep 8, 2015

Building nightly with

JEMALLOC_FLAGS='--enable-valgrind' make

results in the correct Valgrind behavior.

Not sure how we want to address this, but providing multiple versions of liballoc_jemalloc seems like a possibility.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Oct 2, 2015

FWIW, I ran your example program with rustc 1.5.0-dev (5f06607e2 2015-09-29) and valgrind-3.11.0.SVN on OS X, and it seems to track allocations:

==28285== HEAP SUMMARY:
==28285==     in use at exit: 35,221 bytes in 426 blocks
==28285==   total heap usage: 611 allocs, 185 frees, 42,405 bytes allocated

To my knowledge, I have not specified any custom flags when building. I also see that massif reports heap allocations over time, so I have some hope that things are just magically working together now!

@apasel422
Copy link
Contributor Author

@apasel422 apasel422 commented Oct 6, 2015

Valgrind on Ubuntu still indicates that this code, compiled with opt-level=0 on nightly, has only 5 allocations:

#![feature(alloc_jemalloc)]
#![feature(test)]

extern crate alloc_jemalloc;
extern crate test;

fn main() {
    for i in 0..100 {
        let foo = Box::new(i);
        test::black_box(foo);
    }
}

Removing the allocator has the same effect, while changing to alloc_system reports 112 allocations.

@Hansyperman
Copy link

@Hansyperman Hansyperman commented Apr 20, 2016

I think I reported this same issue on reddit: rust 1.8 stable lost valgrind support
https://www.reddit.com/r/rust/comments/4fddre/i_lost_valgrind_in_rust_18/

I have a mixed rust/extern "C" program. I know it leaks memory and use valgrind to verify behaviour. When I upgraded to rust stable 1.8, it proudly claims not to leak anymore, and even stops allocating memory altogether! While I'm impressed by rusts memory safety guarantees, this is a bit too much for me ;-)

Anyone knows what happens and how to make it un-happen? I read somewhere they integrated jemalloc, so maybe that broke the monitoring allocator from valgrind?

An example on debian 8.4, x86-64:

extern "C" {
  fn malloc(len: usize) -> *mut u8;
}

fn main() {
  println!("Allocated at {:?}",unsafe{malloc(100)});
}

On rust 1.8:

me@aruba:~/atest$ rustc --version
rustc 1.8.0 (db2939409 2016-04-11)
me@aruba:~/atest$ cargo build
me@aruba:~/atest$ valgrind  target/debug/atest
==2921== Memcheck, a memory error detector
==2921== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==2921== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==2921== Command: target/debug/atest
==2921== 
Allocated at 0x6035000
==2921== 
==2921== HEAP SUMMARY:
==2921==     in use at exit: 0 bytes in 0 blocks
==2921==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==2921== 
==2921== All heap blocks were freed -- no leaks are possible
==2921== 
==2921== For counts of detected and suppressed errors, rerun with: -v
==2921== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

On rust 1.7

me@aruba:~/atest$ rustc --version
rustc 1.7.0 (a5d1e7a59 2016-02-29)
me@aruba:~/proggies/atest$ cargo build
Compiling atest v0.1.0 (file:///home/hans/proggies/atest)
me@aruba:~/atest$ valgrind  target/debug/atest
==5975== Memcheck, a memory error detector
==5975== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==5975== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==5975== Command: target/debug/atest
==5975== 
Allocated at 0x5e00570
==5975== 
==5975== HEAP SUMMARY:
==5975==     in use at exit: 100 bytes in 1 blocks
==5975==   total heap usage: 7 allocs, 6 frees, 1,124 bytes allocated
==5975== 
==5975== LEAK SUMMARY:
==5975==    definitely lost: 100 bytes in 1 blocks
==5975==    indirectly lost: 0 bytes in 0 blocks
==5975==      possibly lost: 0 bytes in 0 blocks
==5975==    still reachable: 0 bytes in 0 blocks
==5975==         suppressed: 0 bytes in 0 blocks
==5975== Rerun with --leak-check=full to see details of leaked memory
==5975== 
==5975== For counts of detected and suppressed errors, rerun with: -v
==5975== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented May 4, 2016

The tools team discussed this issue during triage yesterday and the conclusion was that while not high priority we would very much like to see this issue fixed! The --enable-valgrind option mentioned by @apasel422 above seems like the best fix here, and we'd just want to investigate if that has any runtime impact or affect on runtime dependencies.

@alexcrichton alexcrichton added P-medium and removed I-nominated labels May 4, 2016
@michaelwoerister
Copy link
Contributor

@michaelwoerister michaelwoerister commented May 17, 2016

It seems that Valgrind support in jemalloc is going away:
jemalloc/jemalloc#369

@lilith
Copy link

@lilith lilith commented Aug 15, 2016

I have a mixed C/Rust project targeting windows, linux, & mac. Lots of codecs are involved, and valgrind is an essential tool for verifying that integration tests aren't exposing a new bug in libpng or another dependency (this is a very common occurrence).

Does this change mean that I should move all of my integration tests to C++? Or is a custom build of rust required?

@ssokolow
Copy link

@ssokolow ssokolow commented Aug 15, 2016

I believe you just need to ask nightly rust to build your project with the system allocator rather than the bundled jemalloc.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Aug 15, 2016

Or if you produce a shared library, it will use the system allocator instead of jemalloc by default, which should allow valgrind to work.

@lilith
Copy link

@lilith lilith commented Aug 15, 2016

@ssokolow You mean the 'official' nightly build would do this? What flags would I provide? (Building nightly from source means I can't do this in Travis, which is where I valgrind as part of CI).

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Aug 15, 2016

@lilith
Copy link

@lilith lilith commented Aug 15, 2016

Thank you! The "--enable-valgrind" context led me to misunderstand @ssokolow.

So to clarify, there's no compilation flag that can change this; I would need to inject source code #[feature(alloc_system)] extern crate alloc_system; into my tests, but only as a permutation for the nightly channel?

@ssokolow
Copy link

@ssokolow ssokolow commented Aug 15, 2016

Yes. As I understand the situation, compiling jemalloc with --enable-valgrind either is or was a temporary measure before support will be/was dropped entirely.

@lilith
Copy link

@lilith lilith commented Aug 18, 2016

There is the problem that I'm running valgrind against a totally different executable, compiled by nightly instead of stable. Essentially this means I can't valgrind products of a stable rust build.

@ssokolow
Copy link

@ssokolow ssokolow commented Aug 18, 2016

@nathanaeljones I'm no expert, but you could look into making a custom rust build with an older jemalloc and/or --enable-valgrind (as appropriate) while you wait for alloc_system support to be in stable.

@lilith
Copy link

@lilith lilith commented Sep 9, 2016

@ssokolow Thanks! For now it looks like nightly will be a requirement for other reasons anyway :)

Valgrind has been crucial for my work with unsafe Rust. As far as I can tell there's no way to make the generated test harnesses Valgrind-friendly; a standalone executable is required.

@ssokolow
Copy link

@ssokolow ssokolow commented Feb 22, 2017

As a quick update, since this is getting linked to, here's how I make the system allocator opt-in for nightly:

First, add these lines to the top of main.rs:

#![cfg_attr(feature="nightly", feature(alloc_system))]
#[cfg(feature="nightly")]
extern crate alloc_system;

Then, add these to Cargo.toml...

[features]
nightly = []

And, finally, use these commands for setup and building:

rustup toolchain add nightly-x86_64-unknown-linux-gnu
cargo +nightly build --features=nightly

For records purposes:

  • I pieced that together for an automation "script" I haven't yet released, where I want to crunch down the size of a musl build.
  • I first posted those instructions here on reddit.
  • I later cleaned up the reusable bits (so far) of the project I'd developed them in and published this CLI project boilerplate, which incorporates that code and splits it between the just install-rustup-deps and just channel=nightly build-release commands.
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jun 20, 2017

We're unlikely to really do anything to fix this, so I'm going to close this in favor of #27389. It's highly likely that all programs will start to link to jemalloc the system allocator by default once we stabilize that feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.