Lots of unclassified heap memory

[Ms2ger]

For example:

Begin memory reports
|
|  371.13 MiB -- explicit
|   17592186044408.38 MiB -- system-heap-unclassified
|     335.96 MiB -- jemalloc-heap-unclassified
|      39.66 MiB -- url(https://en.wikipedia.org/wiki/Main_Page)
|         15.75 MiB -- js
|             7.62 MiB -- malloc-heap
|             4.14 MiB -- non-heap
|             4.00 MiB -- gc-heap
|                3.34 MiB -- used
|                0.49 MiB -- decommitted
|                0.12 MiB -- admin
|                0.06 MiB -- unused
|         14.07 MiB -- layout-thread
|            13.73 MiB -- stylist
|             0.34 MiB -- display-list
|             0.00 MiB -- local-context
|          3.33 MiB -- dom-tree
|          1.09 MiB -- layout-worker-1-local-context
|          1.09 MiB -- layout-worker-2-local-context
|          1.09 MiB -- layout-worker-3-local-context
|          1.08 MiB -- layout-worker-5-local-context
|          1.08 MiB -- layout-worker-4-local-context
|          1.08 MiB -- layout-worker-0-local-context
|       3.12 MiB -- compositor-thread
|          2.34 MiB -- surface-map
|          0.78 MiB -- layer-tree
|
|  516.08 MiB -- resident-according-to-smaps
|     458.34 MiB -- anonymous (rw-p)
|      34.15 MiB -- /target/release/servo (r-xp)
|      12.49 MiB -- other
|       3.46 MiB -- /usr/lib/x86_64-linux-gnu/dri/i965_dri.so (r-xp)
|       2.21 MiB -- /target/release/servo (r--p)
|       1.50 MiB -- /lib/x86_64-linux-gnu/libc-2.19.so (r-xp)
|       1.40 MiB -- /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (r-xp)
|       0.73 MiB -- /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0 (r-xp)
|       0.72 MiB -- /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19 (r-xp)
|       0.55 MiB -- /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0 (r-xp)
|       0.53 MiB -- /usr/lib/x86_64-linux-gnu/libfreetype.so.6.11.1 (r-xp)
|
|  373.42 MiB -- jemalloc-heap-active
|  359.86 MiB -- jemalloc-heap-allocated
|  667.00 MiB -- jemalloc-heap-mapped
|  512.55 MiB -- resident
|    0.00 MiB -- system-heap-allocated
| 1037.36 MiB -- vsize
|
End memory reports

@froydnj @nnethercote do you know if we could get DMD set up for Servo?

[nnethercote]

|  371.13 MiB -- explicit
|   17592186044408.38 MiB -- system-heap-unclassified
|     335.96 MiB -- jemalloc-heap-unclassified

Something's clearly gone wrong there. Each tree's sub-trees are supposed to add up, and 17592186044408.38 MiB is clearly bogus.

I wanted to do DMD for Servo but never got around to it. The hard part was wrapping the heap allocator. I talked with @acrichton about the required machinery for this, which is theoretically in place, but I never tried it out in earnest. DMD also requires that you can get backtraces (to store) on each allocation, and that wasn't in place when I was looking at it, though it might be now.

[jdm]

We have the ability to grab backtraces while running Rust code: Backtrace::new.

[nnethercote]

Oh, the other complication for DMD with Servo is that you have two malloc heaps -- the system heap, used by all the C/C++ libraries that Servo uses, and the jemalloc heap used by Rust. I never worked out how to deal with that. This problem doesn't occur in Firefox because all heap allocations in Firefox get routed through jemalloc.

[nnethercote]

We have the ability to grab backtraces while running Rust code: Backtrace::new.

Lovely! Looks like a nice design.

[jdm]

Given https://github.com/rust-lang/rfcs/blob/master/text/1183-swap-out-jemalloc.md and rust-lang/rust#27389, it looks like we might be getting close to being able to create a DMD tool for Servo.

[jdm]

I have Servo building with a custom jemalloc-based allocator with some small changes. In particular, we need to use permanent forks of the libc and memchr crates that disable their attempts to re-export the std crate. This fork of japaric's extraction of rustc's jemalloc allocator allows modifying the allocator functions to add custom behaviour (I added a puts call every time an allocation occurred in a local clone).

[jdm]

Next step is to figure out the parts of http://dxr.mozilla.org/mozilla-central/source/memory/replace/dmd/DMD.cpp that need to be ported.

[nnethercote]

rust-lang/rfcs#916 describes in detail my previous attempt at DMD for Servo. RFC 1183 may have changed things significantly -- I haven't looked closely at it -- but that issue would still be a good starting point for anyone wanting to work on this.

[nox]

For anyone reading this and being confused, DMD stands for "dark matter detector".

[nnethercote]

https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD are the docs for DMD.

[nox]

@nnethercote How is this going?

[nnethercote]

I am not working on this.