Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include 3rdparty/jemalloc (Fix for Ubuntu 18.10+, Debian Stable, etc.) #17121

wants to merge 1 commit into from


Copy link

@leezu leezu commented Dec 19, 2019


As of jemalloc 5, jemalloc default build can not be used in libraries that dlopened. However, is dlopened by Python (ctypes). We can thus not link to system but must rather link to a libjemalloc compiled as part of our build process.

jemalloc 5 is distributed as part of Ubuntu 18.10 and higher, as well as Debian Stable.
Users on these systems will be unable to compile MXNet with USE_JEMALLOC=ON (which is the default).

This PR contains the necessary changes.

Fixes #17145



Please feel free to remove inapplicable items for your PR.

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at$PR_ID/$BUILD_ID/index.html
  • To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change


  • Include 3rdparty/jemalloc


@leezu leezu requested a review from cjolivier01 Dec 19, 2019
@leezu leezu force-pushed the fixjemalloc branch 2 times, most recently from 221c674 to 7e99250 Compare Dec 19, 2019
Copy link

@szha szha left a comment

looks like a FreeBSD licensed project. should be ok to include from licensing perspective

Copy link
Contributor Author

@leezu leezu commented Dec 20, 2019

@yajiedesign our existing Windows CI builds do not use jemalloc (-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)). Also it's not mentioned in
So I'll not support building with 3rdparty/jemalloc in this PR on Windows for now.

Same for the edge builds.

We need to fix the upstream CMake jemalloc build on these platforms first.

@leezu leezu mentioned this pull request Dec 24, 2019
Starting with jemalloc 5 we must build jemalloc with --disable-initial-exec-tls
to support linking with

As Debian Stretch+ and Ubuntu 18.10+ ship with jemalloc 5 built without
--disable-initial-exec-tls, building MXNet with jemalloc support on any of those
platforms is currently broken.


To simplify integration with MXNet's CMake build, we rely on the yet to be
merged CMake version of jemalloc:
Copy link
Contributor Author

@leezu leezu commented Jan 21, 2020

Close this until stable upstream. See jemalloc/jemalloc#303.

#17324 disables jemalloc by default until then.

@leezu leezu closed this Jan 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

2 participants