Skip to content
This repository has been archived by the owner on May 23, 2024. It is now read-only.
/ memkind Public archive

Memkind is an easy-to-use, general-purpose allocator which helps to fully utilize various kinds of memory available in the system, including DRAM, NVDIMM, and HBM

License

Notifications You must be signed in to change notification settings

memkind/memkind

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MEMKIND

Build Status MEMKIND version Coverage Status Packaging status Language grade: C/C++

⚠️ Discontinuation of the project

The memkind project will no longer be maintained by Intel.

  • Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.
  • Intel no longer accepts patches to this project.
  • If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.

If you're interested in similar functionality you can take a look at the UMF project.

Introduction

The memkind library is a user extensible heap manager built on top of jemalloc which enables control of memory characteristics and a partitioning of the heap between kinds of memory. The kinds of memory are defined by operating system memory policies that have been applied to virtual address ranges. Memory characteristics supported by memkind without user extension include control of NUMA and page size features. The jemalloc non-standard interface has been extended to enable specialized arenas to make requests for virtual memory from the operating system through the memkind partition interface. Through the other memkind interfaces the user can control and extend memory partition features and allocate memory while selecting enabled features. Memkind interface allows to create and control file-backed memory (PMEM kind) on specified device.

Contents

  1. Interfaces
  2. Building and Installing
  3. Dependencies
  4. Kernel
  5. NVDIMM volatile usage
  6. The Detection Mechanism of the Kind
  7. Setting Logging Mechanism
  8. Setting Heap Manager
  9. Simulating High Bandwidth Memory
  10. Identification of High Bandwidth Memory
  11. Projects
  12. Testing
  13. Notes
  14. Disclaimer

Interfaces

The memkind library delivers five interfaces:

  • hbwmalloc.h - recommended for high-bandwidth memory use cases (stable)
  • memkind.h - generic interface for more complex use cases (stable)
  • pmem_allocator.h - the C++ allocator that satisfies the C++ Allocator requirements used for PMEM use cases (stable)
  • memkind_allocator.h - the C++ allocator that satisfies the C++ Allocator requirements used for static kinds (stable)
  • fixed_allocator.h - the C++ allocator that satisfies the C++ Allocator requirements used for fixed kind, created on area supplied by the user (experimental)

For more detailed information about those interfaces see corresponding manpages (located in man/ subdir):

man memkind

man hbwmalloc

man pmemallocator

man memkindallocator

man fixedallocator

Building and Installing

The memkind library has a dependency on a related fork of jemalloc. The configure scripts and gtest source code are distributed with the source tarball included in the source RPM, and this tarball is created with the memkind "make dist" target. In contrast to the distributed source tarball, the git repository does not include any generated files. For this reason some additional steps are required when building from a checkout of the git repo. Those steps include running the bash script called "autogen.sh" prior to configure. This script will populate a VERSION file based on "git describe", and use autoreconf to generate a configure script.

Building and installing memkind in standard system location can be as simple as typing the following while in the root directory of the source tree:

./autogen.sh
./configure
make
make install

To install this library into other locations, you can use the prefix variable, e.g.:

./autogen.sh
./configure --prefix=/usr
make
make install

This will install files to /usr/lib, /usr/include, /usr/share/doc/, usr/share/man.

See the output of:

./configure --help

for more information about either the memkind or the jemalloc configuration options.

To generate man files in the man/ directory, execute:

make man-local

RPM

Generating an RPM package could be done on Fedora OS and you will need to install:

  • rpmdevtools
  • libdaxctl-devel (despite the fact that it is normally an optional package)
  • hwloc-devel (despite the fact that it is normally an optional package)

Having these dependencies installed, you can generate the RPM package executing:

make rpm

jemalloc

The jemalloc source was forked from jemalloc version 5.3.0. This source tree is located within the jemalloc subdirectory of the memkind source. The jemalloc source code has been kept close to the original form, except for the following items:

  • extend jemalloc API with "arenalookupx" and "check_reallocatex" functions
  • optimization for searching commands in mallctl
  • the build system has been slightly modified.

Dependencies

You will need to install required packages in the system, to build from sources:

  • autoconf
  • automake
  • gcc-c++
  • numactl-devel / libnuma-devel
  • libtool
  • unzip

For using automatic recognition of PMEM NUMA in MEMKIND_DAX_KMEM:

  • libdaxctl-devel / libdaxctl-dev (v66 or later)

For using HMAT features in memkind:

  • hwloc-devel / libhwloc-dev (v2.3.0 or later)

You will need to install pandoc for generating man documentation.

Run Requirements

You will need to install required packages for applications, which are using the memkind library for dynamic linking at runtime:

  • numactl-libs / libnuma
  • numactl
  • pthread

Kind Requirements

The table below summarizes extra requirements, depending on used memory kind:

Memory kind NUMA HBW Memory Hugepages Device DAX Filesystem supporting hole punching HMAT libhwloc
MEMKIND_DEFAULT
MEMKIND_HIGHEST_CAPACITY X
MEMKIND_HIGHEST_CAPACITY_PREFERRED X
MEMKIND_HIGHEST_CAPACITY_LOCAL X X
MEMKIND_HIGHEST_CAPACITY_LOCAL_PREFERRED X X
MEMKIND_LOWEST_LATENCY_LOCAL X X X
MEMKIND_LOWEST_LATENCY_LOCAL_PREFERRED X X X
MEMKIND_HIGHEST_BANDWIDTH_LOCAL X X* X
MEMKIND_HIGHEST_BANDWIDTH_LOCAL_PREFERRED X X* X
MEMKIND_HUGETLB X X
MEMKIND_HBW X X X* X*
MEMKIND_HBW_ALL X X X* X*
MEMKIND_HBW_HUGETLB X X X X* X*
MEMKIND_HBW_ALL_HUGETLB X X X X* X*
MEMKIND_HBW_PREFERRED X X X* X*
MEMKIND_HBW_PREFERRED_HUGETLB X X X X* X*
MEMKIND_HBW_INTERLEAVE X X X* X*
MEMKIND_REGULAR X
MEMKIND_DAX_KMEM X X
MEMKIND_DAX_KMEM_ALL X X
MEMKIND_DAX_KMEM_PREFERRED X X
MEMKIND_DAX_KMEM_INTERLEAVE X X
PMEM kind X
Fixed kind

*not needed for Intel's 2nd generation Intel(R) Xeon Phi(TM)

Kernel

To correctly control NUMA, huge pages and file-backed memory following requirements regarding Linux kernel must be satisfied:

  • NUMA

Requires kernel patch introduced in Linux v3.11 that impacts functionality of the NUMA system calls. Red Hat has back-ported this patch to the v3.10 kernel in the RHEL 7.0 GA release, so RHEL 7.0 onward supports memkind even though this kernel version predates v3.11.

  • Hugepages

Functionality related to hugepages allocation require patches patch1 and patch2 Without them physical memory may end up being located on incorrect NUMA node.

  • 2MB Pages

To use the interfaces for obtaining 2MB pages please be sure to follow the instructions here and pay particular attention to the use of the procfs files:

/proc/sys/vm/nr_hugepages
/proc/sys/vm/nr_overcommit_hugepages

for enabling the kernel's huge page pool.

  • Filesystem supporting hole punching

To use the PMEM kind, please be sure that filesystem which is used for PMEM creation supports FALLOC_FL_PUNCH_HOLE flag.

  • Device DAX

To use MEMKIND_DAX_KMEM_* kinds, you need at least Linux Kernel 5.1 (with enabled DEV_DAX_KMEM Kernel option), which contains following changes:

and created DAX device. See article for details.

  • HMAT

To utilize memory performance characteristics (latency and bandwidth), beside platform support you need at least Linux Kernel 5.5 (with enabled CONFIG_ACPI_HMAT Kernel option).

  • libhwloc

To get locality characteristics of NUMA nodes in the system you need hwloc-devel v2.3.0 (or later).

NVDIMM volatile usage

Memkind supports using persistent memory as an extension of DRAM. This volatile memory solution is provided by the library with two separate ways described below.

DAX device

NVDIMM memory as DAX device is supported by MEMKIND_DAX_KMEM_* kinds. With this solution persistent memory will be seen in OS as separate NUMA nodes.

Memkind allows two ways to use this kind:

  • first, implicitly, by allowing memkind library for automatic recognition of NUMA nodes created from persistent memory using libdaxctl-devel,
  • second, explicitly, by using MEMKIND_DAX_KMEM_NODES environment variable (please refer to memkind(3) section "ENVIRONMENT") set to comma separated list of NUMA nodes which will be treated as NUMA nodes created from persistent memory; this solution overrides the first one.

DAX filesystem

PMEM kind supports the traditional malloc/free interfaces on a memory mapped file. This allows the use of persistent memory as volatile memory, for cases where the region of persistent memory is useful to an application, but when the application doesn't need it to be persistent. PMEM kind is most useful when used with Direct Access storage DAX, which is memory-addressable persistent storage that supports load/store access without being paged via the system page cache.

Application using memkind library supports managing a data placement:

Data placement Memory kind
PMEM (fsdax) PMEM kind
PMEM (devdax) MEMKIND_DAX_KMEM kind
DRAM e.g. MEMKIND_DEFAULT kind

Currently, the PMEM kind is supported only by the jemalloc heap manager.

The Detection Mechanism of the Kind

One of the notable features of the memkind is to detect the correct kind of previously allocated memory.

Operations Memkind API function
Freeing memory memkind_free(kind, ptr)
Reallocating memory memkind_realloc(kind, ptr, size)
Obtaining the size of allocated memory memkind_malloc_usable_size(kind, ptr)
Reallocating memory to reduce fragmentation memkind_defrag_reallocate(kind, ptr)

Operations above could be unified for all used memory kinds by passing a NULL value as a kind to the functions mentioned above.

For more details, please see the following example.

Important Notes: The lookup for correct kind could result in serious performance penalty, which can be avoided by specifying a correct kind explicitly.

Setting Logging Mechanism

In memkind library logging mechanism could be enabled by setting MEMKIND_DEBUG environment variable. Setting MEMKIND_DEBUG to "1" enables printing messages like errors and general information about environment to stderr.

Logs can be saved to files instead of being printed to stderr. Each process using memkind can save logs to a separate file, the name of which is associated with the PID. Files are saved to the location from which an application using memkind was started. This feature could be enabled by passing the following variable to the configure script:

./configure --enable-logging-to-file

Setting Heap Manager

In memkind library heap management can be adjusted with MEMKIND_HEAP_MANAGER environment variable, which allows for switching to one of the available heap managers. Accepted values:

  • JEMALLOC - sets the jemalloc heap manager
  • TBB - sets Intel Threading Building Blocks heap manager. This option requires installed Intel Threading Building Blocks library.

If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager will be used by default.

Simulating High Bandwidth Memory

A method for testing for the benefit of high bandwidth memory on a dual socket Intel(R) Xeon(TM) system is to use the QPI bus to simulate slow memory. This is not an accurate model of the bandwidth and latency characteristics of the Intel's 2nd generation Intel(R) Xeon Phi(TM) Product Family on package memory, but is a reasonable way to determine which data structures rely critically on bandwidth.

If the application a.out has been modified to use high bandwidth memory with the memkind library then this can be done with numactl as follows with the bash shell:

export MEMKIND_HBW_NODES=0
numactl --membind=1 --cpunodebind=0 a.out

or with csh:

setenv MEMKIND_HBW_NODES 0
numactl --membind=1 --cpunodebind=0 a.out

The MEMKIND_HBW_NODES environment variable set to zero will bind high bandwidth allocations to NUMA node 0. The --membind=1 flag to numactl will bind standard allocations, static and stack variables to NUMA node 1. The --cpunodebind=0 option to numactl will bind the process threads to CPUs associated with NUMA node 0. With this configuration standard allocations will be fetched across the QPI bus, and high bandwidth allocations will be local to the process CPU.

Identification of High Bandwidth Memory

High bandwidth memory is identified using hwloc, if available. The default threshold for identifying high bandwidth memory is 200 GB/s, you can change the threshold using the MEMKIND_HBW_THRESHOLD environment variable.

MEMKIND_HBW_THRESHOLD is in MB/s. For example, to set to 100 GB/s:

export MEMKIND_HBW_THRESHOLD=102400.

Projects

The following software use memkind library:

Testing

All existing tests pass. For more information on how to execute tests see the CONTRIBUTING file.

When tests are run on a NUMA platform without high bandwidth memory the MEMKIND_HBW_NODES environment variable is used in conjunction with "numactl --membind" to force standard allocations to one NUMA node and high bandwidth allocations through a different NUMA node. See next section for more details.

Notes

  • Using memkind with Transparent Huge Pages enabled may result in undesirably high memory footprint. To avoid that disable THP using following instruction
  • Different interfaces can represent different maturity level (as described in corresponding man pages).
  • Feedback on design and implementation is greatly appreciated.

Disclaimer

SEE COPYING FILE FOR LICENSE INFORMATION.

THIS SOFTWARE IS PROVIDED AS A DEVELOPMENT SNAPSHOT TO AID COLLABORATION AND WAS NOT ISSUED AS A RELEASED PRODUCT BY INTEL.

About

Memkind is an easy-to-use, general-purpose allocator which helps to fully utilize various kinds of memory available in the system, including DRAM, NVDIMM, and HBM

Resources

License

Stars

Watchers

Forks

Packages

No packages published