MCL is a modern task-based, asynchronous programming model for extremely heterogeneous systems. MCL consists of a single scheduler process and multiple, independent, multi-threaded MCL applications executed concurrently on the same compute node. For example, MCL can seamlessly support the execution of multiple Message Passing Interface (MPI) ranks within a single node as well independent applications programmed with separate programming models and runtimes, e.g., an OpenMP application next to a PyTorch application. Users need not be aware of other applications executed on the same compute node, as the MCL scheduler coordinates access to shared computing resources.
Please, visit the MCL website for a full list of updates.
- Summary
- Installation
- Testing
- Contacts
MCL aims at abstracting the low-level hardware details of a system, supporting the execution of complex workflows that consists of multiple, independent applications (e.g., scientific simulation coupled with in-situ analysis or AI frameworks that analyze the results of a physic simulation), and performing efficient and asynchronous execution of computation tasks. MCL is not meant to be the programming model employed by domain scientists to implement their algorithms, but rather to support several high-level Domain-Specific Language (DSL)s and programming model runtimes.
An MCL application consists of a sequence of tasks that need to be executed on available computing resources. The MCL programming model Application Programming Interface (API) allows user to specify tasks and control dependencies among tasks. Once submitted, tasks are scheduled for execution on a specific device by the MCL scheduler, according to the scheduling algorithm in use. MCL leverages the OpenCL library and API to interface with computing devices and express computational kernels. Normally, users do not need to directly write OpenCL kernels, as they are automatically generated by the higher-level DSL compiler, though directly writing OpenCL kernels and implementing an algorithm using the MCL API is certainly possible. OpenCL allows MCL to execute the same computational kernel on different computing devices, including, CPUs, GPUs, and FPGAs. MCL has been shown to effectively leverage heterogeneous computing resources and scale up to complex multi- device systems and down to efficient embedded systems. Code developed on a laptop computer seamlessly scales to powerful multi-GPU workstations without any modification.
- Feb 2023: PPoPP'23 Tutorial
- June 2022: ICS'22 Tutorial
- February 2022: PPoPP'22 Tutorial
- February 2021: PPoPP'21 Tutorial
Building MCL requires the following libraries and headers to be installed:
- A C compiler toolchain
- OpenCL headers (https://github.com/KhronosGroup/OpenCL-Headers)
- OpenCL libraries
- OpenCL-compatible devices
- UTHash and UTList headers (https://github.com/troydhanson/uthash)
- libAtomicOps (https://github.com/ivmai/libatomic_ops)
Headers and libraries should be installed in path that can either be reachable or specified during the configuration step.
OpenCL libaries for each device should be installed in the system. Genearlly:
- Most NVIDIA CUDA installation come with OpenCL libraries.
- Most AMD installations come with OpenCL libraries
- For CPU OpenCL implementations, Intel, AMD, ARM, and Apple provide OpenCL libraries.
- POCL2 is an open-source OpenCL library for CPU (Intel, AMD, ARM, etc.) and NVIVIDA GPUS (http://portablecl.org)
First, import the submodules:
git submodule init
git submodule update
If you cloned the GIT repository, create the configure script, otherwise skip to the next steps:
autoreconf --install
Run the configure script to configure MCL.
`configure' configures Minos Computing Library 0.5 to adapt to many kinds of systems.
Usage: ./configure [OPTION]... [VAR=VALUE]...
To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE. See below for descriptions of some of the useful variables.
Defaults for the options are specified in brackets.
Configuration:
-h, --help display this help and exit
--help=short display options specific to this package
--help=recursive display the short help of all the included packages
-V, --version display version information and exit
-q, --quiet, --silent do not print `checking ...' messages
--cache-file=FILE cache test results in FILE [disabled]
-C, --config-cache alias for `--cache-file=config.cache'
-n, --no-create do not create output files
--srcdir=DIR find the sources in DIR [configure dir or `..']
Installation directories:
--prefix=PREFIX install architecture-independent files in PREFIX
[/usr/local]
--exec-prefix=EPREFIX install architecture-dependent files in EPREFIX
[PREFIX]
By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc. You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.
For better control, use the options below.
Fine tuning of the installation directories:
--bindir=DIR user executables [EPREFIX/bin]
--sbindir=DIR system admin executables [EPREFIX/sbin]
--libexecdir=DIR program executables [EPREFIX/libexec]
--sysconfdir=DIR read-only single-machine data [PREFIX/etc]
--sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com]
--localstatedir=DIR modifiable single-machine data [PREFIX/var]
--runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run]
--libdir=DIR object code libraries [EPREFIX/lib]
--includedir=DIR C header files [PREFIX/include]
--oldincludedir=DIR C header files for non-gcc [/usr/include]
--datarootdir=DIR read-only arch.-independent data root [PREFIX/share]
--datadir=DIR read-only architecture-independent data [DATAROOTDIR]
--infodir=DIR info documentation [DATAROOTDIR/info]
--localedir=DIR locale-dependent data [DATAROOTDIR/locale]
--mandir=DIR man documentation [DATAROOTDIR/man]
--docdir=DIR documentation root [DATAROOTDIR/doc/mcl]
--htmldir=DIR html documentation [DOCDIR]
--dvidir=DIR dvi documentation [DOCDIR]
--pdfdir=DIR pdf documentation [DOCDIR]
--psdir=DIR ps documentation [DOCDIR]
Program names:
--program-prefix=PREFIX prepend PREFIX to installed program names
--program-suffix=SUFFIX append SUFFIX to installed program names
--program-transform-name=PROGRAM run sed PROGRAM on installed program names
System types:
--build=BUILD configure for building on BUILD [guessed]
--host=HOST cross-compile to build programs to run on HOST [BUILD]
Optional Features:
--disable-option-checking ignore unrecognized --enable/--with options
--disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no)
--enable-FEATURE[=ARG] include FEATURE [ARG=yes]
--enable-silent-rules less verbose build output (undo: "make V=1")
--disable-silent-rules verbose build output (undo: "make V=0")
--enable-dependency-tracking
do not reject slow dependency extractors
--disable-dependency-tracking
speeds up one-time build
--enable-shared[=PKGS] build shared libraries [default=yes]
--enable-static[=PKGS] build static libraries [default=yes]
--enable-fast-install[=PKGS]
optimize for fast installation [default=yes]
--disable-libtool-lock avoid locking (might break parallel builds)
--enable-debug Enable debugging mode (default: disabled)
--enable-stats Enable statistics (default: disabled)
--enable-trace Enable tracing (default: disabled)
--enable-opencl2 Enable OpenCL 2.x (default disabled)
--enable-appleocl Use Apple OpenCL (default enabled)
Optional Packages:
--with-PACKAGE[=ARG] use PACKAGE [ARG=yes]
--without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no)
--with-pic[=PKGS] try to use only PIC/non-PIC objects [default=use
both]
--with-aix-soname=aix|svr4|both
shared library versioning (aka "SONAME") variant to
provide on AIX, [default=aix].
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-sysroot[=DIR] Search for dependent libraries within DIR (or the
compiler's sysroot if not specified).
Some influential environment variables:
CC C compiler command
CFLAGS C compiler flags
LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a
nonstandard directory <lib dir>
LIBS libraries to pass to the linker, e.g. -l<library>
CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
you have headers in a nonstandard directory <include dir>
LT_SYS_LIBRARY_PATH
User-defined run-time library search path.
Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.
Report bugs to <roberto.gioiosa@pnnl.gov>.
A common way to configure MCL is to run the following command:
./configure --prefix=<prefix_dir> CFLAGS=-O3 CPPFLAGS=-I<inc_dir> LDFLAGS=-L<lib_dir>
Other noteworthy options are:
- Debugging mode can be enabled by specifying
--enable-debug - Collecting and displaying statistics can be enabled by specifying
--enable-stats - Tracing can be enabled by specifying
--enable-trace(experimental)
Notes on OpenCL2:
- Not all vendor OpenCL implementations support OpenCL 2.x specifications. However, som
e functionalities in MCL do require an OpenCL 2.x-compatible library. The option
--enable-opencl2can be used to enable/disable OpenCL 2.x functionalities.
Notes on OSX:
- On OSX MCL can either use Apple OpenCL libraries (Default) or other OpenCL libraries.
We have successfully tested POCL (http://portablecl.org). The option
--enable-appleoclcan be used to enable/disable Apple OpenCL library. The default value isenabledon OSX anddisabledon other systems. - OpenCL2 functionalities are disable when Apple OpenCL library is used.
Run:
make
make install
This will install the scheduler mcl_sched in <prefix_dir>/bin, the MCL libraries in <prefix_dir>/lib, and MCL headers in <prefix_dir>/include. Options can be specifying when configuring to select different paths.
MCL comes with a unit test suite that can be executed with (MCL scheduler needs to be running when executing the tests):
./src/sched/mcl_sched &
make check
killall mcl_sched
The test directory also contains OpenCL version of the tests for reference and performance comparison. These versions are built by make check but not executed during testing.
We offer two Rust crates providing bindings for MCL, the source for both crates is hosted in the rust folder of this repository. Both crates are also available on crates.io
- libmcl-sys -- (https://crates.io/crates/libmcl-sys): high-level bindings through an "unsafe" interface
- mcl-rs -- (https://crates.io/crates/mcl-rs): high-level bindings providing a "safe" interface
- mcl-sched -- (https://crates.io/crates/mcl-sched): convenience wrapper for installing the mcl scheduler via cargo
As mentioned above, POCL is an open source implementation of OpenCL for CPUs and some GPUs. Any POCL version which supports OpenCL 1/2 should be inter-operable with MCL. However, we also use the open source nature of the POCL library to support extensions to the OpenCL interface which can be used by MCL. To get these extensions we provide a patch file for a specific POCL release. From the location of the mcl directory (so POCL will be installed as a sibling directory), run:
./mcl/scripts/get-pocl-shared-mem.sh
This will get the source code for the POCL extensions. Then configure and build POCL using CMake in accordance with the POCL documentation. To use the extensions within MCL we must tell the library in the configure script:
./configure --enable-shared-memory --enable-pocl-extensions
MCL is a research prototype and still under development, thus not all intended features are yet implemented.
Please, contact Roberto Gioiosa at PNNL (roberto.gioiosa@pnnl.gov) if you have any question.
Roberto Gioiosa
Rizwan Ashraf
Ryan Friese
Lunzheng (Lenny) Guo
Alok Kamatar
Polykarpos Thomakidis
Burcu Mutlu
Giulio Picierro
This project is licensed under the BSD License - see the LICENSE file for details.
IF you wish to cite MCL, please, use the following reference:
- Roberto Gioiosa, Burcu O. Mutlu, Seyong Lee, Jeffrey S. Vetter, Giulio Picierro, and Marco Cesati. 2020. The Minos Computing Library: efficient parallel programming for extremely heterogeneous systems. In Proceedings of the 13th Annual Workshop on General Purpose Processing using Graphics Processing Unit (GPGPU '20). Association for Computing Machinery, New York, NY, USA, 1–10. DOI:https://doi.org/10.1145/3366428.3380770
Other work that leverage or describe additional MCL features:
- A. V. Kamatar, R. D. Friese and R. Gioiosa, "Locality-Aware Scheduling for Scalable Heterogeneous Environments," 2020 IEEE/ACM International Workshop on Runtime and Operating Systems for Supercomputers (ROSS), 2020, pp. 50-58, doi:10.1109/ROSS51935.2020.00011.
- Rizwan Ashraf and Roberto Gioiosa, "Exploring the Use of Novel Spatial Accelerators in Scientific Applications" 2022 ACM/SPEC International Conference on Performance Engineering (ICPE), 2022.