Skip to content

UCX Loadable Modules

Yossi Itigin edited this page Feb 9, 2019 · 2 revisions

Some parts of the code, especially those with external lib dependencies (e.g cuda, verbs, xpmem..) are built as separate libraries which are dl-opened at runtime. These are the main points of how it's done:

General

  • UCX starting v1.6 is using loadable modules
  • Modules are installed to <libdir>/ucx/
  • Module code should depend only on "parent" components and not on "sibling" / "child" modules

Autoconf/Automake

  • The source directory for the module contains Makefile.am and configure.m4
    • The parent directories (up all way to top_srcdir) must also contain Makefile.am and configure.m4 files
    • Makefile.am is added from parent dir by SUBDIRS = ...
    • configure.m4 is added from parent dir by m4_include([.../configure.m4]). note: must specify full path here
  • configure.m4:
    • AC_ARG_WITH/AC_ARG_ENABLE
    • Autoconf checks for module dependencies (e.g pkg-config)
    • Setting xx_LDFLAGS/xx_CPPFLAGS/xx_LIBS
    • Adding the module to the list of parent component, so it would be loaded. e.g: uct_modules+=":xpmem"
    • AC_CONFIG_FILES([path/to/Makefile]) - to process Makefile.am
  • Makefile.am:
    • There should be a top-level if HAVE_xx / endif wrapping the whole Makefile.am
    • module_LTLIBRARIES += ... - to declare module object as installed to moduledir
    • include $(top_srcdir)/config/module.am - for custom build rules which enable using the module from build directory by creating local symlinks
    • Add -version-info $(SOVERSION) to xx_LDFLAGS

Packaging (RPM)

  • Spec file:
    • Module are built as RPM sub-package (eg ucx-xpmem), so they should have own description, files, etc.
    • Modules are build using RPM conditional builds
    • Spec file should have bcond_with/bcond_without declaration for each module
      • If the dependencies are satisfied in Fedora - it should be ON by default - bcond_without
      • If the dependencies are NOT satisfied in Fedora - it should be OFF by default - bcond_with
    • Build requirements should be added inside a condition the the module is built
    • All modules should be enabled/disabled explicitly for ./configure, to not be dependent on auto-detection, but on rpmbuild configuration only.
  • Build process:
    • Source RPM - no with/without arguments are used.
    • Binary RPM in Fedora - the default with/without settings are used - since it's built as-is from src RPM.
      The following sub-packages would be built: (TBD)
    • Binary RPM for MLNX_OFED - need to use the following arguments: --with knem
      The following sub-packages would be built: (TBD)
    • The contrib/buildrpm.sh script, when run in "binary rpm build" mode, constructs rpmbuild command line according to what the local "./configure" script auto-detected. This is done by replacing the list of build modules in contrib/rpmdef.sh.in, and processing the resulting file in the script. The reason is that currently this step is done as part of regression testing (jenkins) only.
    • A module should be added explicitly to contrib/buildrpm.sh to add it as needed to rpmbuild command

Checklist for new module

  • configure.m4
    • AC_ARG_WITH
    • Add to xx_modules
  • Makefile.am
    • define module_LTLIBRARIES
    • so version
    • include $(top_srcdir)/config/module.am
  • ucx.spec.in
    • bcond_with/bcond_without
    • conditional BuildRequires
    • add --with_xx/--without_xx to ./configure command
    • sub-package for module
  • contrib/buildrpm.sh
    • pass --with/--without to rpmbuild
Clone this wiki locally