Home

What is this

This is the GNI provider fork of ofiwg/libfabric. For more information on libfabric and the OFI WG, see the upstream wiki.

The GNI provider

The GNI provider runs on Cray XC (TM) systems utilizing the user-space Generic Network Interface (uGNI) which provides low-level access to the Aries interconnect. The Aries interconnect is designed for low-latency one-sided messaging and also includes direct hardware support for common atomic operations and optimized collectives. See the fi_gni(7) man page and the provider feature matrix for more details.

Our process

All development on the GNI provider is done in this fork. Please report bugs and other issues with the GNI provider using this repository rather than the upstream.

This fork is kept up-to-date with the upstream. We push our commits upstream on about a weekly basis, and pull down upstream commits at least that frequently.

Testing the GNI provider

Currently, fabtests does not work out of the box on Cray XC systems due to the different job launching mechanism (aprun or srun). We are working on a solution for this problem, but in the meantime if you want to verify that your libfabric installation is working, there are a couple of options:

• fabtests-cray: We have forked the upstream fabtests repository and added a script called cray_runall.sh that runs as many of the existing fabtests as possible (given the launch capabilities) as well as a few other benchmarks we've added. It also includes cray_runall_expected_failures and cray_runall_intermittent_failures to indicate which tests are expected to fail.

• gnitest: We have a large number of unit tests in this repository used for development purposes. Running them requires the Criterion unit testing framework. See this wiki page for information on how to build and run the unit tests.

Other developer resources

• Getting started: Workflow and policies
• IRC channel: #ofi-cray (chat.freenode.net)