Skip to content

Linux NTB HOWTO build and install

Jump to bottom Edit New page
fancer edited this page Nov 27, 2017 · 1 revision

Prerequisites

You should make sure you have the following installed on your machine:

  • curses - needed for menuconfig step
  • make - needed for build
  • gcc - needed for build

On Debian or any Debian-based distribution such as Linux Mint and Ubuntu, run as root (or using sudo(8)):

# apt-get install make gcc libncurses4-dev

On Red Hat or any Red Hat-based distribution such as Fedora and CentOS, run as root (or using sudo(8)):

# yum install make gcc curses-devel

Build Instructions

While it is possible to build a kernel on a machine other than the installation target, the standard method is to build the kernel directly on the machine where you will be installing it. The following instructions follow the standard method. Therefore, run these steps on the machine to be installed.

  1. Create and/or move to the directory where you wish to clone the NTB repository.

  2. Clone the desired NTB git tree

    Most users will clone the ntb branch.

     $ git clone https://github.com/jonmason/ntb.git
 $ cd ntb
 $ git checkout -b ntb origin/ntb

    If you would like to work with the ntb-next branch, you can do the following.

     $ git clone https://github.com/jonmason/ntb.git
 $ cd ntb
 $ git checkout ntb-next

    TIP
    After you have cloned the repository, you may want to run $ git status and verify that your clone is clean. It should look something like this:

     $ git status
 On branch ntb
 Your branch is up-to-date with 'origin/ntb'.
 nothing to commit, working directory clean

    If status reports a number of modified files in the working directory, then you may be using a case-insensitive file system (often seen on Mac OS X). The Linux repository requires a case-sensitive file system.

  3. Copy the system's current configuration to .config.

    All existing configurations reside in /boot. You can see what is present like this:

    $ ls /boot/config*

    To use the configuration associated with the currently booted kernel (this is typical), do this:

    $ cp /boot/config-$(uname -r)  .config

    Or explicitly, something like this:

    $ cp /boot/config-4.13.2-1.el7.elrepo.x86_64 .config

    As an alternative, if you know there's a particular configuration that you want, copy that one to .config.

  4. Run menuconfig so the kernel build can be configured.

    The following will start a curses-based configuration utility int he window where you run it. (Note that in the Prerequistes section above that you were to ensure that curses was installed.)

    $ make menuconfig

  5. Navigate menuconfig set the NTB configuration

    Note that the following are only the steps required for NTB functionality. If you have configuration requirements in addition to these, they should also be put into place at this time.

    For those not familiar with menuconfig, there's a diversity of online help. Essential instructions are at the top of the window.

    • The up/down arrows will navigate the menu hierarchy.
    • The left/right arrows will navigate the four options at the bottom of the screen.
    • By default, the "Select" option is chosen at the bottom. That means that the Enter key will select the menu option which is highlighted.
    • The "Exit" option will move you up a level, or exit menuconfig if you are at the top of the hierarchy. Menuconfig will ask if you want to save the configuration exit, if you have not.
    • The "Save" option will save to the file you want at any point.

    Now, on with the configuration.

    1. Down arrow to move to "Device Drivers --->". Then press Enter to select.

    2. Down arrow to "Non-Transparent Bridge Support --->". Press 'M' to modularize it.

      Note: The character between the square brackets [ ] now shows an 'M'.

    3. Now press Enter to select Non-Transparent Bridge Support

    4. Using the up/down arrow keys, press 'M' for all desired options.

      NOTE: If you wish to use ntb_netdev (and perform the network test at the bottom of this wiki, then you must include the NTB Transport Client.

      For example, one might press 'M' to modularize the following if one desired Intel NTB support and the ntb_transport module, but no debugging modules:

      • Intel Non-Transparent Bridge Support
      • NTB Transport Client

      To exclude all other modules, press 'N' for each line.

      As another example, if you are using IDT/Switchtec hardware and desire all of the debug modules, then press 'M' for the following:

      • IDT PCIe-switch Non-Transparent Bridge support
      • Microsemi Switchtec Non-Transparent Bridge Support
      • NTB Ping Pong Test Client
      • NTB Debugging Tool Test Client
      • NTB RAW Perf Measuring Tool
      • NTB Transport Client

      And press 'N' to exclude all other modules.

    5. Exit back to Device Drivers. (Press the right arrow to move to the "Exit" menu option, then press ENTER.)

    6. Move to "Network device support --->". Press ENTER to select.

    7. Move to "Virtual Ethernet over NTB Transport". (It appears as a sub-selection below "Network core driver support".) Press 'M' to modularize it.

    8. Exit back to "Device Drivers".

    9. Exit back to the root menu.

    10. Move to "Processor type and features". Press ENTER to select.

    11. Move to "Contiguous Memory Allocator". Press SPACE to include this built-in module.

    12. Exit back to the root menu. Then re-enter "Device Drivers --->".

    13. Move to "Generic Driver Options". Press ENTER to select.

    14. Move to "DMA Contiguous Memory Allocator (NEW)". Press SPACE to include.

      When you include this module, two options show up.

    15. Move to "Size in Mega Bytes". Press ENTER to get pop-up.

      This is the memory BAR size. For example, 4 MiB.

    16. You should be able to ignore "Select region size" as this should already be in megabytes.

    17. Move to "Maximum PAGE_SIZE order of alignment for contiguous buffers". Press ENTER to get pop-up.

      This value expresses the alignment of the region. This is expressed as a power-of-two multiple of 4 KiB pages. For example, the default value is 8, which results in 1 MiB alignment (i.e 4 KiB * 2^8). You could choose alignment which matches the region size (e.g. 10 matches 4 MiB). If in doubt, leave this value at 8.

    18. Save the configuration (as .config). Exit menuconfig.

  6. Personalize the build (optional).

    If you would like to be able to uniquely identify this build, either or both of the following two methods can be used. This is entirely optional.

    1. CONFIG_LOCALVERSION

      This method will result in placing the desired string in the information presented by 'uname -r' output.

      Edit .config, and place the desired string in the line:

      CONFIG_LOCALVERSION=""

    2. KBUILD_BUILD_USER environment variable

      This method will result in placing additional information in the boot banner string, which appears both in the dmesg boot output and in the ASCII file /proc/version.

      For this method, export the KBUILD_BUILD_USER environment variable using whatever method your shell supports. For example, when using bash, you would do this:

      $ export KBUILD_BUILD_USER="dmeyer-custombuild"

  7. Determine the number of threads in the machine

    You will specify the number of threads in the processor using the -j# option. At least for Intel machines, this number can be determined by looking at how many "processors" there are in the file /proc/cpuinfo. Thus, you can run the following.

    $ cat /proc/cpuinfo | grep processor

    Though these are called processors, they are threads. For example, for a particular i7 CPU, it has eight processors (threads). And so the compilation is run like this.

  8. Compile the kernel.

    IMPORTANT: You must compile the kernel as a normal user. Not as root.

    For the example number of threads given in the previous step:

    $ make -j8

    TIP
    For some make targets, if you run make and get errors that look like the following:

     ./scripts/gcc-version.sh: line 11: $'\r': command not found
 ./scripts/gcc-version.sh: line 33: syntax error: unexpected end of file
 /bin/sh: 1: [: -lt: unexpected operator
 ./scripts/gcc-goto.sh: line 4: $'\r': command not found

    This behavior is likely caused by the clone's files being created with incorrect line termination for Linux. There are many sources of information on this, and how to configure git to use correct line termination. For example, this github line ending article can get you started.

  9. Install.

    IMPORTANT: Pay special attention to the steps performed as root.

     $ make modules_prepare
 $ sudo make modules_install
 $ sudo make headers_install
 $ sudo make install

    If you want, one telltale of the installation is the following:

    $ ls -l /lib/modules

  10. Run grub_customizer (optional).

    This step may be unnecessary, depending on the current grub configuration. If grub is configured to boot the last kernel booted, then simply rebooting will not result in running the newly installed kernel. However, grub is often configured to run the most recent installation (which appears as the top-most line in grub-customizer).

    If you wish, make sure grub-customizer is installed, and run:

    $ sudo grub-customizer

    Make sure your new build is at the top, that grub will boot the top selection, and SAVE. Then exit grub-customizer.

  11. Reboot to boot to the new kernel.

NTB Module Verification and ntb_netdev Test

After rebooting (and assuming that you included the NTB Transport Client in the build, you can run the following steps:

  1. Verify the NTB Module is Loaded.

    The ntb module should be probed when the hardware is discovered by the kernel

    lsmod | grep ntb

  2. Load the ntb_netdev Driver.

    To make use of NTB, a "client" driver will need to be used. Currently, only a virtual Ethernet client driver exists. Probe this driver.

    modprobe ntb_netdev

  3. Configure the Network Interface.

    Configure the interface as you would a normal network device

    ifconfig eth4 192.168.1.100 up

  4. Repeat on the Peer System

    Run the same steps above on the other system sharing the backplane

  5. You should be able to ping and run normal network traffic between the systems

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 United States License.

Add a custom sidebar
Clone this wiki locally