Skip to content

OS install RTC

Jump to bottom
Vincent Deo edited this page Nov 2, 2021 · 10 revisions

1. OS Install - Introduction

1.1. Overview

We recommend using a real-time (RT) Linux kernel for optimal performance, although any recent Linux OS will work well with cacao.

This page describes RT linux installation for Ubuntu distribution. The page is updated from time to time to keep up with recent kernel versions.

1.2. Key elements

The following elements will be installed on the system:

  • Ubuntu OS (recent distribution)
  • RT kernel (patched)
  • NVIDIA driver
  • CUDA library

Before proceeding, we will need to check intercompatibility between these elements. We have validated that the following versions work together (May 2020):

  • Ubuntu 20.04 LTS (based on Kernel 5.4.0-generic)
  • Kernel 5.6.10 (published 2020-05-02)
  • Corresponding PREEMPT_RT patch version 5.6.10-rt5 (published 2020-05-04)
  • NVIDIA driver 440.82
  • CUDA 10.2

And also (Oct 2021):

  • Ubuntu 20.04 LTS (based on Kernel 5.4.0-generic)
  • Kernel 5.10.73 (Oct 13 2021)
  • PREEMPT_RT 5.10.73-rt54 (Oct 16 2021)
  • NVIDIA driver 470.74
  • CUDA 11.4

Try to keep the kernel version in the distribution and the kernel you willl compile as close as possible.

2. Install stock OS

NOTE: Instruction for Ubuntu 20.04 server

Start with a standard installation of a recent Ubuntu distribution. The server distribution is preferred over desktop, as we will not be using a local desktop environment.

Instructions for a USB key install:

  • Pick install Ubuntu Server
  • Pick language English
  • Accept default keyboard choice
  • Chose Install Ubuntu
  • Select interface (shows DHCP connection - set your static LAN IP if necessary)
  • skip proxy
  • skip mirror
  • Pick Manual partitioning
  • Select drive on which to install
  • Add main partition as needed. For example, for a 1 TB drive, pick:
    • ~300GB for the main partition, mount point /, btrfs
    • ~500GB for home, mount point /home, btrfs
    • ~20GB swap
  • Done
  • Install OpenSSH
  • setup user
  • skip server snaps

Install, and reboot.

3. Real-time Kernel

3.1. Grub config

Before installing new kernel, enable grub screen.
Change GRUB_TIMEOUT to 10 in /etc/default/grub (sudo needed):

GRUB_TIMEOUT=10

Apply change:

sudo update-grub

Check current entries:

grep menuentry /boot/grub/grub.cfg

For real-time high performance add kernel boot options
Check the /etc/default/grub.d directory for config files that would override this.
Comment out the GRUB_CMDLINE_LINUX_DEFAULT in any such file.

Misc grub settings (feel free to adapt if familiar) in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable no_stf_barrier mds=off mitigations=off nosoftlockup intel_idle.max_cstate=0 processor.max_cstate=0 mce=ignore_ce idle=poll nmi_watchdog=0"

Note This is unsafe, and not recommended for computers on the wild internet. mitigations=off disables a number of safety patches (including meltdown and spectre) for the sake of performance.

Note This is a hard way of preventing power-saving mode (p and c-states), so the computer will use a lot more Watts, even when idling.

If you want hard shielding of some CPU cores, also add

"isolcpus=16-35 nohz=on nohz_full=16-35 rcu_nocbs=16-35 skewtick=on"

See these links:

3.2. Getting the sources

URLs here tend to changes as kernels are moved into "older" folders.
Get latest kernel (here 5.6.10), found on https://www.kernel.org/. Check the following URLs for the current/latest RT patch you want:

wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.XX.XX.tar.xz # Substitute XXs.

And the corresponding patch, found on https://wiki.linuxfoundation.org/realtime/start:

wget http://cdn.kernel.org/pub/linux/kernel/projects/rt/5.XX/patch-5.XX.XX-rtXX.patch.xz # Substitute XXs.

3.3. Patching the kernel

Unpack the archives and patch the Linux kernel:

xz -cd linux-5.6.10.tar.xz | tar xvf -
cd linux-5.6.10
xzcat ../patch-5.6.10-rt5.patch.xz | patch -p1

3.4. Kernel compilation

Pre-requisites to compile kernel:

sudo apt install make libelf-dev libncurses-dev flex bison libssl-dev wget build-essential linux-source fakeroot libncurses5-dev ccache

Before compiling the kernel, we need to configure which modules are to be included and which ones are to be left out.
There are many ways to go about doing this.

An easy and straightforward way to do this is to first copy your existing kernel config file and then use ‘menuconfig’ to make changes (if necessary). This is the fastest way to do it and probably, the safest.

cp /boot/config-$(uname -r) .config 
make menuconfig

Here are the used modifications:

  • General setup
    • Timers subsystem
      • Timer tick handling
        • Full dynticks system
      • Preemption model
        • Fully preemptible kernel
      • CPU/Task time and stats accounting
        • Fine granularity ... IRQ time accounting
  • Processor type and features
    • Timer frequency
      • 1 kHz

Save the .config file from the TUI, exit, and proceed to compile the patched kernel:

make -j $(nproc)
sudo make INSTALL_MOD_STRIP=1 modules_install -j $(nproc)
sudo make install

Note the INSTALL_MOD_STRIP is important - it shrinks the initial ramdisk size by ~90%. In some cases, when not used, the initial ramdisk would not load and the patched kernel would not boot.

Note2 if make modules_install crashes, comment the CONFIG_SYSTEM_TRUSTED_KEYS line from the .config.

Removing undesired hand-compiled kernels

Get an idea of what's installed by checking

ls /boot/

Locate files for a given kernel

locate -b -e 5.X.X-my-kernel

And remove locate -b -e 5.X.X-my-kernel | xargs -p sudo rm -r

These patterns can be manually checked, but locate usually works fine:

/boot/vmlinuz*KERNEL-VERSION*
/boot/initrd*KERNEL-VERSION*
/boot/System-map*KERNEL-VERSION*
/boot/config-*KERNEL-VERSION*
/lib/modules/*KERNEL-VERSION*/
/var/lib/initramfs/*KERNEL-VERSION*/

Don't forget to

sudo update-grub

Compile cpupower

cpupower is a very convenient tool for managing cpu performance, frequency profiles, C-states, etc, but does not ship with hand-compiled / RT patched kernels. But we can compile it manually.

sudo apt install libpci-dev
cd linux-5.6.10/tools/power/cpupower
make
sudo make install

3.5. Reboot

Deprecated. This update-initramfs and update-grub are actually performed by the kernel's make install. If unsure, it's still recommended to run an update-grub before rebooting, to check that at least something is there to boot on.

Once the kernel and its modules are compiled and installed, we want to be using the new kernel the next time we boot up. For this to happen, we need to use the following command:

sudo update-initramfs -c -k 5.6.10-rt5

Then, use the following command, which automatically looks for the kernels present in the /boot folder and adds them to the grub’s config file.

sudo update-grub

Now, restart the system and you should see that the new kernel is added to the boot loader entries.

compute and control for adaptive optics (cacao) - https://github.com/cacao-org/cacao

Home

Installation, System Setup and Performance

User Guide

cacao by examples

Programming with milk and cacao

Technical notes

Clone this wiki locally