howtos install RHEL6 interlagos support

Warning: This is a huge hack. If you've never built your own kernel, you should probably avoid trying this

The latest AMD Interlagos CPU requires a fairly recent kernel to be properly supported. You can install Rocks 5.4.3 out of the box on an Interlagos based system, but the RHEL/Centos 5.x kernel (even the latest 2.6.18-274.7.1.el5) seems to be lacking the FMA4 instruction sets which is a big draw for using Interlagos. Non-FMA4 code works fine on the system, but you are giving up the ability to use the double wide FPU as it was intended to be used. The procedure outlined below will help you get a RHEL 6 based kernel up and running on your system.

Table of Contents Prerequisites Packages for building a kernel Download and install the kernel source Build a kernel

Prerequisites

You should start with a Rocks 5.4.3 base installation. Something like this:

# rocks list roll
NAME                                 VERSION ARCH   ENABLED
base:                                5.4.3   x86_64 yes
ganglia:                             5.4.3   x86_64 yes
hpc:                                 5.4.3   x86_64 yes
java:                                5.4.3   x86_64 yes
kernel:                              5.4.3   x86_64 yes
perl:                                5.4.3   x86_64 yes
python:                              5.4.3   x86_64 yes
web-server:                          5.4.3   x86_64 yes
Red_Hat_Enterprise_Linux_Client_5.6: 5.4     x86_64 yes

I've got a RHEL 5.6 base that has been patched up to RHEL 5.7. YMMV, but the plain base of Rocks 5.4.3 should suffice

Packages for building a kernel

You should probably do this work on a compute node of your cluster, you need at least the following. This could take some time as there are X11 requirements that you probably don't have on your compute nodes. You have to make sure rocks-grub is not running otherwise updates you do to grub won't be honored. My compute hostname for testing is fat0001

ssh fat0001
chkconfig rocks-grub off
service rocks-grub stop
yum install ncurses-devel elfutils-libelf-devel patchutils binutils-devel newt-devel xmlto

Download and install the kernel source

I chose the latest Scientific Linux kernel sources. I'm assuming you NAT through the head node and can grab this file from a compute node. If not, download the RPM to the head node and copy it out to the compute node.

cd /tmp
wget http://ftp.scientificlinux.org/linux/scientific/6.1/SRPMS/vendor/kernel-2.6.32-131.17.1.el6.src.rpm
rpm -ivh --nomd5 kernel-2.6.32-131.17.1.el6.src.rpm

Now you have the latest kernel source installed. Let's first edit the spec file to remove dependencies that are not really needed and can't be easily satisfied on RHEL 5. We're going to hack this into place anyway.

Build a kernel

cd /usr/src/redhat/SPECS
vi kernel.spec

Go to line 513 and remove the version dependency on rpm-build. The resulting line should be

BuildRequires: net-tools, patchutils, rpm-build

Also comment out the next line requiring xmlto and asciidoc

#BuildRequires: xmlto, asciidoc

Prepare the source to be built. By default, Redhat is requiring that you sign kernel modules. You'll want to start up a random number generator to get some entropy. Do that and then prepare the source with an rpmbuild command.

rngd -r /dev/urandom
rpmbuild -bp kernel.spec

You now have a kernel source area where you can configure and build a kernel. This is the scary part. You might want to skip the menuconfig step or at least start with my working config file.

cd /usr/src/redhat/BUILD/kernel-2.6.32-131.17.1.el6/linux-2.6.32-131.17.1.el6.x86_64
make menuconfig
make -j 12
make modules_install
make install

Edit the /etc/grub.conf file and make this kernel the new default kernel. You should also remove the "hidemenu" option so you can get into the system if you just bricked it trying this kernel. My grub.conf then looks like

default=0
timeout=5
title Red Hat Enterprise Linux Client (2.6.32)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32 ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.32.img
title Rocks (2.6.18-238.12.1.el5)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.18-238.12.1.el5 ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.18-238.12.1.el5.img

Reboot and hope for the best. You could also do the above and create a kernel RPM.

cd /usr/src/redhat/BUILD/kernel-2.6.32-131.17.1.el6/linux-2.6.32-131.17.1.el6.x86_64
make menuconfig
make rpm
rpm -ivh /usr/src/redhat/RPMS/x86_64/kernel-2.6.32-1.x86_64.rpm
mkinitrd /boot/initrd-2.6.32.img 2.6.32
/sbin/grubby --copy-default --make-default --initrd=/boot/initrd-2.6.32.img --add-kernel=/boot/vmlinuz-2.6.32 --title=2.6.32
reboot

[root@fat0001 ~]# uname -a
Linux fat0001.local 2.6.32 #3 SMP Mon Oct 24 11:37:58 PDT 2011 x86_64 x86_64 x86_64 GNU/Linux