Skip to content

JenkinsExecutorConfiguration

Jump to bottom
Ryan Curtin edited this page Apr 28, 2021 · 1 revision

Jenkins Executor Configuration

Basic toolchain

Basic packages required include

  • gcc
  • g++
  • make
  • cmake
  • sloccount
  • libxml2 (and headers)
  • blas / lapack / atlas
  • openjdk jre
  • hdf5 (and headers)
  • armadillo (and headers)
  • arpack (and headers)

Packages required for the benchmarking scripts include

  • python
  • python-yaml
  • valgrind
  • matlab (must be installed by hand using GT software licenses into /opt/matlab/)

Packages required for other libraries by the benchmarking scripts include

shogun
  • pkg-config
  • octave
  • octave-config (liboctave-dev on debian)
  • R (r-base-core on debian)
  • python-numpy
  • eigen3
  • lua (lua5.2 on debian)
  • lua-dev (liblua5.2-dev on debian)
  • ruby
  • ruby-dev
  • java jdk
  • swig
  • ant

Armadillo

We use a matrix build for mlpack which varies the Armadillo version. Each version is kept in /opt/armadillo/${armadillo_version}/, and /opt/armadillo/armadillo-svn/ is a symlink to the latest development branch of armadillo (arma/branch-3.6/, for instance). Instead of downloading and unpacking each version, it may be easier to simply copy it off of an existing slave:

root@newslave# scp -r adam.cc.gt.atl.ga.us:/opt/armadillo /opt/
root@newslave# chown -R jenkins:jenkins /opt/armadillo
root@newslave# su - jenkins
jenkins@newslave$ crontab -e

And in the crontab you'll need to put this line:

0 5 * * * /opt/armadillo/update-svn.sh

You may need to add the group jenkins (and add the user jenkins to that group). The update-svn.sh script updates Armadillo from svn and potentially the symlink (if necessary).

gcc

Using different versions of gcc is a nightmare. Every other mlpack dependency must be linked specifically against that compiler's version of libstdc++.so (and other system libraries). This means that for each version of gcc, we must build each library dependency against it. Most notably this includes Boost, which unfortunately takes a long time to build. Armadillo is exempt from this, though, because it is a header-only library. Also because we simply build Armadillo at Jenkins build-time (it only takes a second, so it's a pre-build step).

Boost

Windows Executor Configuration

Windows executor configuration is another matter entirely, but here is what needs to be done to hook it into Jenkins.

  • If it has been configured to work with Jenkins in the past, remove all of that. Delete C:\Jenkins, and remove the service "Jenkins Slave" with the command prompt (as Administrator): sc delete name_of_jenkins_service. You can find the name of the jenkins service by inspecting the service in the Services dialog.

  • Configure the executor in Jenkins to start with "Java Web Start". Download the starter using rdesktop to the Windows slave. Once it connects, do File->Install as Service.

  • Be sure to open ports 135, 139, and 445 to the Jenkins master.

  • If Jenkins isn't run as a service on the Windows box, builds may fail with very weird Java errors, so it must be run as a service.

Software installation

The Jenkins Windows job is set up in such a way that all the libraries mlpack needs to link against must be in C:\lib. See the actual Jenkins job configuration for the list of dependencies that is necessary and the directories they should be in. Each time a library is updated, it will need be updated on all of the Windows machines and the Jenkins Windows job configuration. It's a bit of a hackish solution, but it gets the job done...

LDAP/Kerberos setup

We have as many slaves as Thomas Jefferson, but we don't share his changing and somewhat unknown views about the ethics of the institution. As such it's somewhat unwieldy to have to set up user accounts on every single different system and maintain entirely separate accounts. LDAP and Kerberos help solve this problem by centralizing authentication to the master (big.cc.gt.atl.ga.us). Unfortunately I don't remember exactly what I did to set up the LDAP server and Kerberos server on big, so I can't document that anymore, but there are a multitude of online tutorials (I mostly followed this one and this one).

Debian

To make a Debian box use LDAP/Kerberos is fairly straightforward. This follows this guide and this guide. First, install the necessary Kerberos packages so PAM can use Kerberos as an authentication method:

# apt-get install krb5-user libpam-krb5

The parameters for the dpkg configuration are:

  • Default Kerberos 5 realm: CC.GT.ATL.GA.US
  • Kerberos servers for your realm: big.cc.gt.atl.ga.us
  • Administrative server for your Kerberos realm: big.cc.gt.atl.ga.us

Now we have to add the host principal:

# kadmin -q "addprinc -randkey host/HOSTNAME.cc.gt.atl.ga.us"
# kadmin -q "ktadd host/HOSTNAME.cc.gt.atl.ga.us"

Now, the Kerberos server (big) should know about the existence of HOSTNAME. After that the Kerberos client is set up, we can set up the LDAP client and then integrate it all with PAM.

We need a few packages:

# apt-get install ldap-utils libpam-ldap nscd libnss-ldap

There are some dpkg configuration options:

  • LDAP server Uniform Resource Identifier: ldap://big.cc.gt.atl.ga.us/
  • Distinguished name of the search base: dc=cc,dc=gt,dc=atl,dc=ga,dc=us
  • LDAP version to use: 3
  • LDAP account for root: cn=root,dc=cc,dc=gt,dc=atl,dc=ga,dc=us
  • LDAP root account password:

The nslcd configuration options will be the same. Reconfigure libpam-runtime (dpkg-reconfigure libpam-runtime) and disable LDAP authentication.

Set the defaults for LDAP in /etc/ldap/ldap.conf:

BASE dc=cc,dc=gt,dc=atl,dc=ga,dc=us
URI ldap://big.cc.gt.atl.ga.us

Configure PAM to automatically create a home directory by modifying /etc/pam.d/common-session:

session required pam_mkhomedir.so

Lastly, configure /etc/nsswitch.conf to allow login via LDAP:

passwd:         compat ldap
group:          compat ldap
shadow:         compat ldap

Restart nscd, install zsh because logins will fail if the preferred shell doesn't exist (and Marcus uses zsh), and you should be done. If you have made local user accounts, you can delete them (though you don't need to delete the home directory; just chown -R username:user, and it should now be owned by the LDAP user).

Fedora / RHEL

Getting LDAP/Kerberos working on Fedora is a bit different because Fedora prefers that the whole thing be done through sssd (whose documentation is poor). Basically, the krb5 and openldap support packages need to be installed and configured by hand; then, authconfig --enablesssd --enablesssdauth --enablemkhomedir --update can be used to rebuild the PAM configuration files, and it should work. Of course, it won't actually be that simple...

Debugging

Kerberos will refuse to allow logins if clock skew is too great. This is probably a good thing to check. The ntpd package can be used to keep clocks synchronized.

NFS setup

With so many build slaves, it doesn't make sense to replicate /opt/armadillo or any other libraries across every single system. As a result, we can use a centralized NFS mount for each system. This way, when we add a new Armadillo version, we only need to do it in one place.

Server setup

We use NFS4 + Kerberos. After installing nfs-common and nfs-kernel-server (these package names are specific to Debian), we must modify /etc/default/nfs-common and /etc/default/nfs-kernel-server so that

  • NEED_IDMAPD="yes" (nfs-common)
  • NEED_GSSD="yes" (nfs-common)
  • NEED_SVCGSSD="yes" (nfs-kernel-server)
  • RPCMOUNTDOPTS="--manage-gids" (nfs-kernel-server)

Next, we have to add the NFS server principal to Kerberos (this assumes the host/ principal is already there for this host, which should have happened when you set up LDAP/Kerberos):

# kadmin -q "addprinc -randkey nfs/HOST.cc.gt.atl.ga.us"
# kadmin -q "ktadd nfs/HOST.cc.gt.atl.ga.us"

So now, the nfs/HOST.cc.gt.atl.ga.us principal should be in the NFS host's keytab. The last thing to do is set up the actual export, which is pretty simple. Add it to /etc/exports, force krb5 security (sec=krb5:krb5i:krb5p), and restart nfs-common and nfs-kernel-server.

Client setup

The client requires the nfs-common package, with NEED_IDMAPD="yes" and NEED_GSSD="yes". Add a principal to Kerberos for this system too:

# kadmin -q "addprinc -randkey nfs/HOST.cc.gt.atl.ga.us"
# kadmin -q "ktadd nfs/HOST.cc.gt.atl.ga.us"

Once this is set up, you can test a mount with

mount -t nfs4 -o sec=krb5,rw HOST.cc.gt.atl.ga.us:/dir /mount/point/

Then you can add it to /etc/fstab so it mounts on boot:

HOST.cc.gt.atl.ga.us:/dir   /mount/point   nfs4   sec=krb5,rw   0   0

Reboot and test. Hopefully it works.

Clone this wiki locally