-
Notifications
You must be signed in to change notification settings - Fork 4
Setup for the WUSTL HPC Cluster
This document is a work in progress - if it fails you, but you go on to succeed, please take a moment to come back and make an addition that might help the next person.
Before moving forward, you should:
- have an account set up on the WUSTL HPC cluster
- have the cluster admin to set up a /BlueArc-scratch directory for you.
- be somewhat comfortable using the bash shell.
- know how to use vim, emacs, or some other text editor.
- know the basics of using git from the command line. GitHub is here to help!
- have access to this repository.
- log onto one of the cluster's login nodes
1. Cluster SSH key -> GitHub
Now that you're logged in to the cluster, we're going to need get git set up in order to get the code. Let's do it right and use an ssh key. You'll already have one of these, so let's just print and copy the public key:
$> cat ~/.ssh/id_rsa.pub
# lots of text will appear here. Copy it to your clipboard.Come back to github, go into your account settings, and add the ssh key to your account.
2. Put git on your path
You're going to be making several modifications to your ~/.bashrc file. I use vim. You can use whatever you want.
$> vim ~/.bashrc
# the file will open. google if you need vim help.
# add the following:
# We'll use this enthought python distro...
EPD_PYTHON_HOME=/export/epd-7.0.2
# Put git on the path
export PATH=$PATH:/export/git-1.7.4.1/binIn the future, we'll make more additions, but I won't always include the vim command. I'll just say, "add this to .bashrc," and leave it to you how you get that done.
3. Clone hcpre
NOTE: Instead of cloning the repo, you can also install using pip (see the main README for more info). If you want to go this route, skip to step 4b, and install hcpre using pip after step 5 (activate your virtual environment, then call pip install hcpre per the README instructions). Otherwise, read on:
I like to keep all of my code in a subdirectory of my home folder. You can keep it where you like. So where I refer to /home/aclandb/devdir, or ~/devdir, you should substitute whatever folder it is in which you choose to do the following. Your home directory would be fine, but I highly recommend a subdirectory of your home directory, used to hold code in general. Anyhow, here we go:
$> cd ~/devdir
$> git clone git@github.com:beOn/hcpre.git
# stuff will happen here. it's all good.4a. Put the module's dir on your path
Add all of the following snippets to ~.bashrc, remembering to look out for places where you need to change the paths.
# Put the hcp pipeline stuff on the path
export PATH=$PATH:~/devdir/hcpre/hcpre
# Python needs to know where the hcp stuff is... and my other nipype stuff too...
export PYTHONPATH=/home/aclandb/devdir/hcpre4b. Other env setup
Just a few more additions to ~/.bashrc...
# Point to your personal scratch space, or another. Make sure this dir exists.
export TMPDIR=/BlueArc-scratch/aclandb/tmp
# Freesurfer, FSL setup
. /export/freesurfer-5.3/SetUpFreeSurfer.sh > /dev/null
export FSLDIR=/export/fsl-5.0.5
. /export/fsl-5.0.5/etc/fslconf/fsl.sh >/dev/null
# Put MRIcron on your path, so we can use dcm2nii
export PATH=$PATH:/export/MRIcronHere you should take a moment to consider the decisions we have just made. Importantly, we have chosen versions of FreeSurfer and FSL. It's up to you to ensure that you're using the right version. These are the ones that I have been using as of spring 2014. You should take a look at the HCP Pipeline code's documentation to see which versions they expect you to use, and work with the cluster admin to make sure it's available.
If you haven't used virtualenv before, and you use python on shared systems to which you do not have administrator access, prepare to be liberated. It doesn't even matter if virtualenv isn't installed on the system yet. Since we're such rockstar ninjas, we're going to help ourselves.
If you need extra help with virtualenv, do please check out http://www.virtualenv.org/en/latest/virtualenv.html.
First, we'll need a directory that we'll keep our python related stuff in. For me, that's going to be ~/python_stuff. You might do the same, or something different, but if you use something different, remember to look out for paths you might need to change in the commands below!
$> mkdir ~/python_stuff
$> cd ~/python_stuff/Cool. Now we're going to download virtualenv, decompress it, and install it into our python directory (check https://pypi.python.org/pypi/virtualenv#downloads for the latest .tar.gz url).
$> curl -O "https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.2.tar.gz#md5=d3d915836c1ada1be731ccaa12412b98"
$> tar -xzvf ./virtualenv-1.11.2.tar.gz#md5\=d3d915836c1ada1be731ccaa12412b98
# the decompression/unarchiving dumps some text here...
$> cd virtualenv-1.11.2
$> /export/epd-7.0.2/bin/python setup.py install --prefix ~/python_stuffOh no, this throws an error!
That's okay. The error is helpful:
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/home/aclandb/python_stuff/lib/python2.7/site-packages/
Oh, that's not so bad! All we have to do is to modify our .bashrc file again, adding that path to PYTHONPATH. While we're at it, I'm also going to add another directory to PATH, and you'll see why in a second. Just remember to swap out my home directory with your own!
Add the following to .bashrc:
# Add virtualenv stuff to PATH and PYTONPATH
export PYTHONPATH=/home/aclandb/python_stuff/lib/python2.7/site-packages/:$PYTHONPATH
export PATH=/home/aclandb/python_stuff/bin:$PATHAlright, let's read those changes in from our startup script then try that install again (from within the unarchived virtualenv directory):
$> source ~/.bashrc
$> /export/epd-7.0.2/bin/python setup.py install --prefix ~/python_stuff
# this time, lots of stuff happens, after which:
$> which virtualenv
~/python_stuff/bin/virtualenvAwesome. It may now be clear why we added /home/aclandb/python_stuff/bin:$PATH to PATH.
Now let's change back to our python directory, clean up, and set up our HCP-ready python environment.
$> cd ~/python_stuff/
$> rm -rf virtualenv-1.11.2*
$> ls
bin lib
$> virtualenv hcp_mods --python /export/epd-7.0.2/bin/python --system-site-packages
New python executable in hcp_mods/bin/python
Installing setuptools, pip...done.To fully understand what we have done here, go check out the virtualenv documentation. For now, here are the highlights. We have set up an semi-isolated python environment to which you have write access (since it's in your home directory). This means that you will now be able to install python modules for yourself, instead of having to bug the systems admin or mess up anyone else's installations. Because we used --system-site-packages, we'll also have access to the modules installed on the version of python that we based our environment on, namely /export/epd-7.0.2/bin/python.
In practical terms, this means the following:
$> which python
/usr/bin/python
$> source ~/python_stuff/hcp_mods/bin/activate
(hcp_mods)$> which python
~/python_stuff/hcp_mods/bin/python
(hcp_mods)$> deactivate
$> which python
/usr/bin/pythonWhoa! Did you see that? When I called that activate script, it changed which python executable I was using. When I called deactivate, it changed back. You'll also notice that my prompt changed to include the name of the environment. That's helpful, because it lets you know if you're in an environment when you don't want to be, or the opposite.
So let's go ahead and make one more modification to .bashrc, so that we'll automatically wind up in the hcp_mods environment whenever we log in:
# Activate virtualenv
. ~/python_stuff/hcp_mods/bin/activateOkay, now we finally get down to the reason we did all of this: to easily install python dependencies. At this point, the only thing that has been giving me trouble is gradient_unwarp. As of Feb 2014, the publicly released version is not compatible with the HCP code. A compatible version comes bundled in the Pipelines directory, under Pipelines/src/gradient_unwarping/. To install it, make sure your hcp_mods environment is active, download the HCP Pipelines, and do the following (substituting your location for the Pipelines dir wherever you see /x/Pipelines below):
(hcp_mods)$> cd /x/Pipelines/src/gradient_unwarping
(hcp_mods)$> python setup.py installNotice that things are a little easier than when we ran virtualenv's setup.py. Specifically, we didn't have to specify an installation path using --prefix. That's because, roughly speaking, setup.py files install their payload into directories used by the version of python used to call setup.py. In this case, that's our hcp_mods version, so the payload gets installed into subdirectories of ~/python_stuff/hcp_mods/... automatically.
There's so much more to say about everything we just did, but that's enough for now. If you want a better understanding, the virtualenv documentation and the python documentation on installing modules would be good places to start.
6. Install the HCP Pipelines
TODO: this, once there's a public release.
For info on actually using the script, including .bashrc modifications, configuration options, and more, please see the hcpre readme.md file, visible on the repo's main github page.
The thing about the cluster is that you probably don't have a VNC session open on the login nodes. If you do, fine, but if you don't then you need some way of making sure that your jobs don't die when you log out of the login node. Enter "nohup."
(hcp_mods)$> nohup bash -c "hcpipe.py -c hcp.conf --pbs -r -o /BlueArc/aclandb/MOOD_RISK/preprocessed" >po.txt 2>&1 &Let's take this apart bit by bit. For starters, I'm executing this from within some directory specific to the analysis. Inside a BlueArc-scratch subdirectory might be a great place.
First let's look at the code surrounding our call to hcpipe.py:
$> nohup bash -c "" >po.txt 2>&1 &nohup prevents jobs you start during a terminal session from accepting the HUP signal, which would usually cause them to terminate. We're using it to call bash, and we're handing bash the -c argument, which causes it to execute the string that comes next. We'll get to the contents of that string in a second, but spoiler alert: it's our call to hcpipe.py.
>po.txt sends the standard output stream of the bash command to the file po.txt in my current directory. You could point to any file that you want, but take note that if there is already a file there, it will be overwritten.
2>&1 sends the standard error stream (2) to the same place as the standard output stream (1), in this case that's po.txt.
The trailing & puts the nohup call in the background.
If you want to follow what's happening in po.txt, you can check follow it with:
$> tail -f po.txtYou can get out of tail -f by hitting Ctrl+c.
If you need to kill the nohup job, you can find its job number with:
$> ps aux | grep your_user_name.*hcpipe.pyThen just pass that number to the kill command.
Let's take a quick look at the pipeline call I made inside the quotation marks:
hcpipe.py -c hcp.conf --pbs -r -o /BlueArc/aclandb/MOOD_RISK/preprocessedMost of this should be covered in the main hcpre readme. Do take special care to use the --pbs argument when running on the cluster. This tells hcpipe.py to dispatch its workflow nodes using PBS (Portable Batch System), which the cluster uses to schedule jobs on worker nodes. Otherwise, you might wind up running the pipeline on the login node, which would be bad cluster etiquette. If you accidentally do find yourself running something on the login node, don't panic - just kill the process.
The other important thing is that I used -o to tell hcpipe.py to put the finished product into the project's main bluearc folder. You can make your data land wherever you like. Not specifying -o will cause data to be dumped to the current directory.
For now just email me. We'll expand this as you run into things.
a note on worker node requests
writing your own script based on hcpipe.py