Setup python (including mpi4py) without admin rights

giadarol edited this page Aug 11, 2018 · 16 revisions

The following guide shows how to get a complete python installation without administrator rights:

Step 1: We move to the folder where we want to stare our installation:

/afs/cern.ch/work/g/giadarol/sim_workspace_mpi

Step 2: We download, compile and install python:

wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tar.xz
mkdir python_src
mv Python-2.7.11.tar.xz python_src/
cd python_src
tar xvf Python-2.7.11.tar.xz
cd Python-2.7.11
mkdir /afs/cern.ch/work/g/giadarol/sim_workspace_mpi/python27
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/python27
make
make install

Step 3: We install virtualenv

 cd ..
 cd python_src
 curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-15.0.0.tar.gz
 tar -zxvf virtualenv-15.0.0.tar.gz
 cd virtualenv-15.0.0
 /afs/cern.ch/work/g/giadarol/sim_workspace_mpi/python27/bin/python setup.py install
 cd ..

Step 4: We create our virtual environment

cd /afs/cern.ch/work/g/giadarol/sim_workspace_mpi
mkdir virtualenvs
cd virtualenvs
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/python27/bin/virtualenv py2.7 [continues]
         --python=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/python27/bin/python

Step 5: We activate our brand new virtual environment

cd py2.7/bin
source ./activate

If we type:

which python

we get:

/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/virtualenvs/py2.7/bin/python

Step 6: We use pip to install the python modules that we need

 pip install numpy
 pip install scipy
 pip install cython
 pip install ipython
 pip install matplotlib
 pip install h5py

Step 7 (optional): Installing mpi4pi

I we do not have an MPI installation we need to get one (for CERN users: skip this for CNAF cluster):

 cd /afs/cern.ch/work/g/giadarol/sim_workspace_mpi/python_src
 wget https://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.2.tar.bz2
 tar jxf openmpi-1.10.2.tar.bz
 cd openmpi-1.10.2
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/openmpi
 make all install

We set the environment variable for the MPI compiler:

export MPICC=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/openmpi/bin/mpicc

At this point be careful not to have other MPI paths in your environment (for example set in your .bashrc)

pip will use the compiler pointed by your MPICC variable to compile mpi4py:

pip install mpi4py

Important: Python jobs using mpi4py must be run with the mpiexec corresponding to the MPI compiler that has been used. In our case:

/afs/cern.ch/work/g/giadarol/sim_workspace_mpi/openmpi/bin/mpiexec

Of course we can add the folder to our PATH or create a shortcut.

Step 8 (optional):: At this point we are able to download and install the PyECLOUD-PyHEADTAIL environment:

wget http://giadarol.web.cern.ch/giadarol/download_and_install_PyECPyHT.sh
sh download_and_install_PyECPyHT.sh

For the CERN linux servers (lxplus-lxbatch): when compiling PySUSSIX (with f2py) we found that some libraries were missing. The problem could be solved by:

export LD_LIBRARY_PATH=/afs/cern.ch/sw/lcg/contrib/gcc/4.8.1/x86_64-slc6-gcc48-opt/lib64

A couple of references that helped with this task:

[1] https://www.open-mpi.org/faq/?category=building#easy-build

[2] http://stackoverflow.com/questions/5506110/is-it-possible-to-install-another-version-of-python-to-virtualenv

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.