Before you begin all the steps below to install the Python runtime for the Whole Cell Model, consider how much easier it is to run it within a Docker container.
The script cloud/build-containers-locally.sh
will build a Docker Image on
your computer if you have Docker Engine or Docker Desktop installed.
The script cloud/build.sh
will do the same using Google Container Registry if
you have a Google Cloud project set up.
Either way, you can run the Whole Cell Model (WCM) in a Docker Container, isolated from your computer's operating system, Python versions, binary libraries, and everything else installed.
See docs/README.
The requirements.txt file lists the Python packages needed for the Python runtime environment, with terse setup instructions.
This page goes through the Python environment setup steps in more detail and with more options, but requirements.txt
gets updated more frequently since that file specifies the current package versions.
NOTE: While you can create virtual environments using
virtualenv
or venv
in place of pyenv
, be sure to put the environment
outside the wcEcoli/
directory. Otherwise make clean
will break it!
Sherlock: Sherlock is the Stanford scientific computing cluster. Outside the Covert lab, just skip our Sherlock notes. Inside the lab, look in $GROUP_HOME/downloads/
and $GROUP_HOME/installation_notes/
for downloaded software packages and notes on recompiling them as needed to install new packages, new libraries, and new Python releases for the team.
- Install the software tools as described in dev-tools, including
- pyenv and pyenv-virtualenv
- initializing pyenv in your shell profile
- gcc or llvm
- git
- a programming editor such as PyCharm, Sublime Text, or Visual Studio Code
- Set up Git and GitHub including Connecting to GitHub with SSH.
- Clone the repo wcEcoli git to a local directory like
~/dev/wcEcoli/
. See About remote repositories. It's probably better to use the SSH URLgit@github.com:CovertLab/wcEcoli.git
but it's also possible to use an HTTPS URL.
-
Use your package manager to install the needed libraries [see the
requirements.txt
file for the latest list] or compile them from source.Most of this list comes from pyenv's requirements to install Python releases, so check the pyenv wiki for the latest list of libraries.
We no longer recommend installing
openblas
library using a package manager or by compiling from source. Instead we let NumPy and SciPy install their own copy and let Aesara find NumPy's copy.You can optionally install
openblas
via package manager or source code, but be sure to get at least release v0.3.9.On macOS
brew install glpk openssl readline swig suite-sparse xz
On Ubuntu
sudo apt install -y glpk-utils libglpk-dev glpk-doc libssl-dev libreadline-dev \ libncurses5-dev libncursesw5-dev libffi-dev zlib1g-dev libbz2-dev xz-utils \ libsqlite3-dev tk-dev
For Ubuntu, you might also need to find and install the proprietary package
python-glpk
.Don't use apt-get to install
libopenblas-dev
until that package repository updates to a recent release of OpenBLAS such as the one that's embedded in numpy and scipy package wheels. On Ubuntu, useapt-cache policy libopenblas-dev
to check the candidate version. To see which version of OpenBLAS is embedded in numpy, see openblas_support.py in the relevant numpy release tag. Recommendation: Let numpy and scipy install their embedded copies of OpenBLAS (see below).On Sherlock
The needed packages are already installed. Set up your bash profile to locate the group environment modules, load the git and python modules, and initialize
pyenv
. You'll need these newer git modules since they use a compatible version oflibressl
.export PI_HOME=$GROUP_HOME ##### Add group-wide path settings ##### if [ -f "${PI_HOME}/etc/bash_profile" ]; then . "${PI_HOME}/etc/bash_profile" fi module load git/2.27.0 git-lfs/2.11. module load wcEcoli/python3 export PYENV_ROOT="${PI_HOME}/pyenv" if [ -d "${PYENV_ROOT}" ]; then export PATH="${PYENV_ROOT}/bin:${PATH}" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)" fi
-
Optional: Download and install other packages according to their instructions or take a wait-and-see approach with them.
- CPLEX from IBM (free for students)
-
Install Python 3 in a shared pyenv for the team if it needs updating.
See
$GROUP_HOME/installation_notes/python3.txt
.If you need to update binary libraries like libressl, readline, or libffi, see their
$GROUP_HOME/installation_notes/*.txt
files.Each of these libraries and tools needs an environment module. We
module load
the module to make it accessible via environment variable paths likeCPPFLAGS
. See for example$GROUP_HOME/modules/xz/5.2.5.lua
.
-
Install Python using
pyenv
, enabling Python to be loaded as a shared library.PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.11.3
Note: If running Parca gets an error message that "the loader can't load libpython"
or an error related to "shared objects" and "needing to compile with -fPIC", use
the above command that has --enable-shared
.
Note: If pyenv install
produces an error message like
No module named _lizma
in WSL on Windows, try running
sudo apt install liblzma_dev
, then uninstall and reinstall Python with pyenv.
-
Use
pyenv
.pyenv install 3.11.3
On Sherlock: Skip this section and instead run pyenv local wcEcoli3
to set up
your project directory unless you need to update or rebuild the team's shared
virtualenv.
-
Create a pyenv virtualenv and select it in your project directory.
cd ~/dev/wcEcoli # or wherever you cloned the `wcEcoli` project to pyenv virtualenv 3.11.3 wcEcoli3 && pyenv local wcEcoli3
-
Upgrade this virtual environment's installers.
pip install --upgrade pip setuptools virtualenv virtualenvwrapper virtualenv-clone wheel
-
CONDITIONAL: Link numpy and scipy to the BLAS linear algebra library of your choice.
See Issue #931. There are several degrees of freedom for installing a BLAS library, numpy, and scipy that might change the computed results. We do not know how to set up environments to get consistent cross-platform results. Unfortunately, that means WCM simulation results and bugs can vary across platforms.
The simplest setup is to
pip install
numpy and scipy from precompiled binary "wheels", that is, without the--no-binary numpy,scipy
option. This uses their default BLAS library -- usually an embedded copy of OpenBLAS.You can use a package manager to install a specific version of the OpenBLAS library to help with consistency or bug fixes.
TODO: Document how to install numpy 1.26+ and scipy 1.11+ using
--no-binary
and specified linear algebra libraries, e.g.,-Csetup-args=-Dblas=accelerate -Csetup-args=-Dlapack=accelerate
. (The~/.numpy-site.cfg
file is obsolete.)To use an OpenBLAS library, you need to know where it is.
- Brew on macOS installs OpenBLAS in
brew --prefix openblas
. - For other package managers, find out where they installed
lib/libopenblas*
. - Compiling OpenBLAS from source in Ubuntu goes into
/opt/OpenBLAS/
by default. - Compiling from source with
make FC=gfortran && make PREFIX=/XYZ install
installs it in that specified/XYZ
PREFIX directory. - On Sherlock, it's installed in
$GROUP_HOME/downloads-sherlock2/compiled/openblas
. (Using an environment module to load the OpenBLAS when installing numpy and scipy works if the same environment module is loaded at runtime.)
- Brew on macOS installs OpenBLAS in
-
Install NumPy.
Install numpy before installing
scipy
andstochastic-arrow
to avoid installation errors.pip install numpy==1.26.3 # see requirements.txt for the right version
Or:
pip install numpy==1.26.3 --no-binary numpy # see requirements.txt for the right version
-
Install the packages listed in
requirements.txt
.pip install -r requirements.txt && pyenv rehash
Or:
LDFLAGS="-shared $LDFLAGS" pip install -r requirements.txt --no-binary numpy,scipy && pyenv rehash
The
LDFLAGS="-shared $LDFLAGS"
preamble fixes dozens of scipy build errors starting with
In function _start (.text+0x20): undefined reference to main
and
undefined reference to PyFloat_FromDouble
. -
Prerequisite for the following Python steps: Set the
PYTHONPATH
environment variable, e.g.export PYTHONPATH=$PWD
or run the
ppath
alias recommended in dev-tools.md. -
Test the NumPy and SciPy installation.
python runscripts/debug/summarize_environment.py
It should print entries like the ones below for numpy and scipy showing which BLAS library they link to.
library_dirs = ['/usr/local/opt/openblas/lib']
is a path to source openblas whilelibrary_dirs = ['/usr/local/lib']
indicates numpy's embedded openblas.lapack_opt_info: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/local/opt/openblas/lib'] define_macros = [('HAVE_CBLAS', None)] language = c
or this:
lapack_opt_info: libraries = ['openblas', 'openblas'] library_dirs = ['/usr/local/lib'] language = c define_macros = [('HAVE_CBLAS', None)]
(NumPy's embedded OpenBLAS library is in the virtualenv's
site-packages/numpy.libs/
on Linux, orsite-packages/numpy/.dylibs/
on macOS, orsite-packages/numpy/.libs/
on Windows.) -
Required: Add the following line to your shell profile and run it in your current shell. This gets more consistent results from OpenBLAS and it improves performance significantly, especially when called from multiple processes.
export OPENBLAS_NUM_THREADS=1
-
Time the NumPy and SciPy libraries
runscripts/debug/time_libraries.sh
(It might fail some timing expectations with a message like "AssertionError: 0.xxxxx not less than or equal to 0.4.")
-
Test Aesara:
python -c 'import aesara; print([aesara.config.blas.ldflags, aesara.config.device, aesara.config.floatX])'
It should print something like this (with variations if you compiled OpenBLAS from source):
['-lblas', 'cpu', 'float64']
-
Compile the project's native code.
make clean compile
(Yes, it's expected to print deprecation warnings.)
-
Run the unit tests and runParca.py.
pytest ppath python runscripts/manual/runParca.py
If the unit tests or runParca.py fail with an error message saying the loader can't load ...libpython... or an error related to "shared objects" and "needing to compile with -fPIC", that means you need to
--enable-shared
when installing python. Go back to that step, runPYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.11.3 --force
then delete and recreate the virtualenv
wcEcoli3
. Delete it via the commandpyenv virtualenv-delete wcEcoli3
orpyenv uninstall wcEcoli3
. -
Remember to copy the git hooks from the repo (see git hooks) to your
.git
directory to maintain an up to date environment while doing development:cp runscripts/git_hooks/*[^.md] .git/hooks/
-
If you're using PyCharm, be sure to select the project's Python interpreter so PyCharm understands the version of Python and its installed libraries. This enables code completion, usage documentation in context, visual debugging, warnings about code problems, click-through to library source code, etc.
PyCharm >
Preferences >
Project: wcEcoli >
Project Interpreter >
gear ⚙️ >
Add... >
Virtualenv Environment >
Existing environment >
Interpreter >
[runpyenv which python
in a shell to find the python location, something like/usr/local/var/pyenv/versions/wcEcoli3/python
, and paste that path into the text box or navigate there].
-
Make sure the model's output goes to the
$SCRATCH
filesystem (which is larger) rather than SHERLOCK HOME.mkdir $SCRATCH/wcEcoli_out cd wcEcoli ln -s $SCRATCH/wcEcoli_out out
-
Create a symbolic link to a shared sim data cache directory in
$GROUP_SCRATCH
that should contain a copy of the newest sim data object (it should be updated by the daily build):ln -s $GROUP_SCRATCH/wc_ecoli/cached cached