Merge pull request #180 from eirrgang/gmxapi-177

Update documentation to resolve #177

.travis.yml

@@ -64,10 +64,7 @@ install:
 before_script:
   - export CCACHE_DIR=$HOME/.ccache_gmxapi
   - ccache -s
-  - $PYTHON -m pip install --upgrade pip setuptools -q
-  # Make sure mpi4py is built with the intended compiler
-  - $PYTHON -m pip install --no-cache-dir --upgrade --no-binary ":all:" --force-reinstall mpi4py -q
-  - $PYTHON -m pip install -r requirements.txt -q
+  - eval $(./ci_scripts/prepare_python.sh)
 
 script:
   - source $HOME/install/gromacs_2019/bin/GMXRC && ./ci_scripts/pygmx.sh

README.md

@@ -1,18 +1,21 @@
-The gmxapi project provides interfaces for managing and extending molecular dynamics simulation workflows.
-In this repository, a Python package provides the `gmx` module for high-level interaction with GROMACS.
-`gmx.core` provides Python bindings to the `gmxapi` C++ GROMACS external API.
-
-The project is hosted on [GitHub](https://github.com/kassonlab/gmxapi) and includes
-the `gmxapi` repository along with supporting respositories.
-The [`gromacs-gmxapi`](https://github.com/kassonlab/gromacs-gmxapi) repository includes a modified version of GROMACS that
-supports the latest `gmxapi` features not yet available through an official GROMACS distribution.
-A [sample repository](https://github.com/kassonlab/sample_restraint) illustrates
+The gmxapi project provides interfaces for managing and extending molecular
+dynamics simulation workflows. In this repository, a Python package provides the 
+`gmx` module for high-level  interaction with GROMACS. `gmx.core` provides 
+Python bindings to the `gmxapi` C++ GROMACS external API.
+
+The project is hosted on [GitHub](https://github.com/kassonlab/gmxapi) and 
+includes the `gmxapi` repository along with supporting respositories. The 
+[`gromacs-gmxapi`](https://github.com/kassonlab/gromacs-gmxapi) repository 
+includes a modified version of GROMACS that supports the latest `gmxapi` 
+features not yet available through an official GROMACS distribution. A 
+[sample repository](https://github.com/kassonlab/sample_restraint) illustrates
 how to implement a GROMACS plugin that applies restrained-ensemble forces
 to a loosely-coupled ensemble of simulation trajectories.
 
 The whole thing is driven at the highest level by a simple Python interface.
 
-See the latest gmxapi documentation at [http://gmxapi.readthedocs.io/en/latest/](http://gmxapi.readthedocs.io/en/latest/)
+See the latest documentation at 
+[http://gmxapi.readthedocs.io/](http://gmxapi.readthedocs.io/en/latest/)
 
 ##### Reference:
 Irrgang, M. E., Hays, J. M., & Kasson, P. M.
@@ -65,14 +68,15 @@ number of nodes equal to the length of the `tpr_list` array, parallelizing each
 simulation across a single node, periodically sharing data via MPI calls across
 the ensemble.
 
-gmxapi is just entering beta, so many more features and more flexibility are to
-come. Feel free to make suggestions or describe your own priorities and research
-needs in the issue tracking system.
+Many more features and more flexibility are still to come. Feel free to make
+suggestions or describe your own  priorities and research needs in the 
+[issue tracking system](https://github.com/kassonlab/gmxapi/issues).
 
 # Installation and getting started
 
-A compatible version of GROMACS must either be already installed or may be installed
-as part of package installation. See [`gromacs-gmxapi`](https://github.com/kassonlab/gromacs-gmxapi)
+A compatible version of GROMACS must be installed before building the Python
+package. See [`gromacs-gmxapi`](https://github.com/kassonlab/gromacs-gmxapi)
+or GROMACS 2019.
 
 We recommend installing gmxapi in a Python virtual environment. See [docs/install.rst](docs/install.rst) for details
-or refer to the online [user documentation](http://gmxapi.readthedocs.io/en/latest/).
+or refer to the online [user documentation](http://gmxapi.readthedocs.io/).

RELEASE.txt

@@ -1,26 +1,32 @@
-Future release notes go here...
+# Brief release notes
 
-When a new release is about to be tagged, version and release information needs to be updated in a few places.
+The gmxapi 0.0.7 C++ API, defined by GROMACS 2019, is incompatible with gmxapi 0.0.6.
+The gromacs-gmxapi 0.0.8 development branch will claim compatibility with 0.0.7
+while possible.
 
-Once the required features in gromacs-gmxapi have been merged to its master branch, update the several lines in
-.travis.yml to reference the gromacs-gmxapi release archive instead of master or a development branch.
+Note that version compatibility in ``find_package`` is handled by the installed
+cmake version file of the targeted package, including handling of the `EXACT`
+key word, but the CMake macros used in gromacs-gmxapi should take care of this.
+
+See docs/changelog.rst and https://github.com/kassonlab/gmxapi for more information.
+
+# Preparing for a release
+
+When a new release is about to be tagged, version and release information needs
+to be updated in a few places.
+
+Once the required features in gromacs-gmxapi have been merged to its master
+branch, update the several lines in .travis.yml to reference the gromacs-gmxapi
+release archive instead of master or a  development branch.
 
 Similarly, update the `gromacs_url` embedded in setup.py
 
 After merging the gmxapi development branch to master and before tagging the release, 
 update version.in so that it writes a version.py file with `release = True`.
 
-Once the gmxapi gmxpy release is tagged and any touch-up commits are merged (fast-forwarded) to the refreshed
-development branch,
+Once the gmxapi gmxpy release is tagged and any touch-up commits are merged
+(fast-forwarded) to the refreshed development branch,
 bump the version in the development branch in CMakeLists.txt _and_ in setup.py.
 
-If / when feature branches require new libgmxapi features, `find_package(gmxapi...` can be updated in
-`src/gmx/core/CMakeLists.txt`.
-
-As of gromacs-gmxapi 0.0.6, we are tracking compatibility more closely, and the gromacs-gmxapi 0.0.7 development branch
-will claim compatibility with 0.0.6 while possible.
-
-Note that version compatibility in find_package is handled by the installed
-cmake version file of the targeted package, including handling of the `EXACT` key word, but the CMake macros used in
-gromacs-gmxapi should take care of this.
-
+If / when feature branches require new libgmxapi features,
+`find_package(gmxapi...)` can be updated in `src/gmx/core/CMakeLists.txt`.

ci_scripts/install_gromacs_2019.sh

@@ -5,8 +5,14 @@ export GMX_DOUBLE=OFF
 export GMX_MPI=OFF
 export GMX_THREAD_MPI=ON
 
-export CCACHE_DIR=$HOME/.ccache_gmxapi
-ccache -s
+CCACHE=`which ccache`
+if [ "${CCACHE}" ] ; then
+    export CCACHE_DIR=$HOME/.ccache_gmxapi
+    ccache -s
+    export ENABLE_CCACHE=ON
+else
+    export ENABLE_CCACHE=OFF
+fi
 
 pushd $HOME
  [ -d gromacs-gmxapi ] || git clone --depth=1 --no-single-branch https://github.com/gromacs/gromacs.git gromacs
@@ -17,8 +23,9 @@ pushd $HOME
   rm -rf build
   mkdir build
   pushd build
-   cmake -DCMAKE_CXX_COMPILER=$CXX \
-         -DGMX_ENABLE_CCACHE=ON \
+   cmake -DGMX_BUILD_HELP=OFF \
+         -DGMX_ENABLE_CCACHE=$ENABLE_CCACHE \
+         -DCMAKE_CXX_COMPILER=$CXX \
          -DCMAKE_C_COMPILER=$CC \
          -DGMX_DOUBLE=$GMX_DOUBLE \
          -DGMX_MPI=$GMX_MPI \
@@ -29,4 +36,4 @@ pushd $HOME
   popd
  popd
 popd
-ccache -s
+[ "$ENABLE_CCACHE" ] && ccache -s

ci_scripts/install_gromacs_devel.sh

@@ -5,6 +5,15 @@ export GMX_DOUBLE=OFF
 export GMX_MPI=OFF
 export GMX_THREAD_MPI=ON
 
+CCACHE=`which ccache`
+if [ "${CCACHE}" ] ; then
+    export CCACHE_DIR=$HOME/.ccache_gmxapi
+    ccache -s
+    export ENABLE_CCACHE=ON
+else
+    export ENABLE_CCACHE=OFF
+fi
+
 pushd $HOME
  [ -d gromacs-gmxapi ] || \
      git clone \
@@ -20,7 +29,7 @@ pushd $HOME
   mkdir build
   pushd build
    cmake -DGMX_BUILD_HELP=OFF \
-         -DGMX_ENABLE_CCACHE=ON \
+         -DGMX_ENABLE_CCACHE=$ENABLE_CCACHE \
          -DCMAKE_CXX_COMPILER=$CXX \
          -DCMAKE_C_COMPILER=$CC \
          -DGMX_DOUBLE=$GMX_DOUBLE \
@@ -32,4 +41,4 @@ pushd $HOME
   popd
  popd
 popd
-ccache -s
+[ "$ENABLE_CCACHE" ] && ccache -s

ci_scripts/prepare_python.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+#
+# Prepare the environment for the current Python interpreter. Assumes the
+# current shell environment variable PYTHON contains the path to the intended
+# Python interpreter. Assumes the environment variable MPICC is set to an
+# appropriate compiler wrapper (will be used to build mpi4py). Then this script
+# makes sure mpi4py is built with the intended compiler.
+#
+# To use, evaluate the results of this script in the root directory of the
+# gmxapi repository, i.e. the directory containing `requirements.txt`
+#
+# Example:
+#     $ export PYTHON=`which python`
+#     $ eval $(./ci_scripts/prepare_python.sh)
+#
+
+if [ -x "$PYTHON" ] ; then
+    # Quiet output if QUIET is non-null
+    echo "$PYTHON -m pip install --upgrade pip setuptools ${QUIET:+'-q'};
+    $PYTHON -m pip install --no-cache-dir --upgrade --no-binary \":all:\" --force-reinstall mpi4py  ${QUIET:+'-q'};
+    $PYTHON -m pip install -r requirements.txt ${QUIET:+'-q'}"
+else
+    echo "echo Set PYTHON to a Python interpreter before using prepare_python.sh"
+fi

ci_scripts/pygmx.sh

@@ -1,22 +1,45 @@
 #!/bin/bash
+#
+# Build, install, and test the `gmx` Python package against a gmxapi-compatible
+# GROMACS installation.
+#
+# Requirements:
+#
+# The script assumes the following environment has been set.
+# - PYTHON holds the full path to the Python interpreter for a (virtual) environment
+#   in which pip, pytest, and the various dependency modules are installed.
+# - CC holds the compiler that is also associated with the `mpicc` used to build
+#   the mpi4py Python package.
+# - CXX holds the C++ compiler used to build GROMACS.
+# - GMXRC for a thread-MPI GROMACS installation has been sourced.
+#
 set -ev
 
+#
 # Test install with pip
+#
 $PYTHON -m pip uninstall -y gmx
+# Clean up some of the dirty built-in-source directories that could cause
+# inappropriate caching.
 rm -rf _skbuild dist gmx* build
 $PYTHON -m pip install .
 
 $PYTHON -m pytest src/gmx/test
 
+#
 # Test install with setup.py
+#
 $PYTHON -m pip uninstall -y gmx
 rm -rf _skbuild dist gmx* build
 $PYTHON setup.py install
 
 $PYTHON -m pytest src/gmx/test
 
+#
 # Test CMake-driven install
+#
 pip uninstall -y gmx
+rm -rf _skbuild dist gmx* build
 
 rm -rf build
 mkdir -p build
@@ -28,5 +51,9 @@ popd
 
 mpiexec -n 2 $PYTHON -m mpi4py -m pytest src/gmx/test
 
+# Check how well our compiler cache is working.
+`which ccache` && ccache -s
+
+# Generate the list of Python packages present in the current environment.
+# Helpful, but not necessarily sufficient, for reproducibility.
 $PYTHON -m pip freeze
-ccache -s

ci_scripts/sample_restraint.sh

@@ -1,4 +1,21 @@
 #!/bin/bash
+#
+# Build, install, and test the MD plugin code from the kassonlab/sample_restraint
+# repository on GitHub.
+#
+# This script uses an argument (${1}) to specify the branch to check out and build.
+#
+# Requirements:
+#
+# The script assumes the following environment has been set.
+# - PYTHON holds the full path to the Python interpreter for a (virtual) environment
+#   in which pip, pytest, and the various dependency modules are installed.
+# - the `gmx` Python package has been installed before tests are run.
+# - GMXRC for a thread-MPI GROMACS installation has been sourced.
+# - mpiexec is available on the PATH and able to launch a 2-rank environment.
+#
+
+#Be verbose and exit quickly.
 set -ev
 
 pushd $HOME
@@ -10,11 +27,14 @@ pushd $HOME
   pushd build
    cmake .. -DPYTHON_EXECUTABLE=$PYTHON
    make -j2 install
+   # Run C++ unit tests.
    make -j2 test
   popd
   pushd tests
+   # Run basic integration tests
    $PYTHON -m pytest
-   mpiexec -n 2 $PYTHON -m mpi4py -m pytest --log-cli-level=DEBUG -s --verbose
+   # Run integration tests for full parallelism, such as ensemble operations.
+   mpiexec -n 2 $PYTHON -m mpi4py -m pytest # --log-cli-level=DEBUG -s --verbose
   popd
  popd
 popd

docs/changelog.rst

@@ -2,13 +2,26 @@
 Change Log
 ==========
 
-API and Python module versioning information.
+.. rubric:: 0.0.7
 
-List of features and available or target versions, organized by component.
-(See :doc:`components`). Link to documentation in :doc:`reference` section and
-appropriate content in :doc:`userguide` or :doc:`developerguide` if available.
-Should be accompanied by appropriate annotations elsewhere in the docs.
-I.e. See `Sphinx docs. <http://www.sphinx-doc.org/en/stable/markup/para.html>`_
+Interface and feature updates
+
+- :py:class:`gmx.context.Context` is generic (ParallelArrayContext, DefaultContext deprecated)
+- :py:class:`gmx.system.System` deprecated (use :py:func:`gmx.workflow.from_tpr`)
+- Add ``end_time`` keyword argument to :py:func:`gmx.workflow.from_tpr()`
+
+Internal
+
+- substantial updates for compatibility with GROMACS 2019
+- various improvements to CI testing
+- Context functionality is now composed according to available resources.
+  (Resource tags, such as ``ensemble_update`` will be specified in
+  *workspec version 0.2* before the gmxapi 0.1 release.)
+
+Bug fixes
+
+- :py:func:`gmx.get_context()` and :py:func:`gmx.run()` did not behave as expected for all work types.
+- substantial documentation updates
 
 .. rubric:: 0.0.6