Merge pull request #43 from desihub/desi-conda

Support new DESI anaconda infrastructure.

doc/changes.rst

@@ -5,7 +5,12 @@ Change Log
 1.9.0 (unreleased)
 ------------------
 
-* No changes yet.
+* Shorten Python version printed in dependency headers.
+* :mod:`desiutil.test.test_plots` was not cleaning up after itself.
+* Support new DESI+Anaconda software stack infrastructure (`PR #43`_).
+
+.. _`PR #43`: https://github.com/desihub/desiutil/pull/43
+
 
 1.8.0 (2016-09-10)
 ------------------

doc/desiInstall.rst

@@ -45,7 +45,16 @@ file::
     # This section can override details of Module file installation.
     #
     [Module Processing]
+    #
+    # nersc_module_dir overrides the Module file install directory for
+    # ALL NERSC hosts.
+    #
     nersc_module_dir = /project/projectdirs/desi/test/modules
+    #
+    # cori_module_dir overrides the Module file install directory only
+    # on cori.
+    #
+    cori_module_dir = /global/common/cori/contrib/desi/test/modules
 
 Finally, desiInstall both reads and sets several environment variables.
 
@@ -66,7 +75,7 @@ Environment variables that strongly affect the behavior of desiInstall.
     :command:`svn`.
 
 Environment variables that are *set* by desiInstall for use by
-``python setup.py`` or ``make``.
+:command:`python setup.py` or :command:`make`.
 
 :envvar:`INSTALL_DIR`
     This variable is *set* by desiInstall to the directory that will contain
@@ -96,7 +105,8 @@ manipulated by setting up Modules, or loading Module files.
     It may be manipulated by :mod:`desiutil.modules`.
 :envvar:`MODULESHOME`
     This variable points to the Modules infrastructure.  If it is not set,
-    it typically means that the system has no Modules infrastructure.
+    it typically means that the system has no Modules infrastructure. This
+    is needed to find the executable program that reads Module files.
 :envvar:`PYTHONPATH`
     Obviously this is important for any Python package!  :envvar:`PYTHONPATH`
     may be manipulated by :mod:`desiutil.modules`.
@@ -106,6 +116,59 @@ manipulated by setting up Modules, or loading Module files.
 
 .. _desiutil: https://github.com/desihub/desiutil
 
+Directory Structure Assumed by the Install
+==========================================
+
+desiInstall is primarily intended to run in a production environment that
+supports Module files.  In practice, this means NERSC, though it can also
+install on any other system that has a Modules infrastructure installed.
+
+*desiInstall does not install a Modules infrastructure for you.* You have to
+do this yourself, if your system does not already have this.
+
+For the purposes of this section, we define ``$product_root`` as the
+directory that desiInstall will be writing to.  This directory could be the
+same as :envvar:`DESI_PRODUCT_ROOT`, but for standard NERSC installs it
+defaults to a pre-defined value. ``$product_root`` may contain the following
+directories:
+
+code/
+    This contains the installed code, the result of :command:`python setup.py install`
+    or :command:`make install`.  The code is always placed in a ``product/version``
+    directory.  So for example, the full path to desiInstall might be
+    ``$product_root/code/desiutil/1.8.0/bin/desiInstall``.
+conda/
+    At NERSC, this contains the Anaconda_ infrastructure.  desiInstall does
+    not manipulate this directory in any way, though it may *use* the
+    :command:`python` executable installed here.
+modulefiles/
+    This contains the the Module files installed by desiInstall.  A Module
+    file is almost always named ``product/version``.  For example, the
+    Module file for desiutil might be ``$product_root/modulefiles/desiutil/1.8.0``.
+
+.. _Anaconda: https://www.continuum.io
+
+Within a ``$product_root/code/product/version`` directory, you might see the
+following:
+
+bin/
+    Contains command-line executables, including Python or Shell scripts.
+data/
+    Rarely, packages need data files that cannot be incorporated into the
+    package structure itself, so it will be installed here.  desimodel_ is
+    an example of this.
+etc/
+    Miscellaneous metadata and configuration.  In most packages this only
+    contains a template Module file.
+lib/pythonX.Y/site-packages/
+    Contains installed Python code.  ``X.Y`` would be ``2.7`` or ``3.5``.
+py/
+    Sometimes we need to install a git checkout rather than an installed package.
+    If so, the Python code will live in *this* directory not the ``lib/``
+    directory, and the product's Module file will be adjusted accordingly.
+
+.. _desimodel: https://github.com/desihub/desimodel
+
 Stages of the Install
 =====================
 
@@ -131,14 +194,14 @@ Product Existence
 After the product name and version have been determined, desiInstall
 constructs the full URL pointing to the product/version and runs the code
 necessary to verify that the product/version really exists.  Typically, this
-will be ``svn ls``, unless a GitHub install is detected.
+will be :command:`svn ls`, unless a GitHub install is detected.
 
 Download Code
 -------------
 
-The code is downloaded, using ``svn export`` for standard (tag) installs, or
-``svn checkout`` for trunk or branch installs.  For GitHub installs, desiInstall
-will look for a release tarball, or do a ``git clone`` for tag or master/branch
+The code is downloaded, using :command:`svn export` for standard (tag) installs, or
+:command:`svn checkout` for trunk or branch installs.  For GitHub installs, desiInstall
+will look for a release tarball, or do a :command:`git clone` for tag or master/branch
 installs.  desiInstall will set the environment variable :envvar:`WORKING_DIR`
 to point to the directory containing this downloaded code.
 
@@ -153,14 +216,14 @@ plain
     is simply copied to the final install directory.
 py
     If a setup.py file is detected, desiInstall will attempt to execute
-    ``python setup.py install``.  This build type can be suppressed with the
+    :command:`python setup.py install`.  This build type can be suppressed with the
     command line option ``--compile-c``.
 make
     If a Makefile is detected, desiInstall will attempt to execute
-    ``make install``.
+    :command:`make install`.
 src
     If a Makefile is not present, but a src/ directory is,
-    desiInstall will attempt to execute ``make -C src all``.  This build type
+    desiInstall will attempt to execute :command:`make -C src all`.  This build type
     *is* mutually exclusive with 'make', but is not mutually exclusive with
     the other types.
 
@@ -171,12 +234,24 @@ Determine Install Directory
 ---------------------------
 
 The install directory is where the code will live permanently.  If the
-install is taking place at NERSC, the install directory will be placed in
-``/project/projectdirs/desi/software/${NERSC_HOST}``.
+install is taking place at NERSC, the top-level install directory is
+predetermined based on the value of :envvar:`NERSC_HOST`.
+
+edison
+    ``/global/common/edison/contrib/desi``
+cori
+    ``/global/common/cori/contrib/desi``
+datatran
+    ``/global/project/projectdirs/desi/software/datatran``
+scigate
+    ``/global/project/projectdirs/desi/software/scigate``
 
 At other locations, the user must set the environment variable
 :envvar:`DESI_PRODUCT_ROOT` to point to the equivalent directory.
 
+The actual install directory is determined by appending ``/code/product/verson``
+to the combining the top-level directory listed above.
+
 If the install directory already exists, desiInstall will exit, unless the
 ``--force`` parameter is supplied on the command line.
 
@@ -213,7 +288,7 @@ desiInstall will scan :envvar:`WORKING_DIR` to determine the details that need
 to be added to the module file.  The final module file will then be written
 into the DESI module directory at NERSC or the module directory associated
 with :envvar:`DESI_PRODUCT_ROOT`.  If ``--default`` is specified on the command
-line, an approproate .version file will be created.
+line, an appropriate .version file will be created.
 
 Load Module
 -----------
@@ -239,7 +314,7 @@ Copy All Files
 
 The entire contents of :envvar:`WORKING_DIR` will be copied to :envvar:`INSTALL_DIR`.
 If this is a trunk or branch install and a src/ directory is detected,
-desiInstall will attempt to run ``make -C src all`` in :envvar:`INSTALL_DIR`.
+desiInstall will attempt to run :command:`make -C src all` in :envvar:`INSTALL_DIR`.
 For trunk or branch installs, no further processing is performed past this
 point.
 
@@ -248,19 +323,19 @@ Create site-packages
 
 If the build-type 'py' is detected, a site-packages directory will be
 created in :envvar:`INSTALL_DIR`.  If necessary, this directory will be
-added to Python's ``sys.path``.
+added to Python's :data:`sys.path`.
 
 Run setup.py
 ------------
 
-If the build-type 'py' is detected, ``python setup.py install`` will be run
+If the build-type 'py' is detected, :command:`python setup.py install` will be run
 at this point.
 
 Build C/C++ Code
 ----------------
 
-If the build-type 'make' is detected, ``make install`` will be run in
-:envvar:`WORKING_DIR`.  If the build-type 'src' is detected, ``make -C src all``
+If the build-type 'make' is detected, :command:`make install` will be run in
+:envvar:`WORKING_DIR`.  If the build-type 'src' is detected, :command:`make -C src all`
 will be run in :envvar:`INSTALL_DIR`.
 
 Cross Install

etc/desiutil.module

@@ -52,11 +52,11 @@ module-whatis "Sets up $product/$version in your environment."
 # will need to set the DESI_PRODUCT_ROOT environment variable
 #
 if {{[info exists env(DESI_PRODUCT_ROOT)]}} {{
-    set PRODUCT_ROOT $env(DESI_PRODUCT_ROOT)
+    set code_root $env(DESI_PRODUCT_ROOT)/code
 }} else {{
-    set PRODUCT_ROOT /project/projectdirs/desi/software/$env(NERSC_HOST)
+    set code_root {product_root}
 }}
-set PRODUCT_DIR $PRODUCT_ROOT/$product/$version
+set PRODUCT_DIR $code_root/$product/$version
 #
 # This line creates an environment variable pointing to the install
 # directory of your product.

py/desiutil/depend.py

@@ -134,15 +134,18 @@ def iterdep(header):
     'redmonster', 'specter', 'speclite', 'specsim',
     ]
 
-def add_dependencies(header, module_names=None):
+def add_dependencies(header, module_names=None, long_python=False):
     '''Adds DEPNAMnn, DEPVERnn keywords to header for imported modules
 
     Args:
-        header : dict-like object, e.g. astropy.io.fits.Header
+        header : dict-like object, *e.g.* :class:`astropy.io.fits.Header`.
 
     Options:
         module_names : list of module names to check
             if None, checks desiutil.depend.possible_dependencies
+        long_python : If ``True`` use the full, verbose ``sys.version``
+            string for the Python version.  Otherwise, use a short
+            version, *e.g.*, ``3.5.2``.
 
     Only adds the dependency keywords if the module has already been
     previously loaded in this python session.  Uses module.__version__
@@ -151,7 +154,10 @@ def add_dependencies(header, module_names=None):
     import sys
     import importlib
 
-    setdep(header, 'python', sys.version.replace('\n', ' '))
+    py_version = ".".join(map(str, sys.version_info[0:3]))
+    if long_python:
+        py_version = sys.version.replace('\n', ' ')
+    setdep(header, 'python', py_version)
 
     if module_names is None:
         module_names = possible_dependencies