Support the use of environments with names that do not match the cylc…

… flow version (e.g. cylc-8.0.0-1)

Ensure workflows always run under fixed environments rather than symlinked versions which could change
Provide legacy support for rose edit & cylc review (partially address cylc#4555)

cylc/flow/etc/cylc

@@ -17,95 +17,71 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 #------------------------------------------------------------------------------
-# Wrapper script to support multiple installed Cylc & Rose versions. Handles
-# Conda and Python virtual environments, and legacy plain installations.
+# Wrapper script to support multiple installed Cylc, Rose & Isodatetime
+# versions. Handles Conda and Python virtual environments, and legacy plain
+# installations.
 #
 # WRAPPER INSTALLATION AND CONFIGURATION
-#---------------------------------------
+# --------------------------------------
 # - Copy this script as "cylc" into default $PATH, on scheduler and job hosts
-# - If using Rose, create a "rose" symlink (ln -s cylc rose)
+# - Create a "isodatetime" symlink (ln -s cylc isodatetime)
+# - If using Rose, create "rose" & "rosie" symlinks
 # - Set CYLC_HOME_ROOT ("EDIT ME" below) to default to the parent directory of
 #   installed versions and set the global config locations if necessary
 #
 # HOW IT WORKS
-#-------------
-# Intercept "cylc" and "rose" commands and re-invoke them with the version
-# selected. If $CYLC_HOME is defined it is used as the installation location.
-# Otherwise, the script looks for a directory named "cylc-$CYLC_VERSION"
-# (or just "cylc" if $CYLC_VERSION is not defined) in $CYLC_HOME_ROOT or
-# $CYLC_HOME_ROOT_ALT.
+# ------------
+# Intercept "cylc", "rose", "rosie" and "isodatetime" commands and re-invoke
+# them with the version selected via the environment variables described below.
 #
-# Additional legacy logic is used when calling rose with Cylc 7.
+# Additional legacy logic is used when calling Rose with Cylc 7.
 #
 # ENVIRONMENT VARIABLES
 # ---------------------
 #
-# Location variables must be set on workflow and job hosts, e.g. in .bashrc.
-#
-#> CYLC_HOME_ROOT - location of installed Cylc environments.
-#   E.g. /opt/cylc for centrally installed releases.
-#
-#> CYLC_HOME_ROOT_ALT - alternate location of installed Cylc environments. Can
-# be set by users with their own Cylc releases or git clones.
-#   E.g. $HOME/cylc
-#
-#> CYLC_VERSION - this wrapper will look for an installed environment called
-# cylc-$CYLC_VERSION in the ROOT locations. The scheduler propagates its own
-# CYLC_VERSION to task job scripts, so if set in (e.g.) .bashrc it should
-# default to scheduler version:
-#   E.g. CYLC_VERSION=${CYLC_VERSION:-8.0.0}
-#
-# In Cylc 8 the scheduler sets CYLC_VERSION=cylc.flow.__version__ in task job
-# scripts. This value only increments with cylc-flow releases so CYLC_VERSION
-# cannot be used for fine-grained selection among (e.g.) git clones. For that,
-# use CYLC_HOME to select a specific virtual environment, or CYLC_HOME_ROOT
-# and CYLC_ENV_NAME (below).
-#
-#> CYLC_HOME - can be set to a specific Cylc environment outside of the ROOT
-# locations, e.g. for a venv inside a Cylc git clone. If set, it overrides
-# CYLC_VERSION selection.
-#
-#> CYLC_ENV_NAME - can be an absolute path to a specific Cylc environment, or
-# a path relative to the ROOT locations. If set, it overrides CYLC_VERSION
-# selection.
-#
-# If CYLC_VERSION and CYLC_ENV_NAME are both set, CYLC_ENV_NAME will be used
-# and CYLC_VERSION ignored.
-
-# INSTALLING Cylc 8
-#------------------
-#
-# Releases should be installed into environments in the ROOT location, named as
-# cylc-$CYLC_VERSION, either via conda (full system) or pip (for cylc-flow only)
-# for selection by the CYLC_VERSION mechanism.
-#
-# To work with git clones (developers) `pip install` your clone into a Python
-# or conda environment, and use CYLC_ENV_NAME to select it.
-#
-#   $ CYLC_ENV_NAME=cylc-violet-capybara cylc version --long
-#   > 8.0.0 (/home/user/miniconda3/envs/cylc-violet-capybara)
-#
-# INSTALLING LEGACY cylc 7 RELEASE TARBALLS BY HAND
-#--------------------------------------------------
-# cylc-flow release tarballs now unpack to (e.g.) cylc-flow-7.9.1. To work with
-# this wrapper the directory should be renamed to "cylc-7.9.1". Then follow
-# version-specific installation instructions. Running "make" should create a
-# file called VERSION that contains just the version string (e.g.) "7.9.1".
-#
-# INSTRUCTIONS FOR USERS
-#-----------------------
-# + Set CYLC_HOME_ROOT_ALT to point local conda releases, e.g.:
-#     $ export CYLC_HOME_ROOT_ALT=$HOME/miniconda3/envs
-# + Set CYLC_VERSION e.g. "8.0.0" to select a specific version in the root
-#     locations. CYLC_VERSION is propagated to task jobs by the scheduler; to
-#     avoid overiding this you should only default to your version:
-#       $ export CYLC_VERSION=${CYLC_VERSION:-8.0.0}
-#   - Do not explicitly select the default "cylc" symlink as a version
-# + Set CYLC_HOME to select a specific Cylc 8 venv or plain Cylc 7 directory
-#   outside of the ROOT locations
-# + Set CYLC_ENV_NAME to a select a specific arbitrarily name Cylc venv (directly,
-#   or under the ROOT locations).
-# + These settings (e.g. in .bashrc) must be replicated on job hosts too.
+# CYLC_VERSION - can be set by users (e.g. in .bash_profile) in order to select a
+# specific Cylc environment. If set this wrapper will look for an installed
+# environment called cylc-$CYLC_VERSION in the ROOT locations (if not set it will
+# look for an installed environment called cylc).
+# e.g.    export CYLC_VERSION=8.0.0-1
+#
+# CYLC_HOME_ROOT - location of installed Cylc environments.
+# Usually defined in this wrapper script.
+#
+# CYLC_HOME_ROOT_ALT - alternate location of installed Cylc environments.
+# Can be set by users for use with their own Cylc environments.
+# If used it must be set on workflow and job hosts, e.g. in .bash_profile.
+# e.g.    export CYLC_HOME_ROOT_ALT=$HOME/miniconda3/envs
+#
+# CYLC_ENV_NAME - gets set to the basename of the selected environment.
+# If set this defines the environment name rather than CYLC_VERSION.
+# The scheduler sets CYLC_ENV_NAME for all remote commands to ensure the same
+# environment is used on all platforms.
+# Users should not set CYLC_ENV_NAME (use CYLC_VERSION).
+#
+# CYLC_HOME - full path to the Cylc environment.
+# Can be set by users in order to use an environment outside of the ROOT
+# locations. However, it is not passed by the scheduler to remote platforms so
+# use of CYLC_VERSION & CYLC_HOME_ROOT_ALT is preferred.
+#
+# INSTALLING ENVIRONMENTS
+# -----------------------
+#
+# Releases should be installed into environments in the ROOT location, named
+# cylc-$CYLC_VERSION. We recommend using the cylc-flow version plus an additional
+# suffix to allow for multiple environments containing the same cylc-flow
+# version. e.g. cylc-8.0.0-1
+#
+# A cylc symlink should be created in the ROOT location to define the default
+# version to used. e.g. ln -s cylc-8.0.0-1 cylc
+# Other symlinks can be created in order to create additional named environments
+# for selection using CYLC_VERSION. e.g. ln -s cylc-8.0.1-1 cylc-next
+#
+# Legacy Cylc 7 and Rose 2019.01 versions can also be installed into environments
+# in the ROOT location. The legacy support for rose edit and cylc review requires
+# cylc-7 and rose-2019.01 symlinks to be created to the preferred environments.
+# e.g.    ln -s cylc-7.9.5 cylc-7
+#         ln -s rose-2019.01.5 rose-2019.01
 #
 ##############################!!! EDIT ME !!!##################################
 # Centrally installed Cylc releases:
@@ -119,77 +95,89 @@ CYLC_HOME_ROOT="${CYLC_HOME_ROOT:-/opt}"
 # export ROSE_SITE_CONF_PATH="${ROSE_SITE_CONF_PATH:-/etc/rose}"
 ###############################################################################
 
-# Prior to Cylc 8, Rose used a standalone installation
-# Note: assumes Cylc 7 is the default version - once Cylc 8 becomes the default
-# the if test below needs to change to
-# if [[ ${0##*/} =~ ^rose && ${CYLC_VERSION:-} =~ ^7 ]]; then
-if [[ ${0##*/} =~ ^rose && \
-      ((-n "${CYLC_ENV_NAME}" && ${CYLC_VERSION:-} =~ ^7) || \
-       (-z "${CYLC_ENV_NAME}" && ! ${CYLC_VERSION:-} =~ ^8)) ]]; then
-    if [[ -z "${ROSE_HOME:-}" ]]; then
-        ROSE_HOME_ROOT="${ROSE_HOME_ROOT:-$CYLC_HOME_ROOT}"
-        if [[ -n "${ROSE_VERSION:-}" ]]; then
-            CYLC_HOME="${ROSE_HOME_ROOT}/rose-${ROSE_VERSION}"
-        else
-            # Default version symlink
-            CYLC_HOME="${ROSE_HOME_ROOT}/rose"
-        fi
-    else
-        CYLC_HOME="${ROSE_HOME}"
-    fi
-fi
-# Note: the code above is only needed if still using standalone Rose 2019
-# versions
-
-# CYLC_ENV_NAME construction:
-#                           ╔════════════════════════════╗
-#                           ║        CYLC_VERSION        ║
-#                           ╠══════════════════╦═════════╣
-#                           ║        set       ║ not set ║
-# ╔═══════════════╦═════════╬══════════════════╩═════════╣
-# ║ CYLC_ENV_NAME ║   set   ║      Use CYLC_ENV_NAME     ║
-# ║               ╠═════════╬══════════════════╦═════════╣
-# ║               ║ not set ║ Use CYLC_VERSION ║  "cylc" ║
-# ╚═══════════════╩═════════╩══════════════════╩═════════╝
 if [[ -z "${CYLC_HOME}" ]]; then
     if [[ -z "${CYLC_ENV_NAME}" ]]; then
         if [[ -n "${CYLC_VERSION}" ]]; then
             CYLC_ENV_NAME="cylc-$CYLC_VERSION"
         else
-            # Default version symlink - export CYLC_ENV_NAME to ensure it
-            # does not get overridden by CYLC_VERSION
-            export CYLC_ENV_NAME="cylc"
+            # Use default version (symlink)
+            CYLC_ENV_NAME="cylc"
         fi
     fi
+    # CYLC_VERSION gets set to the actual version of Cylc used within workflows.
+    # Therefore we export CYLC_ENV_NAME to ensure it gets used rather than
+    # CYLC_VERSION in subsequent calls to the wrapper.
+    export CYLC_ENV_NAME
     for ROOT in "${CYLC_HOME_ROOT}" "${CYLC_HOME_ROOT_ALT}"; do
         if [[ -d "${ROOT}/${CYLC_ENV_NAME}" ]]; then
             CYLC_HOME="${ROOT}/${CYLC_ENV_NAME}"
+            # If CYLC_HOME is a symlink then replace it with the real path and
+            # set CYLC_ENV_NAME to the linked environment to ensure changes to
+            # symlinked environments can't affect running workflows.
+            if [[ -L ${CYLC_HOME} ]]; then
+                CYLC_HOME=$(readlink -f "${CYLC_HOME}")
+                CYLC_ENV_NAME=${CYLC_HOME##*/}
+            fi
             break
         fi
     done
+    if [[ -z "${CYLC_HOME}" ]]; then
+        MSG="ERROR: $CYLC_ENV_NAME not found in $CYLC_HOME_ROOT"
+        if [[ -n "${CYLC_HOME_ROOT_ALT}" ]]; then
+            MSG="${MSG} or ${CYLC_HOME_ROOT_ALT}"
+        fi
+        echo 1>&2 "$MSG"
+        exit 1
+    fi
 fi
-if [[ -z "${CYLC_HOME}" ]]; then
-    MSG="ERROR: $CYLC_ENV_NAME not found in $CYLC_HOME_ROOT"
-    if [[ -n "${CYLC_HOME_ROOT_ALT}" ]]; then
-        MSG="${MSG} or ${CYLC_HOME_ROOT_ALT}"
+
+# Legacy support for Rose
+if [[ ${0##*/} =~ ^ros ]]; then
+    # Prior to Cylc 8, Rose used a standalone installation
+    if [[ -n "${CYLC_ENV_NAME}" ]]; then
+        ROSE_HOME_ROOT="${ROSE_HOME_ROOT:-$CYLC_HOME_ROOT}"
+        if [[ ${CYLC_ENV_NAME:-} =~ ^cylc-7 ]]; then
+            # Cylc 7: Use ROSE_HOME / ROSE_VERSION to select the installation
+            if [[ -z "${ROSE_HOME:-}" ]]; then
+                if [[ -n "${ROSE_VERSION:-}" ]]; then
+                    CYLC_HOME="${ROSE_HOME_ROOT}/rose-${ROSE_VERSION}"
+                else
+                    # Use default version (symlink)
+                    CYLC_HOME="${ROSE_HOME_ROOT}/rose"
+                fi
+            else
+                CYLC_HOME="${ROSE_HOME}"
+            fi
+        elif [[ ${1:-} == "edit" || ${1:-} == "config-edit" ]]; then
+            # Cylc 8: Use Rose 2019.01 to run "config-edit"
+            CYLC_HOME="${ROSE_HOME_ROOT}/rose-2019.01"
+        fi
+    elif [[ -z "${CYLC_ENV_NAME}" ]]; then
+        # If CYLC_HOME was set externally, use ROSE_HOME if it is set
+        CYLC_HOME="${ROSE_HOME:-$CYLC_HOME}"
     fi
-    echo 1>&2 "$MSG"
-    exit 1
 fi
 
-# Warn if executable not at $CYLC_HOME
+# Legacy support for cylc review
+if [[ ${0##*/} == "cylc" && ${1:-} == "review" && \
+      ! ${CYLC_ENV_NAME:-} =~ ^cylc-7 ]]; then
+    # Cylc 8: Use Cylc 7 to run "review"
+    CYLC_HOME="${CYLC_HOME_ROOT}/cylc-7"
+fi
+
 if [[ ! -x "${CYLC_HOME}/bin/${0##*/}" ]]; then
     echo 1>&2 "ERROR: ${0##*/} not found in ${CYLC_HOME}"
     exit 1
 fi
+
 # Set PATH when running cylc hub so that configurable-http-proxy can find node
-if [[ ${0##*/} =~ ^cylc && ${1:-} == "hub" && ! ${CYLC_VERSION:-} =~ ^7 ]]; then
-  PATH=${CYLC_HOME}/bin:${PATH}
+if [[ ${0##*/} == "cylc" && ${1:-} == "hub" ]]; then
+    PATH=${CYLC_HOME}/bin:${PATH}
 fi
 
-# Execute Cylc from the selected installation.
-# n.b. If CYLC_HOME points to a cylc binary inside a
-# ``conda/envs/bin directory``this is sufficent without using
+# Run the executable from the selected installation.
+# n.b. If CYLC_HOME points to a binary inside a
+# ``conda/envs/bin directory`` this is sufficent without using
 # ``conda activate``. We avoid ``conda activate`` because:
 # - It's slow.
 # - Subsequent environment changes (conda activate or deactivate)