Merge pull request #3181 from radical-cybertools/fix/resource_cfg_def…

…aults

cleanup defaults for virtenv setup

docs/source/tutorials/configuration.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -161,9 +160,9 @@
     "        \"default_remote_workdir\"      : \"$HOME\",\n",
     "\n",
     "        \"pre_bootstrap_0\"             : [\n",
-    "                                        \"module unload intel impi\",\n",
-    "                                        \"module load   intel impi\",\n",
-    "                                        \"module load   python3/3.9.2\"\n",
+    "                                         \"module unload intel impi\",\n",
+    "                                         \"module load   intel impi\",\n",
+    "                                         \"module load   python3/3.9.2\"\n",
     "                                        ],\n",
     "        \"launch_methods\"              : {\n",
     "                                         \"order\"  : [\"MPIRUN\"],\n",
@@ -174,14 +173,12 @@
     "                                         }\n",
     "                                        },\n",
     "        \n",
-    "        \"python_dist\"                 : \"default\",\n",
     "        \"virtenv_mode\"                : \"local\"\n",
     "    }\n",
     "}"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -194,8 +191,8 @@
     "* __filesystem_endpoint__ - access URL for file staging (interpreted by RADICAL-SAGA).\n",
     "* __default_queue__ (_optional_) - queue name to be used for pilot submission to a corresponding batch system (see __job_manager_endpoint__).\n",
     "* __resource_manager__ - the type of job management system. Valid values are: `CCM`, `COBALT`, `FORK`, `LSF`, `PBSPRO`, `SLURM`, `TORQUE`, `YARN`.\n",
-    "* __cores_per_node__ (_optional_) - number of available CPU cores per compute node. If not provided then it will be discovered by RADICAL-SAGA and by Resource Manager in RADICAL-Pilot.\n",
-    "* __gpus_per_node__ (_optional_) - number of available GPUs per compute node. If not provided then it will be discovered by RADICAL-SAGA and by Resource Manager in RADICAL-Pilot.\n",
+    "* __cores_per_node__ (_optional_) - number of available CPU cores per compute node. If not provided then it will be discovered by Resource Manager in RADICAL-Pilot.\n",
+    "* __gpus_per_node__ (_optional_) - number of available GPUs per compute node. If not provided then it will be discovered by Resource Manager in RADICAL-Pilot.\n",
     "* __system_architecture__ (_optional_) - set of options that describe platform features:\n",
     "   * __smt__ - Simultaneous MultiThreading (i.e., threads per physical core). If it is not provided then the default value `1` is used. It could be reset with env variable `RADICAL_SMT` exported before running RADICAL-Pilot application. RADICAL-Pilot uses `cores_per_node x smt` to calculate all available cores/CPUs per node.\n",
     "   * __options__ - list of job management system specific attributes/constraints, which are provided to RADICAL-SAGA.\n",
@@ -215,15 +212,15 @@
     "* __launch_methods__ - set of supported launch methods. Valid values are `APRUN`, `CCMRUN`, `FLUX`, `FORK`, `IBRUN`, `JSRUN` (`JSRUN_ERF`), `MPIEXEC` (`MPIEXEC_MPT`), `MPIRUN` (`MPIRUN_CCMRUN`, `MPIRUN_DPLACE`, `MPIRUN_MPT`, `MPIRUN_RSH`), `PRTE`, `RSH`, `SRUN`, `SSH`. For each launch method, a subsection is needed, which specifies __pre_exec_cached__ with list of commands to be executed to configure the launch method, and method related options (e.g., __dvm_count__ for `PRTE`).\n",
     "   * __order__ - sets the order of launch methods to be selected for the task placement (the first value in the list is a default launch method).\n",
     "* __python_dist__ - python distribution. Valid values are `default` and `anaconda`.\n",
-    "* __virtenv_mode__ - bootstrapping process set the environment for RADICAL-Pilot Agent:\n",
+    "* __virtenv_mode__ - bootstrapping process set the environment for RADICAL-Pilot Agent (default value is `local`):\n",
     "   * `create` - create a python virtual environment from scratch;\n",
     "   * `recreate` - delete the existing virtual environment and build it from scratch, if not found then `create`;\n",
     "   * `use` - use the existing virtual environment, if not found then `create`;\n",
-    "   * `update` - update the existing virtual environment, if not found then `create` (_default_);\n",
+    "   * `update` - update the existing virtual environment, if not found then `create`;\n",
     "   * `local` - use the client existing virtual environment (environment from where RADICAL-Pilot application was launched).\n",
     "* __virtenv__ (_optional_) - path to the existing virtual environment or its name with the pre-installed RCT stack; use it only when `virtenv_mode=use`.\n",
-    "* __rp_version__ - RADICAL-Pilot installation or reuse process:\n",
-    "   * `local` - install from tarballs, from client existing environment (_default_);\n",
+    "* __rp_version__ - RADICAL-Pilot installation or reuse process (default value is `installed`):\n",
+    "   * `local` - install from tarballs, from client existing environment;\n",
     "   * `release` - install the latest released version from PyPI;\n",
     "   * `installed` - do not install, target virtual environment has it.\n",
     "\n",
@@ -268,7 +265,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -329,7 +325,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -367,7 +362,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -380,7 +374,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "session.close(cleanup=True)"
+    "session.close()"
    ]
   }
  ],
@@ -400,7 +394,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.9.13"
   }
  },
  "nbformat": 4,

src/radical/pilot/pmgr/launching/base.py

@@ -15,7 +15,6 @@
 
 import threading          as mt
 
-import radical.gtod       as rg
 import radical.utils      as ru
 
 from ... import states    as rps
@@ -31,14 +30,6 @@
 RP_UL_NAME_PSI_J = "PSI_J"
 
 
-# ------------------------------------------------------------------------------
-# local constants
-DEFAULT_AGENT_SPAWNER = 'POPEN'
-DEFAULT_RP_VERSION    = 'local'
-DEFAULT_VIRTENV_MODE  = 'update'
-DEFAULT_AGENT_CONFIG  = 'default'
-
-
 # ------------------------------------------------------------------------------
 #
 class PilotLauncherBase(object):
@@ -718,19 +709,9 @@ def _prepare_pilot(self, resource, rcfg, pilot, expand, tar_name):
         # above syntax is ignored, and the fallback stage@local
         # is used.
 
-        if not rp_version:
-            if virtenv_mode == 'local': rp_version = 'installed'
-            else                      : rp_version = DEFAULT_RP_VERSION
-
-        if not rp_version.startswith('@') and \
-               rp_version not in ['installed', 'local', 'release']:
-            raise ValueError("invalid rp_version '%s'" % rp_version)
-
-        if rp_version.startswith('@'):
-            rp_version  = rp_version[1:]  # strip '@'
-
         # use local VE ?
         if virtenv_mode == 'local':
+            rp_version = 'installed'
             if os.environ.get('VIRTUAL_ENV'):
                 python_dist = 'default'
                 virtenv     = os.environ['VIRTUAL_ENV']
@@ -742,6 +723,13 @@ def _prepare_pilot(self, resource, rcfg, pilot, expand, tar_name):
                 self._log.error('virtenv_mode is local, no local env found')
                 raise ValueError('no local env found')
 
+        if not rp_version.startswith('@') and \
+               rp_version not in ['installed', 'local', 'release']:
+            raise ValueError("invalid rp_version '%s'" % rp_version)
+
+        if rp_version.startswith('@'):
+            rp_version  = rp_version[1:]  # strip '@'
+
         # ----------------------------------------------------------------------
         # sanity checks
         RE = RuntimeError

src/radical/pilot/resource_config.py

@@ -23,7 +23,6 @@
 RP_VERSION             = 'rp_version'
 VIRTENV                = 'virtenv'
 VIRTENV_MODE           = 'virtenv_mode'
-VIRTENV_DIST           = 'virtenv_dist'
 PYTHON_DIST            = 'python_dist'
 PYTHON_INTERPRETER     = 'python_interpreter'
 LAUNCH_METHODS         = 'launch_methods'
@@ -121,7 +120,6 @@ class ResourceConfig(ru.TypedDict):
         RP_VERSION             : str         ,
         VIRTENV                : str         ,
         VIRTENV_MODE           : str         ,
-        VIRTENV_DIST           : str         ,
         PYTHON_DIST            : str         ,
         PYTHON_INTERPRETER     : str         ,
         LAUNCH_METHODS         : {str: None} ,
@@ -166,10 +164,9 @@ class ResourceConfig(ru.TypedDict):
         AGENT_SPAWNER          : 'POPEN'     ,
         PRE_BOOTSTRAP_0        : list()      ,
         PRE_BOOTSTRAP_1        : list()      ,
-        RP_VERSION             : 'local'     ,
+        RP_VERSION             : 'installed' ,
         VIRTENV                : ''          ,
-        VIRTENV_MODE           : 'update'    ,
-        VIRTENV_DIST           : ''          ,
+        VIRTENV_MODE           : 'local'     ,
         PYTHON_DIST            : 'default'   ,
         PYTHON_INTERPRETER     : ''          ,
         LAUNCH_METHODS         : dict()      ,