[feat] Introduce environment variables for controlling system auto-detection

docs/configure.rst

@@ -235,12 +235,15 @@ Picking a System Configuration
 
 As discussed previously, ReFrame's configuration file can store the configurations for multiple systems.
 When launched, ReFrame will pick the first matching configuration and load it.
-This process is performed as follows:
-ReFrame first tries to obtain the hostname from ``/etc/xthostname``, which provides the unqualified *machine name* in Cray systems.
-If this cannot be found, the hostname will be obtained from the standard ``hostname`` command.
-Having retrieved the hostname, ReFrame goes through all the systems in its configuration and tries to match the hostname against any of the patterns defined in each system's ``hostnames`` property.
+
+ReFrame uses an auto-detection mechanism to get information about the host it is running on and uses that information to pick the right system configuration.
+Currently, only one auto-detection method is supported that retrieves the hostname.
+Based on this, ReFrame goes through all the systems in its configuration and tries to match the hostname against any of the patterns defined in each system's ``hostnames`` property.
 The detection process stops at the first match found, and that system's configuration is selected.
 
+The auto-detection process can be controlled through the :envvar:`RFM_AUTODETECT_METHOD`, :envvar:`RFM_AUTODETECT_FQDN` and :envvar:`RFM_AUTODETECT_XTHOSTNAME` environment variables.
+
+
 As soon as a system configuration is selected, all configuration objects that have a ``target_systems`` property are resolved against the selected system, and any configuration object that is not applicable is dropped.
 So, internally, ReFrame keeps an *instantiation* of the site configuration for the selected system only.
 To better understand this, let's assume that we have the following ``environments`` defined:

docs/manpage.rst

@@ -784,7 +784,6 @@ Miscellaneous options
    If this option is not specified, ReFrame will try to pick the correct configuration entry automatically.
    It does so by trying to match the hostname of the current machine again the hostname patterns defined in the :js:attr:`hostnames` system configuration parameter.
    The system with the first match becomes the current system.
-   For Cray systems, ReFrame will first look for the *unqualified machine name* in ``/etc/xthostname`` before trying retrieving the hostname of the current machine.
 
    This option can also be set using the :envvar:`RFM_SYSTEM` environment variable.
 
@@ -941,6 +940,64 @@ Boolean environment variables can have any value of ``true``, ``yes``, ``y`` (ca
 Here is an alphabetical list of the environment variables recognized by ReFrame:
 
 
+.. envvar:: RFM_AUTODETECT_FQDN
+
+   Use the fully qualified domain name as the hostname.
+   This is a boolean variable and defaults to ``1``.
+
+
+   .. table::
+      :align: left
+
+      ================================== ==================
+      Associated command line option     N/A
+      Associated configuration parameter N/A
+      ================================== ==================
+
+
+   .. versionadded:: 3.11.0
+
+
+.. envvar:: RFM_AUTODETECT_METHOD
+
+   Method to use for detecting the current system and pick the right configuration.
+   The following values can be used:
+
+   - ``hostname``: The ``hostname`` command will be used to detect the current system.
+     This is the default value, if not specified.
+
+   .. table::
+      :align: left
+
+      ================================== ==================
+      Associated command line option     N/A
+      Associated configuration parameter N/A
+      ================================== ==================
+
+
+   .. versionadded:: 3.11.0
+
+
+.. envvar:: RFM_AUTODETECT_XTHOSTNAME
+
+   Use ``/etc/xthostname`` file, if present, to retrieve the current system's name.
+   If the file cannot be found, the hostname will be retrieved using the ``hostname`` command.
+   This is a boolean variable and defaults to ``1``.
+
+   This option meaningful for Cray systems.
+
+   .. table::
+      :align: left
+
+      ================================== ==================
+      Associated command line option     N/A
+      Associated configuration parameter N/A
+      ================================== ==================
+
+
+   .. versionadded:: 3.11.0
+
+
 .. envvar:: RFM_CHECK_SEARCH_PATH
 
    A colon-separated list of filesystem paths where ReFrame should search for tests.

reframe/core/config.py

@@ -60,13 +60,40 @@ def _get(site_config, option, *args, **kwargs):
     return _do_normalize
 
 
+def _hostname(use_fqdn, use_xthostname):
+    '''Return hostname'''
+    if use_xthostname:
+        try:
+            xthostname_file = '/etc/xthostname'
+            getlogger().debug(f'Trying {xthostname_file!r}...')
+            with open(xthostname_file) as fp:
+                return fp.read()
+        except OSError as e:
+            '''Log the error and continue to the next method'''
+            getlogger().debug(f'Failed to read {xthostname_file!r}')
+
+    if use_fqdn:
+        getlogger().debug('Using FQDN...')
+        return socket.getfqdn()
+
+    getlogger().debug('Using standard hostname...')
+    return socket.gethostname()
+
+
 class _SiteConfig:
     def __init__(self, site_config, filename):
         self._site_config = copy.deepcopy(site_config)
         self._filename = filename
         self._subconfigs = {}
         self._local_system = None
         self._sticky_options = {}
+        self._autodetect_meth = 'hostname'
+        self._autodetect_opts = {
+            'hostname': {
+                'use_fqdn': False,
+                'use_xthostname': False,
+            }
+        }
 
         # Open and store the JSON schema for later validation
         schema_filename = os.path.join(reframe.INSTALL_PREFIX, 'reframe',
@@ -104,6 +131,15 @@ def __getitem__(self, key):
     def __getattr__(self, attr):
         return getattr(self._pick_config(), attr)
 
+    def set_autodetect_meth(self, method, **opts):
+        self._autodetect_meth = method
+        try:
+            self._autodetect_opts[method].update(opts)
+        except KeyError:
+            raise ConfigError(
+                f'unknown auto-detection method: {method!r}'
+            ) from None
+
     @property
     def schema(self):
         '''Configuration schema'''
@@ -263,21 +299,15 @@ def _create_from_json(cls, filename):
         return _SiteConfig(config, filename)
 
     def _detect_system(self):
-        getlogger().debug('Detecting system')
-        if os.path.exists('/etc/xthostname'):
-            # Get the cluster name on Cray systems
-            getlogger().debug(
-                "Found '/etc/xthostname': will use this to get the system name"
-            )
-            with open('/etc/xthostname') as fp:
-                hostname = fp.read()
-        else:
-            hostname = socket.getfqdn()
-
         getlogger().debug(
-            f'Looking for a matching configuration entry '
-            f'for system {hostname!r}'
+            f'Detecting system using method: {self._autodetect_meth!r}'
+        )
+        hostname = _hostname(
+            self._autodetect_opts[self._autodetect_meth]['use_fqdn'],
+            self._autodetect_opts[self._autodetect_meth]['use_xthostname'],
         )
+        getlogger().debug(f'Retrieved hostname: {hostname!r}')
+        getlogger().debug(f'Looking for a matching configuration entry')
         for system in self._site_config['systems']:
             for patt in system['hostnames']:
                 if re.match(patt, hostname):

reframe/frontend/argparse.py

@@ -82,7 +82,7 @@ def __getattr__(self, name):
         if name not in self.__option_map:
             return ret
 
-        envvar, _, action, arg_type = self.__option_map[name]
+        envvar, _, action, arg_type, default = self.__option_map[name]
         if ret is None and envvar is not None:
             # Try the environment variable
             envvar, *delim = envvar.split(maxsplit=2)
@@ -107,6 +107,8 @@ def __getattr__(self, name):
                             f'cannot convert environment variable {envvar!r} '
                             f'to {arg_type.__name__!r}'
                         ) from err
+            else:
+                ret = default
 
         return ret
 
@@ -183,7 +185,8 @@ def add_argument(self, *flags, **kwargs):
             kwargs.get('envvar', None),
             kwargs.get('configvar', None),
             kwargs.get('action', 'store'),
-            kwargs.get('type', str)
+            kwargs.get('type', str),
+            kwargs.get('default', None)
         )
         # Remove envvar and configvar keyword arguments and force dest
         # argument, even if we guessed it, in order to guard against changes

reframe/frontend/cli.py

@@ -27,6 +27,7 @@
 import reframe.frontend.runreport as runreport
 import reframe.utility.jsonext as jsonext
 import reframe.utility.osext as osext
+import reframe.utility.typecheck as typ
 
 
 from reframe.frontend.printer import PrettyPrinter
@@ -523,6 +524,29 @@ def main():
     )
 
     # Options not associated with command-line arguments
+    argparser.add_argument(
+        dest='autodetect_fqdn',
+        envvar='RFM_AUTODETECT_FQDN',
+        action='store',
+        default=True,
+        type=typ.Bool,
+        help='Use FQDN as host name'
+    )
+    argparser.add_argument(
+        dest='autodetect_method',
+        envvar='RFM_AUTODETECT_METHOD',
+        action='store',
+        default='hostname',
+        help='Method to detect the system'
+    )
+    argparser.add_argument(
+        dest='autodetect_xthostname',
+        envvar='RFM_AUTODETECT_XTHOSTNAME',
+        action='store',
+        default=True,
+        type=typ.Bool,
+        help="Use Cray's xthostname file to find the host name"
+    )
     argparser.add_argument(
         dest='git_timeout',
         envvar='RFM_GIT_TIMEOUT',
@@ -694,6 +718,11 @@ def restrict_logging():
             site_config = config.load_config(converted)
 
         site_config.validate()
+        site_config.set_autodetect_meth(
+            options.autodetect_method,
+            use_fqdn=options.autodetect_fqdn,
+            use_xthostname=options.autodetect_xthostname
+        )
 
         # We ignore errors about unresolved sections or configuration
         # parameters here, because they might be defined at the individual
@@ -877,7 +906,7 @@ def print_infoline(param, value):
         'cmdline': ' '.join(sys.argv),
         'config_file': rt.site_config.filename,
         'data_version': runreport.DATA_VERSION,
-        'hostname': socket.getfqdn(),
+        'hostname': socket.gethostname(),
         'prefix_output': rt.output_prefix,
         'prefix_stage': rt.stage_prefix,
         'user': osext.osuser(),

reframe/schemas/config.json

@@ -457,7 +457,6 @@
                     "compact_test_names": {"type": "boolean"},
                     "git_timeout": {"type": "number"},
                     "ignore_check_conflicts": {"type": "boolean"},
-                    "trap_job_errors": {"type": "boolean"},
                     "keep_stage_files": {"type": "boolean"},
                     "module_map_file": {"type": "string"},
                     "module_mappings": {
@@ -476,6 +475,7 @@
                     "save_log_files": {"type": "boolean"},
                     "target_systems": {"$ref": "#/defs/system_ref"},
                     "timestamp_dirs": {"type": "string"},
+                    "trap_job_errors": {"type": "boolean"},
                     "unload_modules": {"$ref": "#/defs/modules_list"},
                     "use_login_shell": {"type": "boolean"},
                     "user_modules": {"$ref": "#/defs/modules_list"},
@@ -509,7 +509,6 @@
         "general/compact_test_names": false,
         "general/git_timeout": 5,
         "general/ignore_check_conflicts": false,
-        "general/trap_job_errors": false,
         "general/keep_stage_files": false,
         "general/module_map_file": "",
         "general/module_mappings": [],
@@ -523,6 +522,7 @@
         "general/save_log_files": false,
         "general/target_systems": ["*"],
         "general/timestamp_dirs": "",
+        "general/trap_job_errors": false,
         "general/unload_modules": [],
         "general/use_login_shell": false,
         "general/user_modules": [],

unittests/test_argparser.py

@@ -107,6 +107,14 @@ def extended_parser():
         '--git-timeout', envvar='RFM_GIT_TIMEOUT', action='store',
         configvar='general/git_timeout', type=float
     )
+
+    # Option that is only associated with an environment variable
+    parser.add_argument(
+        dest='env_option',
+        envvar='RFM_ENV_OPT',
+        action='store',
+        default='bar'
+    )
     foo_options.add_argument(
         '--timestamp', action='store',
         envvar='RFM_TIMESTAMP_DIRS', configvar='general/timestamp_dirs'
@@ -187,3 +195,14 @@ def test_option_envvar_conversion_error(default_exec_ctx, extended_parser):
         options = extended_parser.parse_args(['--nocolor'])
         errors = options.update_config(site_config)
         assert len(errors) == 2
+
+
+def test_envvar_option(default_exec_ctx, extended_parser):
+    with rt.temp_environment(variables={'RFM_ENV_OPT': 'BAR'}):
+        options = extended_parser.parse_args([])
+        assert options.env_option == 'BAR'
+
+
+def test_envvar_option_default_val(default_exec_ctx, extended_parser):
+    options = extended_parser.parse_args([])
+    assert options.env_option == 'bar'

unittests/test_config.py

@@ -387,3 +387,17 @@ def test_system_create():
     assert partition.processor.num_cores_per_socket == 4
     assert partition.processor.num_numa_nodes == 1
     assert partition.processor.num_cores_per_numa_node == 4
+
+
+def test_hostname_autodetection():
+    # This exercises only the various execution paths
+
+    # We set the autodetection method and we call `select_subconfig()` in
+    # order to trigger the auto-detection
+    site_config = config.load_config('unittests/resources/settings.py')
+    for use_xthostname in (True, False):
+        for use_fqdn in (True, False):
+            site_config.set_autodetect_meth('hostname',
+                                            use_fqdn=use_fqdn,
+                                            use_xthostname=use_xthostname)
+            site_config.select_subconfig()