diff --git a/doc/ref/configuration/minion.rst b/doc/ref/configuration/minion.rst index f53a0e2ef83f..0e135ddb2a96 100644 --- a/doc/ref/configuration/minion.rst +++ b/doc/ref/configuration/minion.rst @@ -1174,8 +1174,73 @@ environments is to isolate via the top file. .. code-block:: yaml - environment: None + environment: dev +.. conf_minion:: top_file_merging_strategy + +``top_file_merging_strategy`` +----------------------------- + +Default: ``merge`` + +When no specific fileserver environment (a.k.a. ``saltenv``) has been specified +for a :ref:`highstate `, all environments' top files are +inspected. This config option determines how the SLS targets in those top files +are handled. + +When set to ``merge``, the targets for all SLS files in all environments are +merged together. A given environment's SLS targets for the highstate will +consist of the collective SLS targets specified for that environment in all top +files. The environments will be merged in no specific order, for greater +control over the order in which the environments are merged use +:conf_minion:`env_order`. + +When set to ``same``, then for each environment, only that environment's top +file is processed, with the others being ignored. For example, only the ``dev`` +environment's top file will be processed for the ``dev`` environment, and any +SLS targets defined for ``dev`` in the ``base`` environment's (or any other +environment's) top file will be ignored. If an environment does not have a top +file, then the top file from the :conf_minion:`default_top` config parameter +will be used as a fallback. + +.. code-block:: yaml + + top_file_merging_strategy: same + +.. conf_minion:: env_order + +``env_order`` +------------- + +Default: ``[]`` + +When :conf_minion:`top_file_merging_strategy` is set to ``merge``, and no +environment is specified for a :ref:`highstate `, this +config option allows for the order in which top files are merged to be +explicitly defined. + +.. code-block:: yaml + + env_order: + - base + - dev + - qa + +.. conf_minion:: default_top + +``default_top`` +--------------- + +Default: ``base`` + +When :conf_minion:`top_file_merging_strategy` is set to ``same``, and no +environment is specified for a :ref:`highstate `, this +config option specifies a fallback environment in which to look for a top file +if an environment lacks one. + +.. code-block:: yaml + + default_top: dev File Directory Settings ======================= diff --git a/doc/ref/states/top.rst b/doc/ref/states/top.rst index caa54084dc8d..6bea5acfa733 100644 --- a/doc/ref/states/top.rst +++ b/doc/ref/states/top.rst @@ -30,22 +30,22 @@ A Basic Example Top files have three components: - - Environment: A state tree directory containing a set of state files to - configure systems. +- **Environment:** A state tree directory containing a set of state files to + configure systems. - - Target: A grouping of machines which will have a set of states applied - to them. +- **Target:** A grouping of machines which will have a set of states applied to + them. - - State files: A list of state files to apply to a target. Each state - file describes one or more states to be configured and enforced - on the targeted machines. +- **State files:** A list of state files to apply to a target. Each state file + describes one or more states to be configured and enforced on the targeted + machines. The relationship between these three components is nested as follows: - - Environments contain targets +- Environments contain targets - - Targets contain states +- Targets contain states Putting these concepts together, we can describe a scenario in which all @@ -172,26 +172,28 @@ Choosing an Environment to Target The top file is used to assign a minion to an environment unless overridden using the methods described below. The environment in the top file must match -an environment in :conf_master:`file_roots` in order for any states to be -applied to that minion. The states that will be applied to a minion in a given -environment can be viewed using the :py:func:`state.show_top -` execution function. - -Minions may be pinned to a particular environment by setting the ``environment`` -value in the minion configuration file. In doing so, a minion will only -request files from the environment to which it is assigned. - -The environment to use may also be dynamically selected at the time that -a ``salt``, ``salt-call`` or ``salt-ssh`` by passing passing a flag to the -execution module being called. This is most commonly done with -functions in the ``state`` module by using the ``saltenv=`` argument. For -example, to run a ``highstate`` on all minions, using the state files in -the ``prod`` state tree, run: ``salt '*' state.highstate saltenv=prod``. +valid fileserver environment (a.k.a. ``saltenv``) in order for any states to be +applied to that minion. When using the default fileserver backend, environments +are defined in :conf_master:`file_roots`. -.. note:: - Not all functions accept ``saltenv`` as an argument See individual - function documentation to verify. +The states that will be applied to a minion in a given environment can be +viewed using the :py:func:`state.show_top ` +function. + +Minions may be pinned to a particular environment by setting the +:conf_minion:`environment` value in the minion configuration file. In doing so, +a minion will only request files from the environment to which it is assigned. +The environment may also be dynamically selected at runtime by passing it to +the ``salt``, ``salt-call`` or ``salt-ssh`` command. This is most commonly done +with functions in the ``state`` module by using the ``saltenv`` argument. For +example, to run a ``highstate`` on all minions, using only the top file and SLS +files in the ``prod`` environment, run: ``salt '*' state.highstate +saltenv=prod``. + +.. note:: + Not all functions accept ``saltenv`` as an argument, see the documentation + for an individual function documentation to verify. Shorthand @@ -305,66 +307,132 @@ of matches you can perform: How Top Files Are Compiled ========================== +When a :ref:`highstate ` is executed and an environment is +specified (either using the :conf_minion:`environment` config option or by +passing the saltenv when executing the :ref:`highstate `), +then that environment's top file is the only top file used to assign states to +minions, and only states from the specified environment will be run. + +The remainder of this section applies to cases in which a :ref:`highstate +` is executed without an environment specified. + +With no environment specified, the minion will look for a top file in each +environment, and each top file will be processed to determine the SLS files to +run on the minions. By default, the top files from each environment will be +merged together. In configurations with many environments, such as with +:ref:`GitFS ` where each branch and tag is treated as a +distinct environment, this may cause unexpected results as SLS files from older +tags cause defunct SLS files to be included in the highstate. In cases like +this, it can be helpful to set :conf_minion:`top_file_merging_strategy` to +``same`` to force each environment to use its own top file. + +.. code-block:: yaml + + top_file_merging_strategy: same + +With :ref:`GitFS `, it can also be helpful to simply manage +each environment's top file separately, and/or manually specify the environment +when executing the highstate to avoid any complicated merging scenarios. +:conf_master:`gitfs_env_whitelist` and :conf_master:`gitfs_env_blacklist` can +also be used to hide unneeded branches and tags from GitFS to reduce the number +of top files in play. + When using multiple environments, it is not necessary to create a top file for -each environment. The most common approach, and the easiest to maintain, is -to use a single top file placed in only one environment. +each environment. The easiest-to-maintain approach is to use a single top file +placed in the ``base`` environment. This is often infeasible with :ref:`GitFS +` though, since branching/tagging can easily result in extra +top files. However, when only the default (``roots``) fileserver backend is +used, a single top file in the ``base`` environment is the most common way of +configuring a :ref:`highstate `. + +The following minion configuration options affect how top files are compiled +when no environment is specified: -However, some workflows do call for multiple top files. In this case, top -files may be merged together to create ``high data`` for the state compiler -to use as a source to compile states on a minion. +- :conf_minion:`top_file_merging_strategy` +- :conf_minion:`env_order` +- :conf_minion:`default_top` -For the following discussion of top file compilation, assume the following -configuration: +Top File Compilation Examples +============================= +For the scenarios below, assume the following configuration: -``/etc/salt/master``: +**/etc/salt/master**: .. code-block:: yaml - file_roots: - first_env: - - /srv/salt/first - second_env: - - /srv/salt/second - + base: + - /srv/salt/base + dev: + - /srv/salt/dev + qa: + - /srv/salt/qa -``/srv/salt/first/top.sls``: +**/srv/salt/base/top.sls**: .. code-block:: yaml - first_env: + base: + '*': + - base1 + dev: + '*': + - dev1 + qa: '*': - - first - second_env: + - qa1 + +**/srv/salt/dev/top.sls**: + +.. code-block:: yaml + + base: + 'minion1': + - base2 + dev: + 'minion2': + - dev2 + qa: '*': - - second + - qa2 + +.. note:: + For the purposes of these examples, there is no top file in the ``qa`` + environment. + +Scenario 1 - ``dev`` Environment Specified +------------------------------------------ + +In this scenario, the :ref:`highstate ` was either invoked +with ``saltenv=dev`` or the minion has ``environment: dev`` set in the minion +config file. The result will be that only the ``dev2`` SLS from the dev +environment will be part of the :ref:`highstate `, and it +will be applied to minion2, while minion1 will have no states applied to it. -The astute reader will ask how the state compiler resolves which should be -an obvious conflict if a minion is not pinned to a particular environment -and if no environment argument is passed into a state function. +If the ``base`` environment were specified, the result would be that only the +``base1`` SLS from the ``base`` environment would be part of the +:ref:`highstate `, and it would be applied to all minions. -Given the above, it is initially unclear whether ``first.sls`` will be applied -or whether ``second.sls`` will be applied in a ``salt '*' state.highstate`` command. +If the ``qa`` environment were specified, the :ref:`highstate +` would exit with an error. -When conflicting keys arise, there are several configuration options which -control the behaviour of salt: +Scenario 2 - No Environment Specified, :conf_minion:`top_file_merging_strategy` is "merge" +------------------------------------------------------------------------------------------ - - ``env_order`` - Setting ``env_order`` will set the order in which environments are processed - by the state compiler. +In this scenario, ``base1`` from the ``base`` environment, ``dev1`` from the +``dev`` environment, and ``qa1`` from the ``qa`` environment are applied to all +minions. Additionally, ``base2`` from the ``base`` environment is applied to +minion1, and ``dev2`` from the ``dev`` environment is applied to minion2. - - ``top_file_merging_strategy`` - Can be set to ``same``, which will process only the top file from the environment - that the minion belongs to via the ``environment`` configuration setting or - the environment that is requested via the ``saltenv`` argument supported - by some functions in the ``state`` module. +Scenario 3 - No Environment Specified, :conf_minion:`top_file_merging_strategy` is "same" +----------------------------------------------------------------------------------------- - Can also be set to ``merge``. This is the default. When set to ``merge``, - top files will be merged together. The order in which top files are - merged together can be controlled with ``env_order``. +In this scenario, ``base1`` from the ``base`` environment is applied to all +minions. Additionally, ``dev2`` from the ``dev`` environment is applied to +minion2. - - ``default_top`` - If ``top_file_merging_strategy`` is set to ``same`` and an environment does - not contain a top file, the top file in the environment specified by - ``default_top`` will be used instead. +If :conf_minion:`default_top` is unset (or set to ``base``, which happens to be +the default), then ``qa1`` from the ``qa`` environment will be applied to all +minions. If :conf_minion:`default_top` were set to ``dev``, then ``qa2`` from +the ``qa`` environment would be applied to all minions. diff --git a/doc/topics/tutorials/gitfs.rst b/doc/topics/tutorials/gitfs.rst index 0cdf53e6bffe..f9612859fa15 100644 --- a/doc/topics/tutorials/gitfs.rst +++ b/doc/topics/tutorials/gitfs.rst @@ -15,6 +15,11 @@ configuring one or more repositories in :conf_master:`gitfs_remotes`. Branches and tags become Salt fileserver environments. +.. note:: + Branching and tagging can result in a lot of potentially-conflicting + :ref:`top files `, for this reason it may be useful to set + :conf_minion:`top_file_merging_strategy` to ``same`` in the minions' config + files if the top files are being managed in a GitFS repo. .. _gitfs-dependencies: diff --git a/doc/topics/tutorials/salt_bootstrap.rst b/doc/topics/tutorials/salt_bootstrap.rst index 3adb784ef1a5..c430c09b25f0 100644 --- a/doc/topics/tutorials/salt_bootstrap.rst +++ b/doc/topics/tutorials/salt_bootstrap.rst @@ -164,7 +164,7 @@ Downloading the script from develop branch: .. code-block:: bash - wget https://bootstrap.saltstack.com/develop + wget -O bootstrap_salt.sh https://bootstrap.saltstack.com/develop sudo sh bootstrap_salt.sh Installing a specific version from git using ``wget``: diff --git a/salt/loader.py b/salt/loader.py index 37ccccec9130..015c8bdc89eb 100644 --- a/salt/loader.py +++ b/salt/loader.py @@ -1364,7 +1364,7 @@ def _load_module(self, name): module_name, ) if virtual_err is not None: - log.debug('Error loading {0}.{1}: {2}'.format(self.tag, + log.trace('Error loading {0}.{1}: {2}'.format(self.tag, module_name, virtual_err, )) diff --git a/salt/modules/dnsmasq.py b/salt/modules/dnsmasq.py index cf029ffae7c2..914ad24e4407 100644 --- a/salt/modules/dnsmasq.py +++ b/salt/modules/dnsmasq.py @@ -118,12 +118,16 @@ def get_config(config_file='/etc/dnsmasq.conf'): ''' Dumps all options from the config file. + config_file + The location of the config file from which to obtain contents. + Defaults to ``/etc/dnsmasq.conf``. + CLI Examples: .. code-block:: bash salt '*' dnsmasq.get_config - salt '*' dnsmasq.get_config file=/etc/dnsmasq.conf + salt '*' dnsmasq.get_config config_file=/etc/dnsmasq.conf ''' dnsopts = _parse_dnamasq(config_file) if 'conf-dir' in dnsopts: diff --git a/salt/modules/glusterfs.py b/salt/modules/glusterfs.py index 6f293892f576..c44d686e4ffe 100644 --- a/salt/modules/glusterfs.py +++ b/salt/modules/glusterfs.py @@ -76,7 +76,11 @@ def _gluster_xml(cmd): result = __salt__['cmd.run']( 'gluster --xml --mode=script', stdin="{0}\n".format(cmd) ) - root = ET.fromstring(_gluster_output_cleanup(result)) + + try: + root = ET.fromstring(_gluster_output_cleanup(result)) + except ET.ParseError: + raise CommandExecutionError('\n'.join(result.splitlines()[:-1])) if int(root.find('opRet').text) != 0: raise CommandExecutionError(root.find('opErrstr').text) diff --git a/salt/state.py b/salt/state.py index 3b339404f5f3..d63f28db5580 100644 --- a/salt/state.py +++ b/salt/state.py @@ -2560,9 +2560,14 @@ def get_tops(self): tops[saltenv].append({}) log.debug('No contents loaded for env: {0}'.format(saltenv)) if found > 1: - log.warning('Top file merge strategy set to \'merge\' and multiple top files found. ' - 'Top file merging order is undefined; ' - 'for better results use \'same\' option') + log.warning( + 'top_file_merging_strategy is set to \'merge\' and ' + 'multiple top files were found. Merging order is not ' + 'deterministic, it may be desirable to either set ' + 'top_file_merging_strategy to \'same\' or use the ' + '\'env_order\' configuration parameter to specify the ' + 'merging order.' + ) if found == 0: log.error('No contents found in top file') diff --git a/tests/integration/states/network.py b/tests/integration/states/network.py index 9925b36861f0..2cded4687950 100644 --- a/tests/integration/states/network.py +++ b/tests/integration/states/network.py @@ -10,29 +10,23 @@ # Salt libs import integration -import salt.utils # Salttesting libs -from salttesting import skipIf from salttesting.helpers import destructiveTest, ensure_in_syspath ensure_in_syspath('../../') -def _check_arch_linux(): - with salt.utils.fopen('/etc/os-release', 'r') as f: - release = f.readline() - r = release.split('=')[1].strip().strip('"') - return r - - @destructiveTest -@skipIf(_check_arch_linux() == 'Arch Linux', 'Network state not supported on Arch') class NetworkTest(integration.ModuleCase, integration.SaltReturnAssertsMixIn): ''' Validate network state module ''' def setUp(self): + os_family = self.run_function('grains.get', ['os_family']) + if os_family not in ('RedHat', 'Debian'): + self.skipTest('Network state only supported on RedHat and Debian based systems') + self.run_function('cmd.run', ['ip link add name dummy0 type dummy']) def tearDown(self): diff --git a/tests/unit/utils/args.py b/tests/unit/utils/args_test.py similarity index 100% rename from tests/unit/utils/args.py rename to tests/unit/utils/args_test.py diff --git a/tests/unit/utils/immutabletypes.py b/tests/unit/utils/immutabletypes_test.py similarity index 100% rename from tests/unit/utils/immutabletypes.py rename to tests/unit/utils/immutabletypes_test.py