Permalink
Browse files

Merge pull request #35822 from terminalmage/issue35045

Fixes for top file merging
  • Loading branch information...
2 parents 135c403 + f2f4482 commit f7d8f408c27be64e35d2b9e72b60e2e026888151 @cachedout cachedout committed on GitHub Aug 30, 2016
Showing with 627 additions and 231 deletions.
  1. +31 −11 doc/ref/configuration/minion.rst
  2. +20 −5 doc/ref/states/top.rst
  3. +2 −2 salt/config/__init__.py
  4. +186 −45 salt/state.py
  5. +0 −159 tests/unit/highstateconf_test.py
  6. +388 −9 tests/unit/state_test.py
View
42 doc/ref/configuration/minion.rst
@@ -1218,12 +1218,15 @@ This option has no default value. Set it to an environment name to ensure that
.. note::
Using this value does not change the merging strategy. For instance, if
- :conf_minion:`top_file_merging_strategy` is left at its default, and
+ :conf_minion:`top_file_merging_strategy` is set to ``default``, and
:conf_minion:`state_top_saltenv` is set to ``foo``, then any sections for
environments other than ``foo`` in the top file for the ``foo`` environment
will be ignored. With :conf_minion:`top_file_merging_strategy` set to
``base``, all states from all environments in the ``base`` top file will
- be applied, while all other top files are ignored.
+ be applied, while all other top files are ignored. The only way to set
+ :conf_minion:`state_top_saltenv` to something other than ``base`` and not
+ have the other environments in the targeted top file ignored, would be to
+ set :conf_minion:`top_file_merging_strategy` to ``merge_all``.
.. code-block:: yaml
@@ -1234,19 +1237,29 @@ This option has no default value. Set it to an environment name to ensure that
``top_file_merging_strategy``
-----------------------------
-Default: ``merge``
+.. versionchanged:: Carbon
+ Default value has been changed from ``merge`` to ``default`` to reflect the
+ fact that states from matching targets in matching environments in multiple
+ top files are not actually being merged. Additonally, the ``merge_all``
+ strategy has been added.
+
+Default: ``default``
When no specific fileserver environment (a.k.a. ``saltenv``) has been specified
for a :ref:`highstate <running-highstate>`, all environments' top files are
inspected. This config option determines how the SLS targets in those top files
are handled.
-When set to the default value of ``merge``, all SLS files are interpreted. The
-first target expression for a given environment is kept, and when the same
+When set to ``default``, the ``base`` environment's top file is evaluated
+first, followed by the other environments' top files. The first target
+expression (e.g. ``'*'``) for a given environment is kept, and when the same
target expression is used in a different top file evaluated later, it is
-ignored. The environments will be evaluated in no specific order, for greater
-control over the order in which the environments are evaluated use
-:conf_minion:`env_order`.
+ignored. Because ``base`` is evaluated first, it is authoritative. For
+example, if there is a target for ``'*'`` for the ``foo`` environment in both
+the ``base`` and ``foo`` environment's top files, the one in the ``foo``
+environment would be ignored. The environments will be evaluated in no specific
+order (aside from ``base`` coming first). For greater control over the order in
+which the environments are evaluated, use :conf_minion:`env_order`.
When set to ``same``, then for each environment, only that environment's top
file is processed, with the others being ignored. For example, only the ``dev``
@@ -1256,6 +1269,12 @@ environment's) top file will be ignored. If an environment does not have a top
file, then the top file from the :conf_minion:`default_top` config parameter
will be used as a fallback.
+When set to ``merge_all``, then all states in all environments in all top files
+will be applied. The order in which individual SLS files will be executed will
+depend on the order in which the top files were evaluated, and the environments
+will be evaluated in no specific order. For greater control over the order in
+which the environments are evaluated, use :conf_minion:`env_order`.
+
.. code-block:: yaml
top_file_merging_strategy: same
@@ -1287,9 +1306,10 @@ explicitly defined.
Default: ``base``
When :conf_minion:`top_file_merging_strategy` is set to ``same``, and no
-environment is specified for a :ref:`highstate <running-highstate>`, this
-config option specifies a fallback environment in which to look for a top file
-if an environment lacks one.
+environment is specified for a :ref:`highstate <running-highstate>` (i.e.
+:conf_minion:`environment` is not set for the minion), this config option
+specifies a fallback environment in which to look for a top file if an
+environment lacks one.
.. code-block:: yaml
View
25 doc/ref/states/top.rst
@@ -354,7 +354,8 @@ used, a single top file in the ``base`` environment is the most common way of
configuring a :ref:`highstate <running-highstate>`.
The following minion configuration options affect how top files are compiled
-when no environment is specified:
+when no environment is specified, it is recommended to follow the below four
+links to learn more about how these options work:
- :conf_minion:`state_top_saltenv`
- :conf_minion:`top_file_merging_strategy`
@@ -404,6 +405,7 @@ For the scenarios below, assume the following configuration:
- dev2
qa:
'*':
+ - qa1
- qa2
.. note::
@@ -426,8 +428,8 @@ If the ``base`` environment were specified, the result would be that only the
If the ``qa`` environment were specified, the :ref:`highstate
<running-highstate>` would exit with an error.
-Scenario 2 - No Environment Specified, :conf_minion:`top_file_merging_strategy` is "merge"
-------------------------------------------------------------------------------------------
+Scenario 2 - No Environment Specified, :conf_minion:`top_file_merging_strategy` is "default"
+--------------------------------------------------------------------------------------------
.. versionchanged:: Carbon
The default merging strategy has been renamed from ``merge`` to
@@ -458,5 +460,18 @@ minion2.
If :conf_minion:`default_top` is unset (or set to ``base``, which happens to be
the default), then ``qa1`` from the ``qa`` environment will be applied to all
-minions. If :conf_minion:`default_top` were set to ``dev``, then ``qa2`` from
-the ``qa`` environment would be applied to all minions.
+minions. If :conf_minion:`default_top` were set to ``dev``, then both ``qa1``
+and ``qa2`` from the ``qa`` environment would be applied to all minions.
+
+Scenario 3 - No Environment Specified, :conf_minion:`top_file_merging_strategy` is "merge_all"
+----------------------------------------------------------------------------------------------
+
+.. versionadded:: Carbon
+
+In this scenario, all configured states in all top files are applied. From the
+``base`` environment, ``base1`` would be applied to all minions, with ``base2``
+being applied only to ``minion1``. From the ``dev`` environment, ``dev1`` would
+be applied to all minions, with ``dev2`` being applied only to ``minion2``.
+Finally, from the ``qa`` environment, both the ``qa1`` and ``qa2`` states will
+be applied to all minions. Note that the ``qa1`` states would not be applied
+twice, even though ``qa1`` appears twice.
View
4 salt/config/__init__.py
@@ -969,7 +969,7 @@ def _gather_buffer_space():
'base': [salt.syspaths.BASE_FILE_ROOTS_DIR,
salt.syspaths.SPM_FORMULA_PATH]
},
- 'top_file_merging_strategy': 'merge',
+ 'top_file_merging_strategy': 'default',
'env_order': [],
'default_top': 'base',
'fileserver_limit_traversal': False,
@@ -1185,7 +1185,7 @@ def _gather_buffer_space():
'thorium_roots': {
'base': [salt.syspaths.BASE_THORIUM_ROOTS_DIR],
},
- 'top_file_merging_strategy': 'merge',
+ 'top_file_merging_strategy': 'default',
'env_order': [],
'environment': None,
'default_top': 'base',
View
231 salt/state.py
@@ -2481,6 +2481,16 @@ def __gen_opts(self, opts):
opts['file_roots'] = mopts['file_roots']
opts['top_file_merging_strategy'] = mopts.get('top_file_merging_strategy',
opts.get('top_file_merging_strategy'))
+ if opts['top_file_merging_strategy'] == 'merge':
+ salt.utils.warn_until(
+ 'Oxygen',
+ 'The top_file_merging_strategy \'merge\' has been renamed '
+ 'to \'default\'. Please comment out the '
+ '\'top_file_merging_strategy\' line or change it to '
+ '\'default\'.'
+ )
+ opts['top_file_merging_strategy'] = 'default'
+
opts['env_order'] = mopts.get('env_order', opts.get('env_order', []))
opts['default_top'] = mopts.get('default_top', opts.get('default_top'))
opts['state_events'] = mopts.get('state_events')
@@ -2504,17 +2514,17 @@ def _get_envs(self):
env_intersection = set(env_order).intersection(client_env_list)
final_list = []
for ord_env in env_order:
- if ord_env in env_intersection:
+ if ord_env in env_intersection and ord_env not in final_list:
final_list.append(ord_env)
- return set(final_list)
+ return final_list
elif env_order:
- return set(env_order)
+ return env_order
else:
for cenv in client_envs:
if cenv not in envs:
envs.append(cenv)
- return set(envs)
+ return envs
def get_tops(self):
'''
@@ -2525,12 +2535,13 @@ def get_tops(self):
done = DefaultOrderedDict(list)
found = 0 # did we find any contents in the top files?
# Gather initial top files
- if self.opts['top_file_merging_strategy'] == 'same' and \
- not self.opts['environment']:
+ merging_strategy = self.opts['top_file_merging_strategy']
+ if merging_strategy == 'same' and not self.opts['environment']:
if not self.opts['default_top']:
- raise SaltRenderError('Top file merge strategy set to same, but no default_top '
- 'configuration option was set')
- self.opts['environment'] = self.opts['default_top']
+ raise SaltRenderError(
+ 'top_file_merging_strategy set to \'same\', but no '
+ 'default_top configuration option was set'
+ )
if self.opts['environment']:
contents = self.client.cache_file(
@@ -2551,10 +2562,16 @@ def get_tops(self):
]
else:
tops[self.opts['environment']] = [{}]
- elif self.opts['top_file_merging_strategy'] == 'merge':
+
+ else:
found = 0
- if self.opts.get('state_top_saltenv', False):
- saltenv = self.opts['state_top_saltenv']
+ state_top_saltenv = self.opts.get('state_top_saltenv', False)
+ if state_top_saltenv \
+ and not isinstance(state_top_saltenv, six.string_types):
+ state_top_saltenv = str(state_top_saltenv)
+
+ for saltenv in [state_top_saltenv] if state_top_saltenv \
+ else self._get_envs():
contents = self.client.cache_file(
self.opts['state_top'],
saltenv
@@ -2574,35 +2591,15 @@ def get_tops(self):
else:
tops[saltenv].append({})
log.debug('No contents loaded for env: {0}'.format(saltenv))
- else:
- for saltenv in self._get_envs():
- contents = self.client.cache_file(
- self.opts['state_top'],
- saltenv
- )
- if contents:
- found = found + 1
- tops[saltenv].append(
- compile_template(
- contents,
- self.state.rend,
- self.state.opts['renderer'],
- self.state.opts['renderer_blacklist'],
- self.state.opts['renderer_whitelist'],
- saltenv=saltenv
- )
- )
- else:
- tops[saltenv].append({})
- log.debug('No contents loaded for env: {0}'.format(saltenv))
- if found > 1:
+
+ if found > 1 and merging_strategy.startswith('merge'):
log.warning(
- 'top_file_merging_strategy is set to \'merge\' and '
+ 'top_file_merging_strategy is set to \'%s\' and '
'multiple top files were found. Merging order is not '
'deterministic, it may be desirable to either set '
'top_file_merging_strategy to \'same\' or use the '
'\'env_order\' configuration parameter to specify the '
- 'merging order.'
+ 'merging order.', merging_strategy
)
if found == 0:
@@ -2653,6 +2650,150 @@ def merge_tops(self, tops):
'''
Cleanly merge the top files
'''
+ merging_strategy = self.opts['top_file_merging_strategy']
+ try:
+ merge_attr = '_merge_tops_{0}'.format(merging_strategy)
+ merge_func = getattr(self, merge_attr)
+ if not hasattr(merge_func, '__call__'):
+ msg = '\'{0}\' is not callable'.format(merge_attr)
+ log.error(msg)
+ raise TypeError(msg)
+ except (AttributeError, TypeError):
+ log.warning(
+ 'Invalid top_file_merging_strategy \'%s\', falling back to '
+ '\'default\'', merging_strategy
+ )
+ merge_func = self._merge_tops_default
+ return merge_func(tops)
+
+ def _merge_tops_default(self, tops):
+ '''
+ The default merging strategy. The base env is authoritative, so it is
+ checked first, followed by the remaining environments. In top files
+ from environments other than "base", only the section matching the
+ environment from the top file will be considered, and it too will be
+ ignored if that environment was defined in the "base" top file.
+ '''
+ top = DefaultOrderedDict(OrderedDict)
+
+ # Check base env first as it is authoritative
+ base_tops = tops.pop('base', DefaultOrderedDict(OrderedDict))
+ for ctop in base_tops:
+ for saltenv, targets in six.iteritems(ctop):
+ if saltenv == 'include':
+ continue
+ try:
+ for tgt in targets:
+ top[saltenv][tgt] = ctop[saltenv][tgt]
+ except TypeError:
+ raise SaltRenderError('Unable to render top file. No targets found.')
+
+ for cenv, ctops in six.iteritems(tops):
+ for ctop in ctops:
+ for saltenv, targets in six.iteritems(ctop):
+ if saltenv == 'include':
+ continue
+ elif saltenv != cenv:
+ log.debug(
+ 'Section for saltenv \'%s\' in the \'%s\' '
+ 'saltenv\'s top file will be ignored, as the '
+ 'top_file_merging_strategy is set to \'default\' '
+ 'and the saltenvs do not match',
+ saltenv, cenv
+ )
+ continue
+ elif saltenv in top:
+ log.debug(
+ 'Section for saltenv \'%s\' in the \'%s\' '
+ 'saltenv\'s top file will be ignored, as this '
+ 'saltenv was already defined in the \'base\' top '
+ 'file', saltenv, cenv
+ )
+ continue
+ try:
+ for tgt in targets:
+ top[saltenv][tgt] = ctop[saltenv][tgt]
+ except TypeError:
+ raise SaltRenderError('Unable to render top file. No targets found.')
+ return top
+
+ def _merge_tops_same(self, tops):
+ '''
+ For each saltenv, only consider the top file from that saltenv. All
+ sections matching a given saltenv, which appear in a different
+ saltenv's top file, will be ignored.
+ '''
+ top = DefaultOrderedDict(OrderedDict)
+ for cenv, ctops in six.iteritems(tops):
+ if all([x == {} for x in ctops]):
+ # No top file found in this env, check the default_top
+ default_top = self.opts['default_top']
+ fallback_tops = tops.get(default_top, [])
+ if all([x == {} for x in fallback_tops]):
+ # Nothing in the fallback top file
+ log.error(
+ 'The \'%s\' saltenv has no top file, and the fallback '
+ 'saltenv specified by default_top (%s) also has no '
+ 'top file', cenv, default_top
+ )
+ continue
+
+ for ctop in fallback_tops:
+ for saltenv, targets in six.iteritems(ctop):
+ if saltenv != cenv:
+ continue
+ log.debug(
+ 'The \'%s\' saltenv has no top file, using the '
+ 'default_top saltenv (%s)', cenv, default_top
+ )
+ for tgt in targets:
+ top[saltenv][tgt] = ctop[saltenv][tgt]
+ break
+ else:
+ log.error(
+ 'The \'%s\' saltenv has no top file, and no '
+ 'matches were found in the top file for the '
+ 'default_top saltenv (%s)', cenv, default_top
+ )
+
+ continue
+
+ else:
+ for ctop in ctops:
+ for saltenv, targets in six.iteritems(ctop):
+ if saltenv == 'include':
+ continue
+ elif saltenv != cenv:
+ log.debug(
+ 'Section for saltenv \'%s\' in the \'%s\' '
+ 'saltenv\'s top file will be ignored, as the '
+ 'top_file_merging_strategy is set to \'same\' '
+ 'and the saltenvs do not match',
+ saltenv, cenv
+ )
+ continue
+
+ try:
+ for tgt in targets:
+ top[saltenv][tgt] = ctop[saltenv][tgt]
+ except TypeError:
+ raise SaltRenderError('Unable to render top file. No targets found.')
+ return top
+
+ def _merge_tops_merge_all(self, tops):
+ '''
+ Merge the top files into a single dictionary
+ '''
+ def _read_tgt(tgt):
+ match_type = None
+ states = []
+ for item in tgt:
+ if isinstance(item, dict):
+ match_type = item
+ if isinstance(item, six.string_types):
+ states.append(item)
+ return match_type, states
+
top = DefaultOrderedDict(OrderedDict)
for ctops in six.itervalues(tops):
for ctop in ctops:
@@ -2664,15 +2805,15 @@ def merge_tops(self, tops):
if tgt not in top[saltenv]:
top[saltenv][tgt] = ctop[saltenv][tgt]
continue
- matches = []
- states = set()
- for comp in top[saltenv][tgt]:
- if isinstance(comp, dict):
- matches.append(comp)
- if isinstance(comp, six.string_types):
- states.add(comp)
- top[saltenv][tgt] = matches
- top[saltenv][tgt].extend(list(states))
+ m_type1, m_states1 = _read_tgt(top[saltenv][tgt])
+ m_type2, m_states2 = _read_tgt(ctop[saltenv][tgt])
+ merged = []
+ match_type = m_type2 or m_type1
+ if match_type is not None:
+ merged.append(match_type)
+ merged.extend(m_states1)
+ merged.extend([x for x in m_states2 if x not in merged])
+ top[saltenv][tgt] = merged
except TypeError:
raise SaltRenderError('Unable to render top file. No targets found.')
return top
View
159 tests/unit/highstateconf_test.py
@@ -1,159 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Import Python libs
-from __future__ import absolute_import
-import os
-import os.path
-import tempfile
-
-# Import Salt Testing libs
-from salttesting import TestCase
-from salttesting.mock import patch, MagicMock
-from salttesting.helpers import ensure_in_syspath
-
-ensure_in_syspath('../')
-
-# Import Salt libs
-import integration
-import salt.config
-from salt.state import HighState
-from salt.utils.odict import OrderedDict, DefaultOrderedDict
-
-
-class HighStateTestCase(TestCase):
- def setUp(self):
- self.root_dir = tempfile.mkdtemp(dir=integration.TMP)
- self.state_tree_dir = os.path.join(self.root_dir, 'state_tree')
- self.cache_dir = os.path.join(self.root_dir, 'cachedir')
- if not os.path.isdir(self.root_dir):
- os.makedirs(self.root_dir)
-
- if not os.path.isdir(self.state_tree_dir):
- os.makedirs(self.state_tree_dir)
-
- if not os.path.isdir(self.cache_dir):
- os.makedirs(self.cache_dir)
- self.config = salt.config.minion_config(None)
- self.config['root_dir'] = self.root_dir
- self.config['state_events'] = False
- self.config['id'] = 'match'
- self.config['file_client'] = 'local'
- self.config['file_roots'] = dict(base=[self.state_tree_dir])
- self.config['cachedir'] = self.cache_dir
- self.config['test'] = False
- self.highstate = HighState(self.config)
- self.highstate.push_active()
-
- def tearDown(self):
- self.highstate.pop_active()
-
- def test_top_matches_with_list(self):
- top = {'env': {'match': ['state1', 'state2'], 'nomatch': ['state3']}}
- matches = self.highstate.top_matches(top)
- self.assertEqual(matches, {'env': ['state1', 'state2']})
-
- def test_top_matches_with_string(self):
- top = {'env': {'match': 'state1', 'nomatch': 'state2'}}
- matches = self.highstate.top_matches(top)
- self.assertEqual(matches, {'env': ['state1']})
-
- def test_matches_whitelist(self):
- matches = {'env': ['state1', 'state2', 'state3']}
- matches = self.highstate.matches_whitelist(matches, ['state2'])
- self.assertEqual(matches, {'env': ['state2']})
-
- def test_matches_whitelist_with_string(self):
- matches = {'env': ['state1', 'state2', 'state3']}
- matches = self.highstate.matches_whitelist(matches,
- 'state2,state3')
- self.assertEqual(matches, {'env': ['state2', 'state3']})
-
-
-class TopFileMergeTestCase(TestCase):
- '''
- Test various merge strategies for multiple tops files collected from
- multiple environments. Various options correspond to merge strategies
- which can be set by the user with the top_file_merging_strategy config
- option.
-
- Refs #12483
- '''
- def setUp(self):
- '''
- Create multiple top files for use in each test
- '''
- self.env1 = {'base': {'*': ['e1_a', 'e1_b', 'e1_c']}}
- self.env2 = {'base': {'*': ['e2_a', 'e2_b', 'e2_c']}}
- self.env3 = {'base': {'*': ['e3_a', 'e3_b', 'e3_c']}}
- self.config = self._make_default_config()
- self.highstate = HighState(self.config)
-
- def _make_default_config(self):
- config = salt.config.minion_config(None)
- root_dir = tempfile.mkdtemp(dir=integration.TMP)
- state_tree_dir = os.path.join(root_dir, 'state_tree')
- cache_dir = os.path.join(root_dir, 'cachedir')
- config['root_dir'] = root_dir
- config['state_events'] = False
- config['id'] = 'match'
- config['file_client'] = 'local'
- config['file_roots'] = dict(base=[state_tree_dir])
- config['cachedir'] = cache_dir
- config['test'] = False
- return config
-
- def _get_tops(self):
- '''
- A test helper to emulate HighState.get_tops() but just to construct
- an appropriate data structure for top files from multiple environments
- '''
- tops = DefaultOrderedDict(list)
-
- tops['a'].append(self.env1)
- tops['b'].append(self.env2)
- tops['c'].append(self.env3)
- return tops
-
- def test_basic_merge(self):
- '''
- This is the default approach for Salt. Merge the top files with the
- earlier appends taking precendence. Since Salt does the appends
- lexecographically, this is effectively a test against the default
- lexecographical behaviour.
- '''
- merged_tops = self.highstate.merge_tops(self._get_tops())
-
- expected_merge = DefaultOrderedDict(OrderedDict)
- expected_merge['base']['*'] = ['e1_c', 'e1_b', 'e1_a']
- self.assertEqual(merged_tops, expected_merge)
-
- def test_merge_strategy_same(self):
- '''
- Test to see if the top file that corresponds
- to the requested env is the one that is used
- by the state system
- '''
- config = self._make_default_config()
- config['top_file_merging_strategy'] = 'same'
- config['environment'] = 'b'
- highstate = HighState(config)
- ret = highstate.get_tops()
- self.assertEqual(ret, OrderedDict([('b', [{}])]))
-
- def test_ordered_merge(self):
- '''
- Test to see if the merger respects environment
- ordering
- '''
- config = self._make_default_config()
- config['top_file_merging_strategy'] = 'merge'
- config['env_order'] = ['b', 'a', 'c']
- with patch('salt.fileclient.FSClient.envs', MagicMock(return_value=['a', 'b', 'c'])):
- highstate = HighState(config)
- ret = highstate.get_tops()
- self.assertEqual(ret, OrderedDict([('a', [{}]), ('c', [{}]), ('b', [{}])]))
-
-
-if __name__ == '__main__':
- from integration import run_tests
- run_tests(HighStateTestCase, needs_daemon=False)
View
397 tests/unit/state_test.py
@@ -5,26 +5,24 @@
# Import Python libs
from __future__ import absolute_import
+import copy
import os
import sys
+import tempfile
# Import Salt Testing libs
-from integration import TMP_CONF_DIR
+import integration
from salttesting import TestCase, skipIf
from salttesting.helpers import ensure_in_syspath
-from salttesting.mock import (
- NO_MOCK,
- NO_MOCK_REASON,
- patch
-)
-
+from salttesting.mock import NO_MOCK, NO_MOCK_REASON, patch
ensure_in_syspath('../')
# Import Salt libs
import salt.state
+import salt.config
import salt.exceptions
-from salt.utils.odict import OrderedDict
+from salt.utils.odict import OrderedDict, DefaultOrderedDict
@skipIf(NO_MOCK, NO_MOCK_REASON)
@@ -55,13 +53,394 @@ def test_render_error_on_invalid_requisite(self, state_patch):
exception when a requisite cannot be resolved
'''
high_data = {'git': OrderedDict([('pkg', [OrderedDict([('require', [OrderedDict([('file', OrderedDict([('test1', 'test')]))])])]), 'installed', {'order': 10000}]), ('__sls__', u'issue_35226'), ('__env__', 'base')])}
- minion_opts = salt.config.minion_config(os.path.join(TMP_CONF_DIR, 'minion'))
+ minion_opts = salt.config.minion_config(os.path.join(integration.TMP_CONF_DIR, 'minion'))
minion_opts['pillar'] = {'git': OrderedDict([('test1', 'test')])}
state_obj = salt.state.State(minion_opts)
with self.assertRaises(salt.exceptions.SaltRenderError):
state_obj.call_high(high_data)
+class HighStateTestCase(TestCase):
+ def setUp(self):
+ self.root_dir = tempfile.mkdtemp(dir=integration.TMP)
+ self.state_tree_dir = os.path.join(self.root_dir, 'state_tree')
+ self.cache_dir = os.path.join(self.root_dir, 'cachedir')
+ if not os.path.isdir(self.root_dir):
+ os.makedirs(self.root_dir)
+
+ if not os.path.isdir(self.state_tree_dir):
+ os.makedirs(self.state_tree_dir)
+
+ if not os.path.isdir(self.cache_dir):
+ os.makedirs(self.cache_dir)
+ self.config = salt.config.minion_config(None)
+ self.config['root_dir'] = self.root_dir
+ self.config['state_events'] = False
+ self.config['id'] = 'match'
+ self.config['file_client'] = 'local'
+ self.config['file_roots'] = dict(base=[self.state_tree_dir])
+ self.config['cachedir'] = self.cache_dir
+ self.config['test'] = False
+ self.highstate = salt.state.HighState(self.config)
+ self.highstate.push_active()
+
+ def tearDown(self):
+ self.highstate.pop_active()
+
+ def test_top_matches_with_list(self):
+ top = {'env': {'match': ['state1', 'state2'], 'nomatch': ['state3']}}
+ matches = self.highstate.top_matches(top)
+ self.assertEqual(matches, {'env': ['state1', 'state2']})
+
+ def test_top_matches_with_string(self):
+ top = {'env': {'match': 'state1', 'nomatch': 'state2'}}
+ matches = self.highstate.top_matches(top)
+ self.assertEqual(matches, {'env': ['state1']})
+
+ def test_matches_whitelist(self):
+ matches = {'env': ['state1', 'state2', 'state3']}
+ matches = self.highstate.matches_whitelist(matches, ['state2'])
+ self.assertEqual(matches, {'env': ['state2']})
+
+ def test_matches_whitelist_with_string(self):
+ matches = {'env': ['state1', 'state2', 'state3']}
+ matches = self.highstate.matches_whitelist(matches,
+ 'state2,state3')
+ self.assertEqual(matches, {'env': ['state2', 'state3']})
+
+
+class TopFileMergeTestCase(TestCase):
+ '''
+ Test various merge strategies for multiple tops files collected from
+ multiple environments. Various options correspond to merge strategies
+ which can be set by the user with the top_file_merging_strategy config
+ option.
+ '''
+ def setUp(self):
+ '''
+ Create multiple top files for use in each test. Envs within self.tops
+ should be defined in the same order as this ordering will affect
+ ordering in merge_tops. The envs in each top file are defined in the
+ same order as self.env_order. This is no accident; it was done this way
+ in order to produce the proper deterministic results to match the
+ tests. Changing anything created in this func will affect the tests, as
+ they would affect ordering in states in real life. So, don't change any
+ of this unless you know what you're doing. If a test is failing, it is
+ likely due to incorrect logic in merge_tops.
+ '''
+ self.env_order = ['base', 'foo', 'bar', 'baz']
+ self.tops = {
+ 'base': OrderedDict([
+ ('base', OrderedDict([('*', ['base_base'])])),
+ ('foo', OrderedDict([('*', ['base_foo'])])),
+ ('bar', OrderedDict([('*', ['base_bar'])])),
+ ('baz', OrderedDict([('*', ['base_baz'])])),
+ ]),
+ 'foo': OrderedDict([
+ ('base', OrderedDict([('*', ['foo_base'])])),
+ ('foo', OrderedDict([('*', ['foo_foo'])])),
+ ('bar', OrderedDict([('*', ['foo_bar'])])),
+ ('baz', OrderedDict([('*', ['foo_baz'])])),
+ ]),
+ 'bar': OrderedDict([
+ ('base', OrderedDict([('*', ['bar_base'])])),
+ ('foo', OrderedDict([('*', ['bar_foo'])])),
+ ('bar', OrderedDict([('*', ['bar_bar'])])),
+ ('baz', OrderedDict([('*', ['bar_baz'])])),
+ ]),
+ # Empty environment
+ 'baz': OrderedDict()
+ }
+
+ # Version without the other envs defined in the base top file
+ self.tops_limited_base = copy.deepcopy(self.tops)
+ self.tops_limited_base['base'] = OrderedDict([
+ ('base', OrderedDict([('*', ['base_base'])])),
+ ])
+
+ @staticmethod
+ def highstate(**opts):
+ config = salt.config.minion_config(None)
+ root_dir = tempfile.mkdtemp(dir=integration.TMP)
+ state_tree_dir = os.path.join(root_dir, 'state_tree')
+ cache_dir = os.path.join(root_dir, 'cachedir')
+ config['root_dir'] = root_dir
+ config['state_events'] = False
+ config['id'] = 'match'
+ config['file_client'] = 'local'
+ config['file_roots'] = dict(base=[state_tree_dir])
+ config['cachedir'] = cache_dir
+ config['test'] = False
+ config['default_top'] = 'base'
+ config.update(opts)
+ return salt.state.HighState(config)
+
+ def get_tops(self, tops=None, env_order=None, state_top_saltenv=None):
+ '''
+ A test helper to emulate salt.state.HighState.get_tops() but just to
+ construct an appropriate data structure for top files from multiple
+ environments
+ '''
+ if tops is None:
+ tops = self.tops
+
+ if state_top_saltenv:
+ append_order = [state_top_saltenv]
+ elif env_order:
+ append_order = env_order
+ else:
+ append_order = self.env_order
+
+ ret = DefaultOrderedDict(list)
+ for env in append_order:
+ item = tops[env]
+ if env_order:
+ for remove in [x for x in self.env_order if x not in env_order]:
+ # Remove this env from the tops from the tops since this
+ # env is not part of env_order.
+ item.pop(remove)
+ ret[env].append(tops[env])
+ return ret
+
+ def test_merge_tops_default(self):
+ '''
+ Test the default merge strategy for top files, in an instance where the
+ base top file contains sections for all envs and the other envs' top
+ files are therefore ignored.
+ '''
+ merged_tops = self.highstate().merge_tops(self.get_tops())
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in self.env_order:
+ expected_merge[env]['*'] = ['base_{0}'.format(env)]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_default_limited_base(self):
+ '''
+ Test the default merge strategy for top files when
+ '''
+ tops = self.get_tops(tops=self.tops_limited_base)
+ merged_tops = self.highstate().merge_tops(tops)
+
+ # No baz in the expected results because baz has no top file
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in self.env_order[:-1]:
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_default_state_top_saltenv_base(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'base' saltenv, with the default merging strategy. This should
+ result in all states from the 'base' top file being in the merged
+ result.
+ '''
+ env = 'base'
+ tops = self.get_tops(state_top_saltenv=env)
+ merged_tops = self.highstate().merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env2 in self.env_order:
+ expected_merge[env2]['*'] = ['_'.join((env, env2))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_default_state_top_saltenv_foo(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'foo' saltenv, with the default merging strategy. This should
+ result in just the 'foo' environment's states from the 'foo' top file
+ being in the merged result.
+ '''
+ env = 'foo'
+ tops = self.get_tops(state_top_saltenv=env)
+ merged_tops = self.highstate().merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_merge_all(self):
+ '''
+ Test the merge_all strategy
+ '''
+ tops = self.get_tops()
+ merged_tops = self.highstate(
+ top_file_merging_strategy='merge_all').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in self.env_order:
+ states = []
+ for top_env in self.env_order:
+ if top_env in tops[top_env][0]:
+ states.extend(tops[top_env][0][env]['*'])
+ expected_merge[env]['*'] = states
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_merge_all_with_env_order(self):
+ '''
+ Test an altered env_order with the 'merge_all' strategy.
+ '''
+ env_order = ['bar', 'foo', 'base']
+ tops = self.get_tops(env_order=env_order)
+ merged_tops = self.highstate(
+ top_file_merging_strategy='merge_all',
+ env_order=env_order).merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in [x for x in self.env_order if x in env_order]:
+ states = []
+ for top_env in env_order:
+ states.extend(tops[top_env][0][env]['*'])
+ expected_merge[env]['*'] = states
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_merge_all_state_top_saltenv_base(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'base' saltenv, with the 'merge_all' merging strategy. This should
+ result in all states from the 'base' top file being in the merged
+ result.
+ '''
+ env = 'base'
+ tops = self.get_tops(state_top_saltenv=env)
+ merged_tops = self.highstate(
+ top_file_merging_strategy='merge_all').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env2 in self.env_order:
+ expected_merge[env2]['*'] = ['_'.join((env, env2))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_merge_all_state_top_saltenv_foo(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'foo' saltenv, with the 'merge_all' merging strategy. This should
+ result in all the states from the 'foo' top file being in the merged
+ result.
+ '''
+ env = 'foo'
+ tops = self.get_tops(state_top_saltenv=env)
+ merged_tops = self.highstate(
+ top_file_merging_strategy='merge_all').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env2 in self.env_order:
+ expected_merge[env2]['*'] = ['_'.join((env, env2))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_same_with_default_top(self):
+ '''
+ Test to see if the top file that corresponds to the requested env is
+ the one that is used by the state system. Also test the 'default_top'
+ option for env 'baz', which has no top file and should pull its states
+ from the 'foo' top file.
+ '''
+ merged_tops = self.highstate(
+ top_file_merging_strategy='same',
+ default_top='foo').merge_tops(self.get_tops())
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in self.env_order[:-1]:
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+ # The 'baz' env should be using the foo top file because baz lacks a
+ # top file, and default_top has been seet to 'foo'
+ expected_merge['baz']['*'] = ['foo_baz']
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_same_without_default_top(self):
+ '''
+ Test to see if the top file that corresponds to the requested env is
+ the one that is used by the state system. default_top will not be set
+ (falling back to 'base'), so the 'baz' environment should pull its
+ states from the 'base' top file.
+ '''
+ merged_tops = self.highstate(
+ top_file_merging_strategy='same').merge_tops(self.get_tops())
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in self.env_order[:-1]:
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+ # The 'baz' env should be using the foo top file because baz lacks a
+ # top file, and default_top == 'base'
+ expected_merge['baz']['*'] = ['base_baz']
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_same_limited_base_without_default_top(self):
+ '''
+ Test to see if the top file that corresponds to the requested env is
+ the one that is used by the state system. default_top will not be set
+ (falling back to 'base'), and since we are using a limited base top
+ file, the 'baz' environment should not appear in the merged tops.
+ '''
+ tops = self.get_tops(tops=self.tops_limited_base)
+ merged_tops = \
+ self.highstate(top_file_merging_strategy='same').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ for env in self.env_order[:-1]:
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_same_state_top_saltenv_base(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'base' saltenv, with the 'same' merging strategy. This should
+ result in just the 'base' environment's states from the 'base' top file
+ being in the merged result.
+ '''
+ env = 'base'
+ tops = self.get_tops(state_top_saltenv=env)
+ merged_tops = self.highstate(
+ top_file_merging_strategy='same').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_same_state_top_saltenv_foo(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'foo' saltenv, with the 'same' merging strategy. This should
+ result in just the 'foo' environment's states from the 'foo' top file
+ being in the merged result.
+ '''
+ env = 'foo'
+ tops = self.get_tops(state_top_saltenv=env)
+ merged_tops = self.highstate(
+ top_file_merging_strategy='same').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+ expected_merge[env]['*'] = ['_'.join((env, env))]
+
+ self.assertEqual(merged_tops, expected_merge)
+
+ def test_merge_tops_same_state_top_saltenv_baz(self):
+ '''
+ Test the 'state_top_saltenv' parameter to load states exclusively from
+ the 'baz' saltenv, with the 'same' merging strategy. This should
+ result in an empty dictionary since this environment has not top file.
+ '''
+ tops = self.get_tops(state_top_saltenv='baz')
+ merged_tops = self.highstate(
+ top_file_merging_strategy='same').merge_tops(tops)
+
+ expected_merge = DefaultOrderedDict(OrderedDict)
+
+ self.assertEqual(merged_tops, expected_merge)
+
+
if __name__ == '__main__':
from integration import run_tests
run_tests(StateCompilerTestCase, needs_daemon=False)

0 comments on commit f7d8f40

Please sign in to comment.