[Resolve #1114,#426] Project Dependencies part 1: ResolvableProperty …

…refactor + tags resolvable property (#1184)

This is the first in a series of pull requests that addresses #1114 , allowing Sceptre to manage its own dependencies.

## In this PR:
* **ResolvableProperty is refactored into a base class with a single subclass (for now): ResolvableContainerProperty.** _This refactor will be necessary so we can build on that base class to make a ResolvableValueProperty in a later PR so we can resolve single values like template_bucket_name, etc..._. 
* **The merge strategies list_join and dict_merge now perform deepcopy on the product of those joins,** preventing changes in child stack configs from modifying the same dict/list in memory that has been inherited by other child stack configs. This change is necessary so inheriting mutable values from parent configs becomes a lot safer.
* **A few bugs have been fixed in ResolvableProperty** in this refactor in preparation for changes that are to come in future Pull requests. These all have to do with being able to inherit resolvers and having more resolvers on stack configurations. 
    - Previously, the `_no_recursive_get` method was effectively blocking access to the same property on a DIFFERENT stack. If a resolver accesses the same property on another stack, there shouldn't be a problem with this. This is relevant in cases like the StackOutput resolver that accesses _another_ stack's attributes. This bug has been fixed now. `_no_recursive_get` specifically disallows recursive gets on the _current stack only_.
    - If a resolver was set on a StackGroup config and then "inherited" by multiple stack groups, the _exact same_ object in memory would now be on two stacks. Worse, they would both have the same Stack object assigned to them... which means that Stack A could use a resolver that was referencing Stack B. This has now been solved by resolvers being cloned when they are assigned to a stack, guaranteeing that every stack has an independent resolver in memory.
    - Deferred resolution is properly handled _within_ the property. Previously it wasn't, so we needed a weird double-layer property on the Stack class to handle sceptre_user_data. This is now no longer necessary and is handled where it should have been handled all along: in the property itself.
* **stack_tags is now a ResolvableProperty**, ultimately finishing what #847 attempted but never finished.
* Documentation on which Stack and StackGroup configuration keys are resolvable and which are not has been added.

docs/_source/conf.py

@@ -42,6 +42,7 @@
 # ones.
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinx_autodoc_typehints',
     'sphinx.ext.doctest',
     'sphinx.ext.intersphinx',
     'sphinx.ext.coverage',
@@ -204,5 +205,11 @@
     ('py:class', 'sceptre.config.reader.Attributes'),
     ('py:class', 'sceptre.diffing.stack_differ.DiffType'),
     ('py:obj', 'sceptre.diffing.stack_differ.DiffType'),
-    ('py:class', 'TextIO')
+    ('py:class', 'DiffType'),
+    ('py:class', 'TextIO'),
+    ('py:class', '_io.StringIO'),
+    ('py:class', 'yaml.loader.SafeLoader'),
+    ('py:class', 'yaml.dumper.Dumper')
 ]
+
+set_type_checking_flag = True

docs/_source/docs/resolvers.rst

@@ -79,10 +79,16 @@ Example:
        VpcIdParameter: !stack_output shared/vpc.yaml::VpcIdOutput
 
 Sceptre infers that the Stack to fetch the output value from is a dependency,
-and builds that Stack before the current one.
-
-This resolver will add a dependency for the Stack in which needs the output
-from.
+adding that stack to the current stack's list of dependencies. This instructs
+Sceptre to build that Stack before the current one.
+
+.. warning::
+   Be careful when using the stack_output resolver that you do not create circular dependencies.
+   This is especially true when using this on StackGroup Configs to create configurations
+   to be inherited by all stacks in that group. If the `!stack_output` resolver would be "inherited"
+   from a StackGroup Config by the stack it references, this will lead to a circular dependency.
+   The correct way to work around this is to move that stack outside that StackGroup so that it
+   doesn't "inherit" that resolver.
 
 stack_output_external
 ~~~~~~~~~~~~~~~~~~~~~

docs/_source/docs/stack_config.rst

@@ -1,8 +1,9 @@
 Stack Config
 ============
 
-Stack config stores config related to a particular Stack, such as the path to
-that Stack’s Template, and any parameters that Stack may require.
+A Stack config stores configurations related to a particular Stack, such as the path to
+that Stack’s Template, and any parameters that Stack may require. Many of these configuration keys
+support resolvers and can be inherited from parent StackGroup configs.
 
 .. _stack_config-structure:
 
@@ -31,6 +32,8 @@ you will receive an error when deploying the stack.
 
 template_path
 ~~~~~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: No
 
 The path to the CloudFormation, Jinja2 or Python template to build the Stack
 from. The path can either be absolute or relative to the Sceptre Directory.
@@ -44,6 +47,8 @@ from the Stack config filename.
 
 template
 ~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: No
 
 Configuration for a template handler. Template handlers can take in parameters
 and resolve that to a CloudFormation template. This enables you to not only
@@ -66,20 +71,39 @@ developing your own in the :doc:`template_handlers` section.
 
 dependencies
 ~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Appended to parent's dependencies
 
 A list of other Stacks in the environment that this Stack depends on. Note that
 if a Stack fetches an output value from another Stack using the
 ``stack_output`` resolver, that Stack is automatically added as a dependency,
 and that Stack need not be added as an explicit dependency.
 
+.. warning::
+   Be careful about how you structure dependencies. It is possible to create circular
+   dependencies accidentally, where multiple stacks depend on each other. Sceptre
+   will detect this and raise an error, blocking this sort of setup. You must be especially careful
+   when specifying ``dependencies`` on a StackGroup config. These dependencies will then be
+   "inherited" by every stack within that StackGroup. If one of those dependencies *inherits* that
+   list of dependencies, it will cause a circular dependency. If this happens, you can resolve the
+   situation by either (a) setting those ``dependencies`` on individual Stack Configs rather than the
+   the StackGroup Config, or (b) moving those dependency stacks outside of the StackGroup.
+
 hooks
 ~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 A list of arbitrary shell or Python commands or scripts to run. Find out more
 in the :doc:`hooks` section.
 
 notifications
 ~~~~~~~~~~~~~
+* Resolvable: Yes
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 List of SNS topic ARNs to publish Stack related events to. A maximum of 5 ARNs
 can be specified per Stack. This configuration will be used by the ``create``,
@@ -89,6 +113,9 @@ documentation`_.
 
 on_failure
 ~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 This parameter describes the action taken by CloudFormation when a Stack fails
 to create. For more information and valid values see the `AWS Documentation`_.
@@ -104,6 +131,9 @@ Examples include:
 
 parameters
 ~~~~~~~~~~
+* Resolvable: Yes
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 .. warning::
 
@@ -166,6 +196,9 @@ Example:
 
 protected
 ~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 Stack protection against execution of the following commands:
 
@@ -180,12 +213,18 @@ throw an error.
 
 role_arn
 ~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 The ARN of a `CloudFormation Service Role`_ that is assumed by CloudFormation
 to create, update or delete resources.
 
 iam_role
 ~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 This is the IAM Role ARN that **Sceptre** should *assume* using AWS STS when executing any actions
 on the Stack.
@@ -208,13 +247,18 @@ permits the user to assume that role.
 
 sceptre_user_data
 ~~~~~~~~~~~~~~~~~
+* Resolvable: Yes
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 Represents data to be passed to the ``sceptre_handler(sceptre_user_data)``
 function in Python templates or accessible under ``sceptre_user_data`` variable
 key within Jinja2 templates.
 
 stack_name
 ~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: No
 
 A custom name to use instead of the Sceptre default.
 
@@ -248,11 +292,17 @@ referring to is in a different AWS account or region.
 
 stack_tags
 ~~~~~~~~~~
+* Resolvable: Yes
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 A dictionary of `CloudFormation Tags`_ to be applied to the Stack.
 
 stack_timeout
 ~~~~~~~~~~~~~
+* Resolvable: No
+* Can be inherited from StackGroup: Yes
+* Inheritance strategy: Overrides parent if set
 
 A timeout in minutes before considering the Stack deployment as failed. After
 the specified timeout, the Stack will be rolled back. Specifiyng zero, as well
@@ -360,19 +410,19 @@ Examples
        tag_1: value_1
        tag_2: value_2
 
-.. _template_path: #template_path
+.. _template_path: #template-path
 .. _template: #template
 .. _dependencies: #dependencies
 .. _hooks: #hooks
 .. _notifications: #notifications
-.. _on_failure: #on_failure
+.. _on_failure: #on-failure
 .. _parameters: #parameters
 .. _protected: #protected
-.. _role_arn: #role_arn
-.. _sceptre_user_data: #sceptre_user_data
-.. _stack_name: #stack_name
-.. _stack_tags: #stack_tags
-.. _stack_timeout: #stack_timeout
+.. _role_arn: #role-arn
+.. _sceptre_user_data: #sceptre-user-data
+.. _stack_name: #stack-name
+.. _stack_tags: #stack-tags
+.. _stack_timeout: #stack-timeout
 .. _AWS CloudFormation API documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_CreateStack.html
 .. _AWS Documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_CreateStack.html
 .. _CloudFormation Service Role: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-servicerole.html

docs/_source/docs/stack_group_config.rst

@@ -28,6 +28,8 @@ made available via ``stack_group_confg`` attribute on ``Stack()``.
 
 profile
 ~~~~~~~
+* Resolvable: No
+* Inheritance strategy: Overrides parent if set by child
 
 The name of the profile as defined in ``~/.aws/config`` and
 ``~/.aws/credentials``. Use the `aws configure --profile <profile_id>` command
@@ -37,17 +39,23 @@ Reference: `AWS_CLI_Configure`_
 
 project_code
 ~~~~~~~~~~~~
+* Resolvable: No
+* Inheritance strategy: Overrides parent if set by child
 
 A string which is prepended to the Stack names of all Stacks built by Sceptre.
 
 region
 ~~~~~~
+* Resolvable: No
+* Inheritance strategy: Overrides parent if set by child
 
 The AWS region to build Stacks in. Sceptre should work in any `region which
 supports CloudFormation`_.
 
 template_bucket_name
 ~~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Inheritance strategy: Overrides parent if set by child
 
 The name of an S3 bucket to upload CloudFormation Templates to. Note that S3
 bucket names must be globally unique. If the bucket does not exist, Sceptre
@@ -60,6 +68,8 @@ supplied in this way have a lower maximum length, so using the
 
 template_key_prefix
 ~~~~~~~~~~~~~~~~~~~
+* Resolvable: No
+* Inheritance strategy: Overrides parent if set by child
 
 A string which is prefixed onto the key used to store templates uploaded to S3.
 Templates are stored using the key:
@@ -77,7 +87,9 @@ Note that if ``template_bucket_name`` is not supplied, this parameter is
 ignored.
 
 j2_environment
-~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~
+* Resolvable: No
+* Inheritance strategy: Child configs will be merged with parent configs
 
 A dictionary that is combined with the default jinja2 environment.
 It's converted to keyword arguments then passed to [jinja2.Environment](https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.Environment).
@@ -134,7 +146,8 @@ General configurations should be defined at a high level, and more specific
 configurations should be defined at a lower directory level.
 
 YAML files that define configuration settings with conflicting keys, the child
-configuration file will take precedence.
+configuration file will usually take precedence (see the specific config keys as documented
+for the inheritance strategy employed).
 
 In the above directory structure, ``config/config.yaml`` will be read in first,
 followed by ``config/account-1/config.yaml``, followed by

integration-tests/steps/helpers.py

@@ -73,7 +73,7 @@ def get_cloudformation_stack_name(context, stack_name):
 
 
 def retry_boto_call(func, *args, **kwargs):
-    delay = 2
+    delay = 5
     max_retries = 150
     attempts = 0
     while attempts < max_retries:

requirements/dev.txt

@@ -15,6 +15,7 @@ setuptools>=40.6.2,<41.0.0
 Sphinx>=1.6.5,<4.3
 sphinx-click>=2.0.1,<4.0.0
 sphinx-rtd-theme==0.5.2
+sphinx-autodoc-typehints==1.12.0
 docutils<0.17 # temporary fix for sphinx-rtd-theme==0.5.2, it depends on docutils<0.17
 tox>=3.23.0,<4.0.0
 twine>=1.12.1,<2.0.0

sceptre/config/strategies.py

@@ -6,6 +6,7 @@
 This module contains the implementations of the strategies used to merge config
 attributes.
 """
+from copy import deepcopy
 
 
 def list_join(a, b):
@@ -21,16 +22,17 @@ def list_join(a, b):
     """
     if a and not isinstance(a, list):
         raise TypeError('{} is not a list'.format(a))
+
     if b and not isinstance(b, list):
         raise TypeError('{} is not a list'.format(b))
 
     if a is None:
-        return b
+        return deepcopy(b)
 
     if b is not None:
-        return a + b
+        return deepcopy(a + b)
 
-    return a
+    return deepcopy(a)
 
 
 def dict_merge(a, b):
@@ -50,13 +52,12 @@ def dict_merge(a, b):
         raise TypeError('{} is not a dict'.format(b))
 
     if a is None:
-        return b
+        return deepcopy(b)
 
     if b is not None:
-        a.update(b)
-        return a
+        return deepcopy({**a, **b})
 
-    return a
+    return deepcopy(a)
 
 
 def child_wins(a, b):