[Resolve #1252] Supporting resolvers in Hook and Resolver arguments, …

…with new !substitute, !join, !split, and !select resolvers! (#1313)

This pull request brings the ability for Hooks and Resolvers to have resolvers in their arguments and have those resolvers automatically resolve when `self.argument` is accessed. When a resolver or hook is `setup`, it will also setup all nested resolvers in its argument. And when it is attached to a Stack, it and its argument will be recursively cloned so that no resolver is associated with a different stack. This will let resolvers be "inherited" from parent stack configurations.

docs/_source/docs/hooks.rst

@@ -207,7 +207,9 @@ This hook can be used in a Stack config file with the following syntax:
 
 hook arguments
 ^^^^^^^^^^^^^^
-Hook arguments can be a simple string or a complex data structure.
+Hook arguments can be a simple string or a complex data structure. You can even use resolvers in
+hook arguments, so long as they're nested in a list or a dict.
+
 Assume a Sceptre `copy` hook that calls the `cp command`_:
 
 .. code-block:: yaml
@@ -224,7 +226,7 @@ Assume a Sceptre `copy` hook that calls the `cp command`_:
        - !copy
            options: "-r"
            source: "from_dir"
-           destination: "to_dir"
+           destination: !stack_output my/other/stack::CopyDestination
 
 .. _Custom Hooks: #custom-hooks
 .. _subprocess documentation: https://docs.python.org/3/library/subprocess.html

docs/_source/docs/resolvers.rst

@@ -5,6 +5,9 @@ Sceptre implements resolvers, which can be used to resolve a value of a
 CloudFormation ``parameter`` or ``sceptre_user_data`` value at runtime. This is
 most commonly used to chain the outputs of one Stack to the inputs of another.
 
+You can use resolvers with any resolvable property on a StackConfig, as well as in the arguments
+of hooks and other resolvers.
+
 If required, users can create their own resolvers, as described in the section
 on `Custom Resolvers`_.
 
@@ -52,6 +55,27 @@ file_contents
 
 **deprecated**: Consider using the `file`_ resolver instead.
 
+join
+~~~~
+
+This resolver allows you to join multiple strings together to form a single string. This is great
+for combining the outputs of multiple resolvers. This resolver works just like CloudFormation's
+``!Join`` intrinsic function.
+
+The argument for this resolver should be a list with two elements: (1) A string to join the elements
+on and (2) a list of items to join.
+
+Example:
+
+.. code-block:: yaml
+
+   parameters:
+     BaseUrl: !join
+       - ":"
+       - - !stack_output my/app/stack.yaml::HostName
+         - !stack_output my/other/stack.yaml::Port
+
+
 no_value
 ~~~~~~~~
 
@@ -79,6 +103,48 @@ A resolver to execute any shell command.
 
 Refer to `sceptre-resolver-cmd <https://github.com/Sceptre/sceptre-resolver-cmd/>`_ for documentation.
 
+select
+~~~~~~
+
+This resolver allows you to select a specific index of a list of items. This is great for combining
+with the ``!split`` resolver to obtain part of a string. This function works almost the same as
+CloudFormation's ``!Select`` intrinsic function, **except you can use this with negative indices to
+select from the end of a list**.
+
+The argument for this resolver should be a list with two elements: (1) A numerical index and (2) a
+list of items to select out of. If the index is negative, it will select from the end of the list.
+For example, "-1" would select the last element and "-2" would select the second-to-last element.
+
+Example:
+
+.. code-block:: yaml
+
+   sceptre_user_data:
+     # This selects the last element after you split the connection string on "/"
+     DatabaseName: !select
+       - -1
+       - !split ["/", !stack_output my/database/stack.yaml::ConnectionString]
+
+split
+~~~~~
+
+This resolver will split a value on a given delimiter string. This is great when combining with the
+``!select`` resolver. This function works the same as CloudFormation's ``!Split`` intrinsic function.
+
+Note: The return value of this resolver is a *list*, not a string. This will not work to set Stack
+configurations that expect strings, but it WILL work to set Stack configurations that expect lists.
+
+The argument for this resolver should be a list with two elements: (1) The delimiter to split on and
+(2) a string to split.
+
+Example:
+
+.. code-block:: yaml
+
+   notifications: !split
+     - ";"
+     - !stack_output my/sns/topics.yaml::SemicolonDelimitedArns
+
 .. _stack_attr_resolver:
 
 stack_attr
@@ -181,6 +247,67 @@ Example:
    parameters:
      VpcIdParameter: !stack_output_external prj-network-vpc::VpcIdOutput prod
 
+
+sub
+~~~
+
+This resolver allows you to create a string using Python string format syntax. This functions as a
+great way to combine together a number of resolver outputs into a single string. This functions
+similarly to Cloudformation's ``!Sub`` intrinsic function.
+
+It should be noted that Jinja2 syntax is far more capable of interpolating values than this resolver,
+so you should use Jinja2 if all you need is to interpolate raw values from environment variables,
+variables from stack group configs, var files, and ``--var`` arguments. **The one thing that Jinja2
+interpolation can't do is interpolate resolver arguments into a string.** And that's what ``!sub``
+can do. For more information on why Jinja2 can't reference resolvers directly, see
+:ref:`resolution_order`.
+
+The argument to this resolver should be a two-element list: (1) Is the format string, using
+curly-brace templates to indicate variables, and (2) a dictionary where the keys are the format
+string's variable names and the values are the variable values.
+
+Example:
+
+.. code-block:: yaml
+
+   parameters:
+     ConnectionString: !sub
+       - "postgres://{username}:{password}@{hostname}:{port}/{database}"
+       # Notice how we're interpolating a username and database via Jinja2? Technically it's not
+       # necessary to pass them this way. They could be interpolated directly. But it might be
+       # easier to read this way if you pass them explicitly like this. See example below for the
+       # other way this can be done.
+       - username: {{ var.username }}
+         password: !ssm /my/ssm/password
+         hostname: !stack_output my/database/stack.yaml::HostName
+         port: !stack_output my/database/stack.yaml::Port
+         database: {{var.database}}
+
+
+It's relevant to note that this functions similarly to the *more verbose* form of CloudFormation's
+``!Sub`` intrinsic function, where you use a list argument and supply the interpolated values as a
+second list item in a dictionary. **Important**: Sceptre's ``!sub`` resolver will not work without
+a list argument. It does **not** directly reference variables without you directly passing them
+in the second list item in its argument.
+
+You *can* combine Jinja2 syntax with this resolver if you want to interpolate in other variables
+that Jinja2 has access to.
+
+Example:
+
+.. code-block:: yaml
+
+   parameters:
+     ConnectionString: !sub
+       # Notice the double-curly braces. That's Jinja2 syntax. Jinja2 will render the username into
+       # the string even before the yaml is loaded. If you use Jinja2 to interpolate the value, then
+       # it's not a template string variable you need to pass in the second list item passed to
+       # !sub.
+       - "postgres://{{ var.username }}:{password}@{hostname}:{port}/{{ stack_group_config.database }}"
+       - password: !ssm /my/ssm/password
+         hostname: !stack_output my/database/stack.yaml::HostName
+         port: !stack_output my/database/stack.yaml::Port
+
 Custom Resolvers
 ----------------
 
@@ -306,18 +433,22 @@ For details on calling AWS services or invoking AWS-related third party tools in
 
 Resolver arguments
 ^^^^^^^^^^^^^^^^^^
-Resolver arguments can be a simple string or a complex data structure.
+Resolver arguments can be a simple string or a complex data structure. You can even use
+other resolvers in the arguments to resolvers! (Note: Other resolvers can only be passed in
+arguments when they're passed in lists and dicts.)
 
 .. code-block:: yaml
 
    template:
      path: <...>
      type: <...>
-    parameters:
-      Param1: !ssm "/dev/DbPassword"
-      Param2: !ssm {"name": "/dev/DbPassword"}
-      Param3: !ssm
-        name: "/dev/DbPassword"
+   parameters:
+     Param1: !ssm "/dev/DbPassword"
+     Param2: !ssm {"name": "/dev/DbPassword"}
+     Param3: !ssm
+       name: "/dev/DbPassword"
+     Param4: !ssm
+       name: !stack_output my/other/stack.yaml::MySsmParameterName
 
 .. _Custom Resolvers: #custom-resolvers
 .. _this is great place to start: https://docs.python.org/3/distributing/

docs/_source/docs/stack_config.rst

@@ -99,7 +99,7 @@ and that Stack need not be added as an explicit dependency.
 
 hooks
 ~~~~~
-* Resolvable: No
+* Resolvable: No (but you can use resolvers _in_ hook arguments!)
 * Can be inherited from StackGroup: Yes
 * Inheritance strategy: Overrides parent if set
 
@@ -556,7 +556,8 @@ order:
 A common point of confusion tends to be around the distinction between **"render time"** (phase 3, when
 Jinja logic is applied) and **"resolve time"** (phase 6, when resolvers are resolved). You cannot use
 a resolver via Jinja during "render time", since the resolver won't exist or be ready to use yet. You can,
-however, use Jinja logic to indicate *whether*, *which*, or *how* a resolver is configured.
+however, use Jinja logic to indicate *whether*, *which*, or *how* a resolver is configured. You can
+also use resolvers like ``!sub`` to interpolate resolved values when Jinja isn't available.
 
 For example, you **can** do something like this:
 
@@ -566,6 +567,11 @@ For example, you **can** do something like this:
      {% if var.use_my_parameter %}
        my_parameter: !stack_output {{ var.stack_name }}::{{ var.output_name }}
      {% endif %}
+       # !sub will let you combine outputs of multiple resolvers into a single string
+       my_combined_parameter: !sub
+         - "{fist_part} - {second_part}"
+         - first_part: !stack_output my/stack/name.yaml::Output
+         - second_part: {{ var.second_part }}
 
 Accessing resolved values in other fields
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

sceptre/config/reader.py

@@ -182,7 +182,7 @@ def constructor_factory(node_class):
             :rtype: func
             """
 
-            # This function signture is required by PyYAML
+            # This function signature is required by PyYAML
             def class_constructor(loader, node):
                 return node_class(
                     loader.construct_object(self.resolve_node_tag(loader, node))

sceptre/exceptions.py

@@ -219,3 +219,9 @@ class CannotPruneStackError(SceptreException):
     Error raised when an obsolete stack cannot be pruned because another stack depends on it that is
     not itself obsolete.
     """
+
+
+class InvalidResolverArgumentError(SceptreException):
+    """
+    Indicates a resolver argument is invalid in some way.
+    """

sceptre/helpers.py

@@ -2,7 +2,7 @@
 from contextlib import contextmanager
 from datetime import datetime
 from os import sep
-from typing import Optional, Any, List
+from typing import Optional, Any, List, Tuple, Union
 
 import dateutil.parser
 import deprecation
@@ -68,6 +68,28 @@ def func_on_instance(key):
     return attr
 
 
+Container = Union[list, dict]
+Key = Union[str, int]
+
+
+def delete_keys_from_containers(keys_to_delete: List[Tuple[Container, Key]]):
+    """Removes the indicated keys/indexes from their paired containers."""
+    list_items_to_delete = []
+    for container, key in keys_to_delete:
+        if isinstance(container, list):
+            # If it's a list, we want to gather up the items to remove from the list.
+            # We don't want to modify the list length yet, since removals will change all the other
+            # list indexes. Instead, we'll get the actual items at those indexes to remove later.
+            list_items_to_delete.append((container, container[key]))
+        else:
+            del container[key]
+
+    # Finally, now that we have all the items we want to remove the lists, we'll remove those
+    # items specifically from the lists.
+    for containing_list, item in list_items_to_delete:
+        containing_list.remove(item)
+
+
 def normalise_path(path):
     """
     Converts a path to use correct path separator.

sceptre/hooks/__init__.py

@@ -1,39 +1,21 @@
 import abc
 import logging
 from functools import wraps
+from typing import TYPE_CHECKING
 
 from sceptre.helpers import _call_func_on_values
-from sceptre.logging import StackLoggerAdapter
-
-from typing import TYPE_CHECKING, Any
+from sceptre.resolvers import CustomYamlTagBase
 
 if TYPE_CHECKING:
     from sceptre.stack import Stack
 
 
-class Hook(abc.ABC):
+class Hook(CustomYamlTagBase, metaclass=abc.ABCMeta):
     """
-    Hook is an abstract base class that should be inherited by all hooks.
-
-    :param argument: The argument of the hook.
-    :param stack: The associated stack of the hook.
+    Hook is an abstract base class that should be subclassed by all hooks.
     """
 
-    def __init__(self, argument: Any = None, stack: "Stack" = None):
-        self.logger = logging.getLogger(__name__)
-
-        if stack is not None:
-            self.logger = StackLoggerAdapter(self.logger, stack.name)
-
-        self.argument = argument
-        self.stack = stack
-
-    def setup(self):
-        """
-        setup is a method that may be overwritten by inheriting classes. Allows
-        hooks to run so initalisation steps when config is first read.
-        """
-        pass  # pragma: no cover
+    logger = logging.getLogger(__name__)
 
     @abc.abstractmethod
     def run(self):
@@ -43,15 +25,6 @@ def run(self):
         """
         pass  # pragma: no cover
 
-    def clone(self, stack: "Stack") -> "Hook":
-        """
-        Produces a "fresh" copy of the Hook, with the specified stack.
-
-        :param stack: The stack to set on the cloned resolver
-        """
-        clone = type(self)(self.argument, stack)
-        return clone
-
 
 class HookProperty(object):
     """
@@ -84,7 +57,7 @@ def __set__(self, instance: "Stack", value):
         """
 
         def setup(attr, key, value: Hook):
-            attr[key] = clone = value.clone(instance)
+            attr[key] = clone = value.clone_for_stack(instance)
             clone.setup()
 
         _call_func_on_values(setup, value, Hook)