[Resolve #1114,#1000] Project Dependencies part 4: Handling resolvers…

… referencing non-deployed stacks in template commands (#1185)

This is the Fourth in a series of pull requests that addresses #1114 , allowing Sceptre to manage its own dependencies.

## In this PR:
* It is possible to enter a global state (using a context manager) that allows Resolvers that raise errors to resolve to a placeholder value. This is explicitly opt-in and will only be enabled on commands that don't actually deploy stacks, such as "generate", "validate", and "diff". This is required so that these commands don't fail when there are resolvers that reference stacks that aren't deployed yet. This ultimately is the bit that resolves issue #1000. This is important because this series of Pull requests more than doubles the number of stack attributes that are resolvable, which are going to be all the more problematic when using `sceptre validate`, etc...

docs/_source/docs/resolvers.rst

@@ -52,6 +52,26 @@ file_contents
 
 **deprecated**: Consider using the `file`_ resolver instead.
 
+no_value
+~~~~~~~~
+
+This resolver "resolves to nothing", functioning just as if it was not set at all. This works just
+like the "AWS::NoValue" special variable that you can reference on a CloudFormation template. It
+can help simplify Stack and StackGroup config Jinja logic in cases where, if a condition is met, a
+value is passed, otherwise no value is passed.
+
+For example, you could use this resolver like this:
+
+.. code-block:: yaml
+
+   parameters:
+    my_parameter: {{ var.some_value_that_might_not_be_set | default('!no_value') }}
+
+In this example, if ``var.some_value_that_might_not_be_set`` is set, ``my_parameter`` will be set to
+that value. But if ``var.some_value_that_might_not_be_set`` is not actually set, ``my_parameter``
+won't even be passed to CloudFormation at all. This might be desired if there is a default value on
+the CloudFormation template for ``my_parameter`` and we'd want to fall back to that default.
+
 rcmd
 ~~~~
 
@@ -162,15 +182,23 @@ custom_resolver.py
 
             Parameters
             ----------
-            argument: str
-                The argument of the resolver.
+            argument: Any
+                The argument of the resolver. This can be any value able to be defined in yaml.
             stack: sceptre.stack.Stack
-                The associated stack of the resolver.
-
+                The associated stack of the resolver. This will normally be None when the resolver is
+                instantiated, but will be set before the resolver is resolved.
             """
 
-            def __init__(self, *args, **kwargs):
-                super(CustomResolver, self).__init__(*args, **kwargs)
+            def __init__(self, argument, stack=None):
+                super(CustomResolver, self).__init__(argument, stack)
+
+            def setup(self):
+                """
+                Setup is invoked after the stack has been set on the resolver, whether or not the
+                resolver is ever resolved.
+
+                Implement this method for any setup behavior you want (such as adding to stack dependencies).
+                """
 
             def resolve(self):
                 """
@@ -223,7 +251,7 @@ This resolver can be used in a Stack config file with the following syntax:
      param1: !<custom_resolver_command_name> <value> <optional-aws-profile>
 
 
-resolver arguments
+Resolver arguments
 ^^^^^^^^^^^^^^^^^^
 Resolver arguments can be a simple string or a complex data structure.
 
@@ -240,3 +268,53 @@ Resolver arguments can be a simple string or a complex data structure.
 
 .. _Custom Resolvers: #custom-resolvers
 .. _this is great place to start: https://docs.python.org/3/distributing/
+
+Resolving to nothing
+^^^^^^^^^^^^^^^^^^^^
+When a resolver returns ``None``, this means that it resolves to "nothing". For resolvers set for
+single values (such as for ``template_bucket_name`` or ``role_arn``), this just means the value is
+``None`` and treated like those values aren't actually set. But for resolvers inside of containers
+like lists or dicts, when they resolve to "nothing", that item gets completely removed from their
+containing list or dict.
+
+This feature would be useful if you wanted to define a resolver that sometimes would resolve to be a
+given stack parameter and sometimes would be not defined at all and use the template's default value
+for that parameter. The resolver could just return `None` in those cases it wants to resolve to
+nothing, similar to the AWS::NoValue pseudo-parameter that can be referenced in a CloudFormation
+template.
+
+Resolver placeholders
+^^^^^^^^^^^^^^^^^^^^^
+Resolvers (especially the !stack_output resolver) often express dependencies on other stacks and
+their outputs. However, there are times when those stacks or outputs will not exist yet because they
+have not yet been deployed. During normal deployment operations (using the ``launch``, ``create``,
+``update``, and ``delete`` commands), Sceptre knows the correct order to resolve dependencies in and will
+ensure that order is followed, so everything works as expected.
+
+But there are other commands that will not actually deploy dependencies of a stack config before
+operating on that Stack Config. These commands include ``generate``, ``validate``, and ``diff``.
+If you have used resolvers to reverence other stacks, it is possible that a resolver might not be able
+to be resolved when performing that command's operations and will trigger an error. This is not likely
+to happen when you have only used resolvers in a stack's ``parameters``, but it is much more likely
+if you have used them in ``sceptre_user_data`` with a Jinja or Python template. At those times (and
+only when a resolver cannot be resolved), a **best-attempt placeholder value** will be supplied in to
+allow the command to proceed. Depending on how your template or Stack Config is configured, the
+command may or may not actually succeed using that placeholder value.
+
+A few examples...
+
+* If you have a stack parameter referencing ``!stack_output other_stack.yaml::OutputName``,
+  and you run the ``diff`` command before other_stack.yaml has been deployed, the diff output will
+  show the value of that parameter to be ``"{ !StackOutput(other_stack.yaml::OutputName) }"``.
+* If you have a ``sceptre_user_data`` value used in a Jinja template referencing
+  ``!stack_output other_stack.yaml::OutputName`` and you run the ``generate`` command, the generated
+  template will replace that value with ``"StackOutputotherstackyamlOutputName"``. This isn't as
+  "pretty" as the sort of placeholder used for stack parameters, but the use of sceptre_user_data is
+  broader, so it placeholder values can only be alphanumeric to reduce chances of it breaking the
+  template.
+* Resolvable properties that are *always* used when performing template operations (like ``iam_role``
+  and ``template_bucket_name``) will resolve to ``None`` and not be used for those operations if they
+  cannot be resolved.
+
+Any command that allows these placeholders can have them disabled with the ``--no-placeholders`` ClI
+option.

docs/_source/docs/stack_group_config.rst

@@ -73,21 +73,25 @@ supplied in this way have a lower maximum length, so using the
    If you resolve ``template_bucket_name`` using the ``!stack_output``
    resolver on a StackGroup, the stack that outputs that bucket name *cannot* be
    defined in that StackGroup. Otherwise, a circular dependency will exist and Sceptre
-   will raise an error when attempting any Stack action. The proper way to do this would
-   be to define all your project stacks inside a StackGroup and then your template bucket
-   stack *outside* that StackGroup. Here's an example project structure for something like
-   this:
-
-   .. code-block:: yaml
-
-      config/
-          - config.yaml           # This is the StackGroup Config for your whole project.
-          - template-bucket.yaml  # The template for this stack outputs the bucket name
-          - project/              # You can put all your other stacks in this StackGroup
-              - config.yaml       # In this StackGroup Config is...
-                                  #  template_bucket_name: !stack_output template-bucket.yaml::BucketName
-              - vpc.yaml          # Put all your other project stacks inside project/
-              - other-stack.yaml
+   will raise an error when attempting any Stack action. There are two ways to avoid this situation:
+
+   1. Set the ``template_bucket_name`` to ``!no_value`` in on the StackConfig that creates your
+      template bucket. This will override the inherited value to prevent them from having
+      dependencies on themselves.
+   2. Define all your project stacks inside a StackGroup and then your template bucket
+      stack *outside* that StackGroup. Here's an example project structure for something like
+      this:
+
+      .. code-block:: yaml
+
+         config/
+           - config.yaml           # This is the StackGroup Config for your whole project.
+           - template-bucket.yaml  # The template for this stack outputs the bucket name
+           - project/              # You can put all your other stacks in this StackGroup
+               - config.yaml       # In this StackGroup Config is...
+                                   #  template_bucket_name: !stack_output template-bucket.yaml::BucketName
+               - vpc.yaml          # Put all your other project stacks inside project/
+               - other-stack.yaml
 
 
 template_key_prefix
@@ -117,7 +121,7 @@ j2_environment
 
 A dictionary that is combined with the default jinja2 environment.
 It's converted to keyword arguments then passed to [jinja2.Environment](https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.Environment).
-This will impact :ref:`Templating` of stacks by modifying the behavior of jinja.
+This will impact the templating of stacks by modifying the behavior of jinja.
 
 .. code-block:: yaml
 
@@ -209,13 +213,16 @@ dependencies.
 
    You might have already considered that this might cause a circular dependency for those
    dependency stacks, the ones that output the template bucket name, role arn, iam_role, or topic arns.
-   In order to avoid the circular dependency issue, it is important that you define these items in a
-   Stack that is *outside* the StackGroup you reference them in. Here's an example project structure
-   that would support doing this:
+   In order to avoid the circular dependency issue, you can either:
+
+   1. Set the value of those configurations to ``!no_value`` in the actual stacks that define those
+      items so they don't inherit a dependency on themselves.
+   2. Define those stacks *outside* the StackGroup you reference them in. Here's an example project
+      structure that would support doing this:
 
-   .. code-block:: yaml
+      .. code-block:: yaml
 
-      config/
+        config/
           - config.yaml               # This is the StackGroup Config for your whole project.
           - sceptre-dependencies.yaml # This stack defines your template bucket, iam role, topics, etc...
           - project/                  # You can put all your other stacks in this StackGroup

docs/_source/docs/templates.rst

@@ -74,7 +74,7 @@ Stack:
 
 Template `dns-extras.j2`:
 
-.. code-block:: yaml
+.. code-block:: jinja
 
     AWSTemplateFormatVersion: '2010-09-09'
     Description: 'Add Route53 - CNAME and ALIAS records'

integration-tests/features/project-dependencies.feature

@@ -1,4 +1,4 @@
-Feature: StackGroup Dependencies managed within Sceptre
+Feature: Project Dependencies managed within Sceptre
 
   Background:
     Given stack_group "project-deps" does not exist
@@ -17,3 +17,19 @@ Feature: StackGroup Dependencies managed within Sceptre
     Given all files in template bucket for stack "project-deps/main-project/resource" are deleted at cleanup
     When the user launches stack_group "project-deps"
     Then the stack "project-deps/main-project/resource" has a notification defined by stack "project-deps/dependencies/topic"
+
+  Scenario: validate a project that isn't deployed yet
+    Given placeholders are allowed
+    When the user validates stack_group "project-deps"
+    Then the user is told "the template is valid"
+
+  Scenario: diff a project that isn't deployed yet
+    Given placeholders are allowed
+    When the user diffs stack group "project-deps" with "deepdiff"
+    Then a diff is returned with "is_deployed" = "False"
+
+  Scenario: tags can be resolved
+    Given all files in template bucket for stack "project-deps/main-project/resource" are deleted at cleanup
+    When the user launches stack_group "project-deps"
+    Then the tag "greeting" for stack "project-deps/main-project/resource" is "hello"
+    And the tag "nonexistant" for stack "project-deps/main-project/resource" does not exist

integration-tests/sceptre-project/config/project-deps/main-project/config.yaml

@@ -4,4 +4,5 @@ notifications:
 
 iam_role: !stack_output project-deps/dependencies/assumed-role.yaml::RoleArn
 stack_tags:
-  greeting: !rcmd echo "hello"
+  greeting: !rcmd "echo 'hello' | tr -d '\n'"
+  nonexistant: !no_value

integration-tests/steps/project_dependencies.py

@@ -1,12 +1,14 @@
 from itertools import chain
+from typing import ContextManager, Dict
 
 import boto3
-from behave import given, then
+from behave import given, then, when
 from behave.runner import Context
 
 from helpers import get_cloudformation_stack_name, retry_boto_call
 from sceptre.context import SceptreContext
 from sceptre.plan.plan import SceptrePlan
+from sceptre.resolvers.placeholders import use_resolver_placeholders_on_error
 
 
 @given('all files in template bucket for stack "{stack_name}" are deleted at cleanup')
@@ -21,6 +23,24 @@ def step_impl(context: Context, stack_name):
     )
 
 
+@given('placeholders are allowed')
+def step_impl(context: Context):
+    placeholder_context = use_resolver_placeholders_on_error()
+    placeholder_context.__enter__()
+    context.add_cleanup(exit_placeholder_context, placeholder_context)
+
+
+@when('the user validates stack_group "{group}"')
+def step_impl(context: Context, group):
+    sceptre_context = SceptreContext(
+        command_path=group,
+        project_path=context.sceptre_dir
+    )
+    plan = SceptrePlan(sceptre_context)
+    result = plan.validate()
+    context.response = result
+
+
 @then('the template for stack "{stack_name}" has been uploaded')
 def step_impl(context: Context, stack_name):
     sceptre_context = SceptreContext(
@@ -58,6 +78,19 @@ def step_impl(context, resource_stack_name, topic_stack_name):
     assert topic in notification_arns
 
 
+@then('the tag "{key}" for stack "{stack_name}" is "{value}"')
+def step_impl(context, key, stack_name, value):
+    stack_tags = get_stack_tags(context, stack_name)
+    result = stack_tags[key]
+    assert result == value
+
+
+@then('the tag "{key}" for stack "{stack_name}" does not exist')
+def step_impl(context, key, stack_name):
+    stack_tags = get_stack_tags(context, stack_name)
+    assert key not in stack_tags
+
+
 def cleanup_template_files_in_bucket(sceptre_dir, stack_name):
     sceptre_context = SceptreContext(
         command_path=stack_name + '.yaml',
@@ -87,10 +120,23 @@ def get_stack_resources(context, stack_name):
     return resources['StackResources']
 
 
-def describe_stack(context, stack_name):
+def get_stack_tags(context, stack_name) -> Dict[str, str]:
+    description = describe_stack(context, stack_name)
+    tags = {
+        tag['Key']: tag['Value']
+        for tag in description['Tags']
+    }
+    return tags
+
+
+def describe_stack(context, stack_name) -> dict:
     cf_stack_name = get_cloudformation_stack_name(context, stack_name)
     response = retry_boto_call(
         context.client.describe_stacks,
         StackName=cf_stack_name
     )
     return response['Stacks'][0]
+
+
+def exit_placeholder_context(placeholder_context: ContextManager):
+    placeholder_context.__exit__(None, None, None)