[Resolve #1114,#886,#491] Project Dependencies part 2: Resolvable rol…

…e_arn and template_bucket_name (#1153)

This is the second in a series of pull requests that addresses #1114 , allowing Sceptre to manage its own dependencies.

## In this PR:
* The ResolvableValueProperty is created, creating a property that can resolve to a single value (not a list or dict)
* role_arn and template_bucket_name are now fully resolvable properties
    - role_arn also resolves the issue #886 
    - template_bucket_name also provides a solution to the issue #491 
* If template_bucket_name is set to None on a stack, it will be interpreted as if it wasn't there at all. _This will be important in future Pull Requests so that if template_bucket_name is inherited from the stack_group, it won't actually create a circular dependency on the stack that outputs that bucket name._
* Documentation is clarified on setting stack dependencies on a StackGroup to be shared across all stacks in that group as well as a warning about circular dependencies.

docs/_source/docs/stack_config.rst

@@ -213,11 +213,11 @@ throw an error.
 
 role_arn
 ~~~~~~~~
-* Resolvable: No
+* Resolvable: Yes
 * Can be inherited from StackGroup: Yes
 * Inheritance strategy: Overrides parent if set
 
-The ARN of a `CloudFormation Service Role`_ that is assumed by CloudFormation
+The ARN of a `CloudFormation Service Role`_ that is assumed by *CloudFormation* (not Sceptre)
 to create, update or delete resources.
 
 iam_role

docs/_source/docs/stack_group_config.rst

@@ -54,7 +54,7 @@ supports CloudFormation`_.
 
 template_bucket_name
 ~~~~~~~~~~~~~~~~~~~~
-* Resolvable: No
+* Resolvable: Yes
 * Inheritance strategy: Overrides parent if set by child
 
 The name of an S3 bucket to upload CloudFormation Templates to. Note that S3
@@ -66,6 +66,28 @@ supplies the template to Boto3 via the ``TemplateBody`` argument. Templates
 supplied in this way have a lower maximum length, so using the
 ``template_bucket_name`` parameter is recommended.
 
+.. warning::
+
+   If you resolve ``template_bucket_name`` using the ``!stack_output``
+   resolver on a StackGroup, the stack that outputs that bucket name *cannot* be
+   defined in that StackGroup. Otherwise, a circular dependency will exist and Sceptre
+   will raise an error when attempting any Stack action. The proper way to do this would
+   be to define all your project stacks inside a StackGroup and then your template bucket
+   stack *outside* that StackGroup. Here's an example project structure for something like
+   this:
+
+   .. code-block:: yaml
+
+      config/
+          - config.yaml           # This is the StackGroup Config for your whole project.
+          - template-bucket.yaml  # The template for this stack outputs the bucket name
+          - project/              # You can put all your other stacks in this StackGroup
+              - config.yaml       # In this StackGroup Config is...
+                                  #  template_bucket_name: !stack_output template-bucket.yaml::BucketName
+              - vpc.yaml          # Put all your other project stacks inside project/
+              - other-stack.yaml
+
+
 template_key_prefix
 ~~~~~~~~~~~~~~~~~~~
 * Resolvable: No
@@ -157,6 +179,50 @@ For example, if you wanted the ``dev`` StackGroup to build to a different
 region, this setting could be specified in the ``config/dev/config.yaml`` file,
 and would only be applied to builds in the ``dev`` StackGroup.
 
+.. _setting_dependencies_for_stack_groups:
+
+Setting Dependencies for StackGroups
+------------------------------------
+There are a few pieces of AWS infrastructure that Sceptre can (optionally) use to support the needs
+and concerns of the project. These include:
+
+* The S3 bucket where templates are uploaded to and then referenced from for stack actions (i.e. the
+  ``template_bucket_name`` config key).
+* The CloudFormation service role added to the stack(s) that CloudFormation uses to execute stack
+  actions (i.e. the ``role_arn`` config key).
+* The role that Sceptre will assume to execute stack actions (i.e. the ``iam_role`` config key).
+* SNS topics that cloudformation will notify with the results of stack actions (i.e. the
+  ``notifications`` config key).
+
+These sorts of dependencies CAN be defined in Sceptre and added at the StackGroup level, referenced
+using ``!stack_output``. Doing so will make it so that every stack in the StackGroup will have those
+dependencies and get those values from Sceptre-managed stacks.
+
+Beyond the above mentioned config keys, it is possible to set the ``dependencies`` config key in a
+StackGroup config to be inherited by all Stack configs in that group. All dependencies in child
+stacks will be added to their inherited StackGroup dependencies, so be careful how you structure
+dependencies.
+
+.. warning::
+
+   You might have already considered that this might cause a circular dependency for those
+   dependency stacks, the ones that output the template bucket name, role arn, iam_role, or topic arns.
+   In order to avoid the circular dependency issue, it is important that you define these items in a
+   Stack that is *outside* the StackGroup you reference them in. Here's an example project structure
+   that would support doing this:
+
+   .. code-block:: yaml
+
+      config/
+          - config.yaml               # This is the StackGroup Config for your whole project.
+          - sceptre-dependencies.yaml # This stack defines your template bucket, iam role, topics, etc...
+          - project/                  # You can put all your other stacks in this StackGroup
+              - config.yaml           # In this StackGroup Config you can use !stack_output to
+                                      # reference outputs from sceptre-dependencies.yaml.
+              - vpc.yaml              # Put all your other project stacks inside project/
+              - other-stack.yaml
+
+
 .. _stack_group_config_templating:
 
 Templating

integration-tests/features/dependency-resolution.feature

@@ -2,17 +2,17 @@ Feature: Dependency resolution
 
   Scenario: launch a stack_group with dependencies that is partially complete
     Given stack "3/A" exists in "CREATE_COMPLETE" state
-    and stack "3/B" exists in "CREATE_COMPLETE" state
-    and stack "3/C" does not exist
+    And stack "3/B" exists in "CREATE_COMPLETE" state
+    And stack "3/C" does not exist
     When the user launches stack_group "3"
     Then all the stacks in stack_group "3" are in "CREATE_COMPLETE"
-    and that stack "3/A" was created before "3/B"
-    and that stack "3/B" was created before "3/C"
+    And that stack "3/A" was created before "3/B"
+    And that stack "3/B" was created before "3/C"
 
   Scenario: delete a stack_group with dependencies that is partially complete
     Given stack "3/A" exists in "CREATE_COMPLETE" state
-    and stack "3/B" exists in "CREATE_COMPLETE" state
-    and stack "3/C" does not exist
+    And stack "3/B" exists in "CREATE_COMPLETE" state
+    And stack "3/C" does not exist
     When the user deletes stack_group "3"
     Then all the stacks in stack_group "3" do not exist
 

integration-tests/features/project-dependencies.feature

@@ -0,0 +1,19 @@
+Feature: StackGroup Dependencies managed within Sceptre
+
+  Background:
+    Given stack_group "project-deps" does not exist
+
+  Scenario: launch stack group with dependencies
+    Given all files in template bucket for stack "project-deps/main-project/resource" are deleted at cleanup
+    When the user launches stack_group "project-deps"
+    Then all the stacks in stack_group "project-deps" are in "CREATE_COMPLETE"
+
+  Scenario: template_bucket_name is managed in stack group
+    Given all files in template bucket for stack "project-deps/main-project/resource" are deleted at cleanup
+    When the user launches stack_group "project-deps"
+    Then the template for stack "project-deps/main-project/resource" has been uploaded
+
+  Scenario: notifications are managed in stack group
+    Given all files in template bucket for stack "project-deps/main-project/resource" are deleted at cleanup
+    When the user launches stack_group "project-deps"
+    Then the stack "project-deps/main-project/resource" has a notification defined by stack "project-deps/dependencies/topic"

integration-tests/features/validate-template.feature

@@ -11,7 +11,7 @@ Feature: Validate template
     Then a "ClientError" is raised
     and the user is told "the template is malformed"
 
-  Scenario: validate a vaild template with ignore dependencies
+  Scenario: validate a valid template with ignore dependencies
     Given the template for stack "1/A" is "valid_template.json"
     When the user validates the template for stack "1/A" with ignore dependencies
     Then the user is told "the template is valid"

integration-tests/sceptre-project/config/project-deps/dependencies/bucket.yaml

@@ -0,0 +1,2 @@
+template:
+  path: project-dependencies/bucket.yaml

integration-tests/sceptre-project/config/project-deps/dependencies/topic.yaml

@@ -0,0 +1,2 @@
+template:
+  path: project-dependencies/topic.yaml

integration-tests/sceptre-project/config/project-deps/main-project/config.yaml

@@ -0,0 +1,5 @@
+template_bucket_name: !stack_output project-deps/dependencies/bucket.yaml::BucketName
+notifications:
+  - !stack_output project-deps/dependencies/topic.yaml::TopicArn
+stack_tags:
+  greeting: !rcmd echo "hello"

integration-tests/sceptre-project/config/project-deps/main-project/resource.yaml

@@ -0,0 +1,2 @@
+template:
+  path: "valid_template.yaml"

integration-tests/sceptre-project/templates/jinja/valid_template.j2

@@ -1,4 +1,4 @@
 Resources:
     WaitConditionHandle:
       Type: "{{ sceptre_user_data.type }}"
-      Properties:
+      Properties: {}

integration-tests/sceptre-project/templates/project-dependencies/bucket.yaml

@@ -0,0 +1,10 @@
+AWSTemplateFormatVersion: "2010-09-09"
+
+Resources:
+  Bucket:
+    Type: AWS::S3::Bucket
+    Properties: { }
+
+Outputs:
+  BucketName:
+    Value: !Ref Bucket

integration-tests/sceptre-project/templates/project-dependencies/topic.yaml

@@ -0,0 +1,10 @@
+AWSTemplateFormatVersion: "2010-09-09"
+
+Resources:
+  Topic:
+    Type: AWS::SNS::Topic
+    Properties: {}
+
+Outputs:
+  TopicArn:
+    Value: !Ref Topic

integration-tests/sceptre-project/templates/valid_template.yaml

@@ -1,4 +1,4 @@
 Resources:
   WaitConditionHandle:
     Type: "AWS::CloudFormation::WaitConditionHandle"
-    Properties:
+    Properties: {}

integration-tests/steps/project_dependencies.py

@@ -0,0 +1,96 @@
+from itertools import chain
+
+import boto3
+from behave import given, then
+from behave.runner import Context
+
+from helpers import get_cloudformation_stack_name, retry_boto_call
+from sceptre.context import SceptreContext
+from sceptre.plan.plan import SceptrePlan
+
+
+@given('all files in template bucket for stack "{stack_name}" are deleted at cleanup')
+def step_impl(context: Context, stack_name):
+    """Add this as a given to ensure that the template bucket is cleaned up before we attempt to
+    delete it; Otherwise, it will fail since you can't delete a bucket with objects in it.
+    """
+    context.add_cleanup(
+        cleanup_template_files_in_bucket,
+        context.sceptre_dir,
+        stack_name
+    )
+
+
+@then('the template for stack "{stack_name}" has been uploaded')
+def step_impl(context: Context, stack_name):
+    sceptre_context = SceptreContext(
+        command_path=stack_name + '.yaml',
+        project_path=context.sceptre_dir
+    )
+    plan = SceptrePlan(sceptre_context)
+    buckets = get_template_buckets(plan)
+    assert len(buckets) > 0
+    filtered_objects = list(chain.from_iterable(
+        bucket.objects.filter(
+            Prefix=stack_name
+        )
+        for bucket in buckets
+    ))
+
+    assert len(filtered_objects) == len(plan.command_stacks)
+    for stack in plan.command_stacks:
+        for obj in filtered_objects:
+            if obj.key.startswith(stack.name):
+                s3_template = obj.get()['Body'].read().decode('utf-8')
+                expected = stack.template.body
+                assert s3_template == expected
+                break
+        else:
+            assert False, "Could not found uploaded template"
+
+
+@then('the stack "{resource_stack_name}" has a notification defined by stack "{topic_stack_name}"')
+def step_impl(context, resource_stack_name, topic_stack_name):
+    topic_stack_resources = get_stack_resources(context, topic_stack_name)
+    topic = topic_stack_resources[0]['PhysicalResourceId']
+    resource_stack = describe_stack(context, resource_stack_name)
+    notification_arns = resource_stack['NotificationARNs']
+    assert topic in notification_arns
+
+
+def cleanup_template_files_in_bucket(sceptre_dir, stack_name):
+    sceptre_context = SceptreContext(
+        command_path=stack_name + '.yaml',
+        project_path=sceptre_dir
+    )
+    plan = SceptrePlan(sceptre_context)
+    buckets = get_template_buckets(plan)
+    for bucket in buckets:
+        bucket.objects.delete()
+
+
+def get_template_buckets(plan: SceptrePlan):
+    s3_resource = boto3.resource('s3')
+    return [
+        s3_resource.Bucket(stack.template_bucket_name)
+        for stack in plan.command_stacks
+        if stack.template_bucket_name is not None
+    ]
+
+
+def get_stack_resources(context, stack_name):
+    cf_stack_name = get_cloudformation_stack_name(context, stack_name)
+    resources = retry_boto_call(
+        context.client.describe_stack_resources,
+        StackName=cf_stack_name
+    )
+    return resources['StackResources']
+
+
+def describe_stack(context, stack_name):
+    cf_stack_name = get_cloudformation_stack_name(context, stack_name)
+    response = retry_boto_call(
+        context.client.describe_stacks,
+        StackName=cf_stack_name
+    )
+    return response['Stacks'][0]

integration-tests/steps/stack_groups.py

@@ -1,11 +1,12 @@
-from behave import *
 import os
 import time
-from sceptre.plan.plan import SceptrePlan
-from sceptre.context import SceptreContext
+
+from behave import *
 from botocore.exceptions import ClientError
-from helpers import read_template_file, get_cloudformation_stack_name
-from helpers import retry_boto_call
+
+from helpers import read_template_file, get_cloudformation_stack_name, retry_boto_call
+from sceptre.context import SceptreContext
+from sceptre.plan.plan import SceptrePlan
 from stacks import wait_for_final_state
 from templates import set_template_path
 
@@ -201,10 +202,12 @@ def step_impl(context):
 
 @then('stack "{stack_name}" is described as "{status}"')
 def step_impl(context, stack_name, status):
-    response = next((
-        stack for stack in context.response
-        if stack_name in stack
-    ), {stack_name: 'PENDING'})
+    response = next(
+        (
+            stack for stack in context.response
+            if stack_name in stack
+        ), {stack_name: 'PENDING'}
+    )
 
     assert response[stack_name] == status
 
@@ -306,8 +309,10 @@ def create_stacks(context, stack_names):
                 TemplateBody=body
             )
         except ClientError as e:
-            if e.response['Error']['Code'] == 'AlreadyExistsException' \
-                    and e.response['Error']['Message'].endswith("already exists"):
+            if (
+                e.response['Error']['Code'] == 'AlreadyExistsException'
+                and e.response['Error']['Message'].endswith("already exists")
+            ):
                 pass
             else:
                 raise e

integration-tests/steps/stacks.py

@@ -505,7 +505,7 @@ def delete_stack(context, stack_name):
     retry_boto_call(stack.delete)
 
     waiter = context.client.get_waiter('stack_delete_complete')
-    waiter.config.delay = 4
+    waiter.config.delay = 5
     waiter.config.max_attempts = 240
     waiter.wait(StackName=stack_name)
 

sceptre/config/reader.py

@@ -475,7 +475,9 @@ def _collect_s3_details(stack_name, config):
         :rtype: dict
         """
         s3_details = None
-        if "template_bucket_name" in config:
+        # If the config explicitly sets the template_bucket_name to None, we don't want to enter
+        # this conditional block.
+        if config.get("template_bucket_name") is not None:
             template_key = "/".join([
                 sceptreise_path(stack_name), "{time_stamp}.json".format(
                     time_stamp=datetime.datetime.utcnow().strftime(