[Resolve #1318] Implement dump template and write output files (#1325)

# Description Proposed implementation of #1318 * This introduces a `dump template` command that behaves similarly to `generate` but also writes output docs to files. * The behaviour of `dump config` is changed here not to print multi-doc output to screen similar to `dump config`. * The `write` helper function is modified to optionally write to a `file_path`. * This also deprecates the generate command in favor of dump command Writes files in a naming convention for `dump config` and `dump template` respectively: ``` f".dump/{stack_name}/config.yaml" f".dump/{stack_name}/template.yaml" ``` # Usage example ## Dump config ``` % sceptre --merge-vars --dir=emr --var-file=/app/sceptre-environment/nonprod/datalake-nonprod/common-env.yaml --var-file=/app/sceptre-environment/common-env.yaml --var-file=/app/sceptre-environment/nonprod/datalake-nonprod/emr/streaming-nonprod-6100/cluster.yaml dump config emr.yaml > /dev/null ``` ## Dump template ``` % sceptre --merge-vars --dir=emr --var-file=/app/sceptre-environment/nonprod/datalake-nonprod/common-env.yaml --var-file=/app/sceptre-environment/common-env.yaml --var-file=/app/sceptre-environment/nonprod/datalake-nonprod/emr/streaming-nonprod-6100/cluster.yaml dump template emr.yaml > /dev/null ``` ## File structure created ``` % tree .dump/ .dump/ ├── streaming-nonprod-6100-efs-pvt │ ├── config.yaml │ └── template.yaml ├── streaming-nonprod-6100-emr │ ├── config.yaml │ └── template.yaml ├── streaming-nonprod-6100-emr-profile │ ├── config.yaml │ └── template.yaml └── streaming-nonprod-6100-emr-sg-pvt ├── config.yaml └── template.yaml 4 directories, 8 files ``` ## File content ```yaml % head .dump/streaming-nonprod-6100-emr/config.yaml --- dependencies: - emr-profile.yaml hooks: after_create: - !!python/object:hook.stack_termination_protection.StackTerminationProtection _argument: enabled _argument_is_resolved: false logger: &id001 !!python/object/apply:logging.getLogger - hook.stack_termination_protection ``` ```yaml % head .dump/streaming-nonprod-6100-emr/template.yaml --- AWSTemplateFormatVersion: '2010-09-09' Parameters: MasterInstanceType: Type: String CoreInstantType: Type: String CoreInstantCount: Type: String ```
Sceptre · May 14, 2023 · dcae1a9 · dcae1a9
1 parent ef9dc21
commit dcae1a9
Show file tree

Hide file tree

Showing 13 changed files with 199 additions and 59 deletions.
diff --git a/docs/_source/docs/resolvers.rst b/docs/_source/docs/resolvers.rst
@@ -476,7 +476,7 @@ have not yet been deployed. During normal deployment operations (using the ``lau
 ensure that order is followed, so everything works as expected.
 
 But there are other commands that will not actually deploy dependencies of a stack config before
-operating on that Stack Config. These commands include ``generate``, ``validate``, and ``diff``.
+operating on that Stack Config. These commands include ``dump template``, ``validate``, and ``diff``.
 If you have used resolvers to reverence other stacks, it is possible that a resolver might not be able
 to be resolved when performing that command's operations and will trigger an error. This is not likely
 to happen when you have only used resolvers in a stack's ``parameters``, but it is much more likely
@@ -491,7 +491,7 @@ A few examples...
   and you run the ``diff`` command before other_stack.yaml has been deployed, the diff output will
   show the value of that parameter to be ``"{ !StackOutput(other_stack.yaml::OutputName) }"``.
 * If you have a ``sceptre_user_data`` value used in a Jinja template referencing
-  ``!stack_output other_stack.yaml::OutputName`` and you run the ``generate`` command, the generated
+  ``!stack_output other_stack.yaml::OutputName`` and you run the ``dump template`` command, the generated
   template will replace that value with ``"StackOutputotherstackyamlOutputName"``. This isn't as
   "pretty" as the sort of placeholder used for stack parameters, but the use of sceptre_user_data is
   broader, so it placeholder values can only be alphanumeric to reduce chances of it breaking the

diff --git a/...sts/features/generate-template-s3.feature → ...n-tests/features/dump-template-s3.feature b/...sts/features/generate-template-s3.feature → ...n-tests/features/dump-template-s3.feature
@@ -1,17 +1,17 @@
 @s3-template-handler
-Feature: Generate template s3
+Feature: Dump template s3
 
-  Scenario: Generating static templates with S3 template handler
+  Scenario: Dumping static templates with S3 template handler
     Given the template for stack "13/B" is "valid_template.json"
-    When the user generates the template for stack "13/B"
+    When the user dumps the template for stack "13/B"
     Then the output is the same as the contents of "valid_template.json" template
 
   Scenario: Render jinja templates with S3 template handler
     Given the template for stack "13/C" is "jinja/valid_template.j2"
-    When the user generates the template for stack "13/C"
+    When the user dumps the template for stack "13/C"
     Then the output is the same as the contents of "valid_template.json" template
 
   Scenario: Render python templates with S3 template handler
     Given the template for stack "13/D" is "python/valid_template.py"
-    When the user generates the template for stack "13/D"
+    When the user dumps the template for stack "13/D"
     Then the output is the same as the contents of "valid_template.json" template
diff --git a/...-tests/features/generate-template.feature → ...tion-tests/features/dump-template.feature b/...-tests/features/generate-template.feature → ...tion-tests/features/dump-template.feature
@@ -1,8 +1,8 @@
-Feature: Generate template
+Feature: Dump template
 
-  Scenario Outline: Generating static templates
+  Scenario Outline: Dumping static templates
     Given the template for stack "1/A" is "<filename>"
-    When the user generates the template for stack "1/A"
+    When the user dumps the template for stack "1/A"
     Then the output is the same as the contents of "<filename>" template
 
   Examples: Json, Yaml
@@ -20,22 +20,22 @@ Feature: Generate template
 
   Scenario: Generate template using a valid python template file that outputs json
     Given the template for stack "1/A" is "valid_template_json.py"
-    When the user generates the template for stack "1/A"
+    When the user dumps the template for stack "1/A"
     Then the output is the same as the contents returned by "valid_template_json.py"
 
   Scenario: Generate template using a valid python template file that outputs json with ignore dependencies
     Given the template for stack "1/A" is "valid_template_json.py"
-    When the user generates the template for stack "1/A" with ignore dependencies
+    When the user dumps the template for stack "1/A" with ignore dependencies
     Then the output is the same as the contents returned by "valid_template_json.py"
 
   Scenario: Generate template using a valid python template file that outputs yaml
     Given the template for stack "1/A" is "valid_template_yaml.py"
-    When the user generates the template for stack "1/A"
+    When the user dumps the template for stack "1/A"
     Then the output is the same as the contents returned by "valid_template_yaml.py"
 
-  Scenario Outline: Generating erroneous python templates
+  Scenario Outline: Dumping erroneous python templates
     Given the template for stack "1/A" is "<filename>"
-    When the user generates the template for stack "1/A"
+    When the user dumps the template for stack "1/A"
     Then a "<exception>" is raised
 
   Examples: Template Errors
@@ -45,12 +45,12 @@ Feature: Generate template
 
   Scenario: Generate template using a template file with an unsupported extension
     Given the template for stack "1/A" is "template.unsupported"
-    When the user generates the template for stack "1/A"
+    When the user dumps the template for stack "1/A"
     Then a "UnsupportedTemplateFileTypeError" is raised
 
   Scenario Outline: Rendering jinja templates
     Given the template for stack "7/A" is "<filename>"
-    When the user generates the template for stack "7/A"
+    When the user dumps the template for stack "7/A"
     Then the output is the same as the contents of "<rendered_filename>" template
 
   Examples: Template file extensions
@@ -59,15 +59,15 @@ Feature: Generate template
 
   Scenario Outline: Render jinja template which uses an invalid key
     Given the template for stack "7/A" is "<filename>"
-    When the user generates the template for stack "7/A"
+    When the user dumps the template for stack "7/A"
     Then a "<exception>" is raised
 
   Examples: Render Errors
     | filename                                | exception          |
     | jinja/invalid_template_missing_key.j2   | UndefinedError     |
     | jinja/invalid_template_missing_attr.j2  | UndefinedError     |
 
-  Scenario: Generating static templates with file template handler
+  Scenario: Dumping static templates with file template handler
     Given the template for stack "13/A" is "valid_template.json"
-    When the user generates the template for stack "13/A"
+    When the user dumps the template for stack "13/A"
     Then the output is the same as the contents of "valid_template.json" template
diff --git a/integration-tests/steps/templates.py b/integration-tests/steps/templates.py
@@ -77,14 +77,15 @@ def step_impl(context, stack_name):
         context.error = e
 
 
-@when('the user generates the template for stack "{stack_name}"')
+@when('the user dumps the template for stack "{stack_name}"')
 def step_impl(context, stack_name):
     sceptre_context = SceptreContext(
         command_path=stack_name + ".yaml", project_path=context.sceptre_dir
     )
 
     config_path = sceptre_context.full_config_path()
     template_path = sceptre_context.full_templates_path()
+
     with open(os.path.join(config_path, stack_name + ".yaml")) as config_file:
         stack_config = yaml.safe_load(config_file)
 
@@ -100,15 +101,15 @@ def step_impl(context, stack_name):
             stack_config = yaml.safe_load(config_file)
 
     sceptre_plan = SceptrePlan(sceptre_context)
+
     try:
-        context.output = sceptre_plan.generate()
+        context.output = sceptre_plan.dump_template()
+        print(f"Got output {context.output}")
     except Exception as e:
         context.error = e
 
 
-@when(
-    'the user generates the template for stack "{stack_name}" with ignore dependencies'
-)
+@when('the user dumps the template for stack "{stack_name}" with ignore dependencies')
 def step_impl(context, stack_name):
     sceptre_context = SceptreContext(
         command_path=stack_name + ".yaml",
@@ -117,7 +118,7 @@ def step_impl(context, stack_name):
     )
     sceptre_plan = SceptrePlan(sceptre_context)
     try:
-        context.output = sceptre_plan.generate()
+        context.output = sceptre_plan.dump_template()
     except Exception as e:
         context.error = e
 

diff --git a/sceptre/cli/dump.py b/sceptre/cli/dump.py
@@ -1,9 +1,13 @@
 import logging
 import click
 
+from pathlib import Path
+
 from sceptre.context import SceptreContext
 from sceptre.cli.helpers import catch_exceptions, write
 from sceptre.plan.plan import SceptrePlan
+from sceptre.helpers import null_context
+from sceptre.resolvers.placeholders import use_resolver_placeholders_on_error
 
 logger = logging.getLogger(__name__)
 
@@ -18,9 +22,12 @@ def dump_group():
 
 @dump_group.command(name="config")
 @click.argument("path")
+@click.option(
+    "--to-file", is_flag=True, help="If True, also dump the template to a local file."
+)
 @click.pass_context
 @catch_exceptions
-def dump_config(ctx, path):
+def dump_config(ctx, to_file, path):
     """
     Dump the rendered (post-Jinja) Stack Configs.
     \f
@@ -39,17 +46,96 @@ def dump_config(ctx, path):
     plan = SceptrePlan(context)
     responses = plan.dump_config()
 
-    output = []
+    output_format = "json" if context.output_format == "json" else "yaml"
+
     for stack, config in responses.items():
-        if config is None:
-            logger.warning(f"{stack.external_name} does not exist")
+        stack_name = stack.external_name
+
+        if to_file:
+            file_path = Path(".dump") / stack_name / f"config.{output_format}"
+            logger.info(f"{stack_name} dumping to {file_path}")
+            write(config, output_format, no_colour=True, file_path=file_path)
+            logger.info(f"{stack_name} dump to {file_path} complete.")
+
         else:
-            output.append({stack.external_name: config})
+            write(config, output_format, no_colour=True)
+
+
+@dump_group.command(name="template")
+@click.argument("path")
+@click.option(
+    "-n",
+    "--no-placeholders",
+    is_flag=True,
+    help="If True, no placeholder values will be supplied for resolvers that cannot be resolved.",
+)
+@click.option(
+    "--to-file", is_flag=True, help="If True, also dump the template to a local file."
+)
+@click.pass_context
+@catch_exceptions
+def dump_template(ctx, to_file, no_placeholders, path):
+    """
+    Prints the template used for stack in PATH.
+    \f
+
+    :param path: Path to execute the command on.
+    :type path: str
+    """
+    context = SceptreContext(
+        command_path=path,
+        command_params=ctx.params,
+        project_path=ctx.obj.get("project_path"),
+        user_variables=ctx.obj.get("user_variables"),
+        options=ctx.obj.get("options"),
+        output_format=ctx.obj.get("output_format"),
+        ignore_dependencies=ctx.obj.get("ignore_dependencies"),
+    )
+    plan = SceptrePlan(context)
+
+    execution_context = (
+        null_context() if no_placeholders else use_resolver_placeholders_on_error()
+    )
+    with execution_context:
+        responses = plan.dump_template()
 
     output_format = "json" if context.output_format == "json" else "yaml"
 
-    if len(output) == 1:
-        write(output[0][stack.external_name], output_format)
-    else:
-        for config in output:
-            write(config, output_format)
+    for stack, template in responses.items():
+        stack_name = stack.external_name
+
+        if to_file:
+            file_path = Path(".dump") / stack_name / f"template.{output_format}"
+            logger.info(f"{stack_name} dumping template to {file_path}")
+            write(template, output_format, no_colour=True, file_path=file_path)
+            logger.info(f"{stack_name} dump to {file_path} complete.")
+
+        else:
+            write(template, output_format, no_colour=True)
+
+
+@dump_group.command(name="all")
+@click.argument("path")
+@click.option(
+    "-n",
+    "--no-placeholders",
+    is_flag=True,
+    help="If True, no placeholder values will be supplied for resolvers that cannot be resolved.",
+)
+@click.option(
+    "--to-file", is_flag=True, help="If True, also dump the template to a local file."
+)
+@click.pass_context
+@catch_exceptions
+def dump_all(ctx, to_file, no_placeholders, path):
+    """
+    Dumps both the rendered (post-Jinja) Stack Configs and the template used for stack in PATH.
+    \f
+
+    :param path: Path to execute the command on.
+    :type path: str
+    """
+    ctx.invoke(dump_config, to_file=to_file, path=path)
+    ctx.invoke(
+        dump_template, to_file=to_file, no_placeholders=no_placeholders, path=path
+    )
diff --git a/sceptre/cli/helpers.py b/sceptre/cli/helpers.py
@@ -1,8 +1,12 @@
 import logging
 import sys
+
 from itertools import cycle
 from functools import partial, wraps
 
+from typing import Any, Optional
+from pathlib import Path
+
 import json
 import click
 import six
@@ -17,6 +21,8 @@
 from sceptre.stack_status import StackStatus
 from sceptre.stack_status_colourer import StackStatusColourer
 
+logger = logging.getLogger(__name__)
+
 
 def catch_exceptions(func):
     """
@@ -63,18 +69,21 @@ def confirmation(command, ignore, command_path, change_set=None):
         click.confirm(msg, abort=True)
 
 
-def write(var, output_format="json", no_colour=True):
+def write(
+    var: Any,
+    output_format: str = "json",
+    no_colour: bool = True,
+    file_path: Optional[Path] = None,
+) -> None:
     """
     Writes ``var`` to stdout. If output_format is set to "json" or "yaml",
     write ``var`` as a JSON or YAML string.
 
     :param var: The object to print
-    :type var: object
     :param output_format: The format to print the output as. Allowed values: \
     "text", "json", "yaml"
-    :type output_format: str
     :param no_colour: Whether to colour stack statuses
-    :type no_colour: bool
+    :param file_path: Optional path to a file to save the output
     """
     output = var
 
@@ -84,6 +93,16 @@ def write(var, output_format="json", no_colour=True):
         output = _generate_yaml(var)
     if output_format == "text":
         output = _generate_text(var)
+
+    if file_path:
+        dir_path = file_path.parent
+        dir_path.mkdir(parents=True, exist_ok=True)
+
+        with open(file_path, "w") as f:
+            f.write(output)
+
+        return
+
     if not no_colour:
         stack_status_colourer = StackStatusColourer()
         output = stack_status_colourer.colour(str(output))

diff --git a/sceptre/cli/template.py b/sceptre/cli/template.py
@@ -1,8 +1,10 @@
 import logging
 import webbrowser
-
 import click
 
+from deprecation import deprecated
+
+from sceptre import __version__
 from sceptre.cli.helpers import catch_exceptions, write
 from sceptre.context import SceptreContext
 from sceptre.helpers import null_context
@@ -65,6 +67,7 @@ def validate_command(ctx, no_placeholders, path):
 @click.argument("path")
 @click.pass_context
 @catch_exceptions
+@deprecated("4.2.0", "5.0.0", __version__, "Use dump template instead.")
 def generate_command(ctx, no_placeholders, path):
     """
     Prints the template used for stack in PATH.