add cfn lookup (#387)

* start adding cfn lookup

* cleanup, add more error handling, register as cfngin lookup

* correct exception type, add tests

* add cfn lookup docs, update formatting, fix missing directives

* deprecate rxref, fix spelling, fix import order

* deprecate terraform_backend_cfn_outputs, update docs

* add more commands to docs makefile

* update changelog

* fix linting

* fix unit tests

* fix python 2 linting

* remove namedtuple.__doc__ for python 2 support

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Terraform remote backend has custom handling around pre-selecting a workspace, not switching workspace, and dumping parameters to a `runway-parameters.auto.tfvars.json` file (only option of variables with remote backend)
 - Terraform workspaces can be specified with the `terraform_workspace` option (mainly needed for remote backend support)
 - Terraform module parameters can now be dumped to a `auto.tfvars` using the `terraform_write_auto_tfvars` option (mainly needed for remote backend support)
+- `cfn` Lookup usable in Runway and CFNgin config files
 
 ### Changed
 - env managers now use pathlib
@@ -19,6 +20,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Terraform backend configuration is now collected and parsed into a dict that is available on the Terraform environment manager object
 - split the `run_terraform` method of the Terraform module class into multiple methods to be more easily tested
 
+- `terraform_backend_cfn_outputs` option is now deprecated
+- `xref` Lookup is now deprecated
+
 ## [1.10.1] - 2020-07-20
 ### Fixed
 - fixed an issue where AWS account alias/id validation was not using the context object with assumed credentials when running in parallel

docs/Makefile

@@ -11,7 +11,15 @@ BUILDDIR      = build
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile
+.PHONY: clean docs open help Makefile
+
+clean:
+	rm -rf build/html/*
+
+docs: clean html open
+
+open:
+	open build/html/index.html
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

docs/source/cfngin/lookups.rst

@@ -6,9 +6,9 @@
 
 .. _cfngin-lookups:
 
-=======
+#######
 Lookups
-=======
+#######
 
 .. note:: Runway lookups and CFNgin lookups are not interchangeable. While they
           do share a similar base class and syntax, they exist in two different
@@ -76,6 +76,7 @@ Runway's CFNgin includes the following lookup types:
 
 - `output lookup`_
 - `ami lookup`_
+- `cfn lookup`_
 - `custom lookup`_
 - `default lookup`_
 - `dynamodb lookup`_
@@ -87,10 +88,22 @@ Runway's CFNgin includes the following lookup types:
 - ssm_ lookup
 - `xref lookup`_
 
+**********
+cfn Lookup
+**********
+
+.. important::
+  The Stack must exist in CloudFormation before the config using this Lookup begins processing to successfully get a value.
+  This means that it must have been deployed using another Runway module, deployed from a config that is run before the one using it, deployed manually, or deployed in the same config using ``required``/``required_by`` to specify a dependency between the Stacks.
+
+.. automodule:: runway.lookups.handlers.cfn
+
+
 .. _`output lookup`:
 
+*************
 Output Lookup
--------------
+*************
 
 The ``output`` lookup takes a value of the format:
 ``<stack name>::<output name>`` and retrieves the output from the given stack
@@ -109,8 +122,9 @@ You can specify an output lookup with the following syntax:
 
 .. _`default lookup`:
 
+**************
 default Lookup
---------------
+**************
 
 The ``default`` lookup type will check if a value exists for the variable
 in the environment file, then fall back to a default defined in the CFNgin
@@ -143,8 +157,9 @@ value.
 
 .. _`kms lookup`:
 
+**********
 KMS Lookup
-----------
+**********
 
 The ``kms`` lookup type decrypts its input value.
 
@@ -183,8 +198,12 @@ value is large) using the ``file://`` prefix, ie::
 
 .. _`xref lookup`:
 
+***********
 XRef Lookup
------------
+***********
+
+.. deprecated:: 1.11.0
+  Replaced by `cfn lookup`_
 
 The ``xref`` lookup type is very similar to the ``output`` lookup type, the
 difference being that ``xref`` resolves output values from stacks that
@@ -211,8 +230,9 @@ requirements.
 
 .. _`rxref lookup`:
 
+************
 RXRef Lookup
-------------
+************
 
 The ``rxref`` lookup type is very similar to the ``xref`` lookup type,
 the difference being that ``rxref`` will lookup output values from stacks
@@ -252,8 +272,9 @@ within the same CFNgin YAML config.
 
 .. _`file lookup`:
 
+***********
 File Lookup
------------
+***********
 
 The ``file`` lookup type allows the loading of arbitrary data from files on
 disk. The lookup additionally supports using a ``codec`` to manipulate or
@@ -367,16 +388,18 @@ Basic examples::
         Resource: "{{MyResource}}"
 
 
+***
 ssm
----
+***
 
 .. automodule:: runway.lookups.handlers.ssm
 
 
 .. _`ssmstore lookup`:
 
+**************************
 SSM Parameter Store Lookup
---------------------------
+**************************
 
 .. deprecated:: 1.5.0
   Replaced by ssm_
@@ -409,10 +432,12 @@ values)
 The region can be omitted (e.g. ``DBUser: ${ssmstore MyDBUser}``), in which
 case ``us-east-1`` will be assumed.
 
+
 .. _`dynamodb lookup`:
 
+***************
 DynamoDb Lookup
---------------------------
+***************
 
 The ``dynamodb`` lookup type retrieves a value from a DynamoDb table.
 
@@ -450,8 +475,9 @@ You can lookup values inside of a map.
 
 .. _`envvar lookup`:
 
+************************
 Shell Environment Lookup
-------------------------
+************************
 
 The ``envvar`` lookup type retrieves a value from a variable in the shell's
 environment.
@@ -472,10 +498,12 @@ in the lookup, like so::
 
   DBUser: ${envvar file://dbuser_file.txt}
 
+
 .. _`ami lookup`:
 
+**************
 EC2 AMI Lookup
---------------
+**************
 
 The ``ami`` lookup is meant to search for the most recent AMI created that
 matches the given filters.
@@ -506,10 +534,12 @@ Example::
   # Note: The region is optional, and defaults to the current CFNgin region
   ImageId: ${ami [<region>@]owners:self,888888888888,amazon name_regex:server[0-9]+ architecture:i386}
 
+
 .. _`hook_data lookup`:
 
+****************
 Hook Data Lookup
-----------------
+****************
 
 When using hooks, you can have the hook store results in the
 `hook_data`_ dictionary on the context by setting ``data_key`` in the hook
@@ -548,15 +578,16 @@ limited or no effect:
 
 .. _`custom lookup`:
 
+*************
 Custom Lookup
---------------
+*************
 
 A custom lookup may be registered within the config.
 For more information see `Configuring Lookups <configuration.html#lookups>`_.
 
 
 Writing A Custom Lookup
-~~~~~~~~~~~~~~~~~~~~~~~
+=======================
 
 A custom lookup must be in an executable, importable python package or standalone file.
 The lookup must be importable using your current ``sys.path``.

docs/source/lookups.rst

@@ -1,7 +1,8 @@
 .. _Lookups:
 
+#######
 Lookups
-=======
+#######
 
 Runway Lookups allow the use of variables within the Runway config file. These
 variables can then be passed along to :ref:`deployments <runway-deployment>`,
@@ -48,17 +49,31 @@ expect.
 .. automodule:: runway.lookups.handlers.base
 
 
-.. _build-in-lookups:
+.. _built-in-lookups:
 
-Build-in Lookups
-^^^^^^^^^^^^^^^^
+****************
+Built-in Lookups
+****************
+
+
+.. _cfn lookup:
+.. _cfn-lookup:
+
+cfn
+===
+
+.. important::
+  The Stack must exist in CloudFormation before the module using this Lookup begins processing to successfully get a value.
+  This means that the Stack must have been deployed by another module, run before the one using this Lookup, or it must have been created external to Runway.
+
+.. automodule:: runway.lookups.handlers.cfn
 
 
 .. _env lookup:
 .. _env-lookup:
 
 env
-~~~
+===
 
 .. automodule:: runway.lookups.handlers.env
 
@@ -67,7 +82,7 @@ env
 .. _ssm-lookup:
 
 ssm
-~~~
+===
 
 .. automodule:: runway.lookups.handlers.ssm
 
@@ -76,6 +91,6 @@ ssm
 .. _var-lookup:
 
 var
-~~~
+===
 
 .. automodule:: runway.lookups.handlers.var

docs/source/terraform/advanced_features.rst

@@ -99,51 +99,18 @@ Backend config options can also be specified as a module option in the `Runway C
   ---
   deployments:
     - modules:
-        - path: sampleapp.tf
+        - path: sampleapp-01.tf
           options:
             terraform_backend_config:
               bucket: mybucket
               dynamodb_table: mytable
               region: us-east-1
-
-.. rubric:: Deployment Level
-.. code-block:: yaml
-
-  ---
-  deployments:
-    - modules:
-        - path: sampleapp-01.tf
         - path: sampleapp-02.tf
-      module_options:  # shared between all modules in deployment
-        terraform_backend_config:
-          bucket: ${ssm ParamNameHere::region=us-east-1}
-          dynamodb_table: ${ssm ParamNameHere::region=us-east-1}
-          region: ${env AWS_REGION}
-
-
-runway.yml From CloudFormation Outputs
-======================================
-
-A recommended option for managing the state bucket and table is to create
-them via CloudFormation (try running ``runway gen-sample cfn`` to get a
-template and stack definition for bucket/table stack). To further support this,
-backend config options can be looked up directly from CloudFormation
-outputs.
-
-.. rubric:: Module Level
-.. code-block:: yaml
-
-  ---
-  deployments:
-    - modules:
-        - path: sampleapp.tf
           options:
             terraform_backend_config:
-              region: us-east-1
-            terraform_backend_cfn_outputs:
-              bucket: StackName::OutputName  # e.g. common-tf-state::TerraformStateBucketName
-              dynamodb_table: StackName::OutputName  # e.g. common-tf-state::TerraformStateTableName
-
+              bucket: ${cfn common-tf-state.TerraformStateBucketName}
+              dynamodb_table: ${cfn common-tf-state.TerraformStateTableName}
+              region: ${env AWS_REGION}
 
 .. rubric:: Deployment Level
 .. code-block:: yaml
@@ -155,10 +122,9 @@ outputs.
         - path: sampleapp-02.tf
       module_options:  # shared between all modules in deployment
         terraform_backend_config:
-          region: us-east-1
-        terraform_backend_cfn_outputs:
-          bucket: StackName::OutputName  # e.g. common-tf-state::TerraformStateBucketName
-          dynamodb_table: StackName::OutputName  # e.g. common-tf-state::TerraformLockTableName
+          bucket: ${ssm ParamNameHere::region=us-east-1}
+          dynamodb_table: ${ssm ParamNameHere::region=us-east-1}
+          region: ${env AWS_REGION}
 
 
 ----

runway/cfngin/lookups/handlers/xref.py

@@ -1,15 +1,20 @@
 """Handler for fetching outputs from fully qualified stacks."""
 # pylint: disable=arguments-differ,unused-argument
+import logging
 from runway.lookups.handlers.base import LookupHandler
 
 from .output import deconstruct
 
+LOGGER = logging.getLogger(__name__)
 TYPE_NAME = "xref"
 
 
 class XrefLookup(LookupHandler):
     """Xref lookup."""
 
+    DEPRECATION_MSG = ('xref Lookup has been deprecated; '
+                       'use the cfn lookup instead')
+
     @classmethod
     def handle(cls, value, context=None, provider=None, **kwargs):
         """Fetch an output from the designated, fully qualified stack.
@@ -36,6 +41,7 @@ def handle(cls, value, context=None, provider=None, **kwargs):
                 conf_value: ${xref fully-qualified-stack-name::SomeOutputName}
 
         """
+        LOGGER.warning(cls.DEPRECATION_MSG)
         if provider is None:
             raise ValueError('Provider is required')