-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Resolve #1252] Supporting resolvers in Hook and Resolver arguments, with new !substitute, !join, !split, and !select resolvers! #1313
Changes from all commits
cb5a477
382938f
caa6d3f
f34631e
24f3333
5376462
8ba2d02
1b56f5e
9712bfa
617c37a
b7f4059
8ccb8bd
5d14ec0
a98a955
c739663
ee70e4e
868448b
1fc438d
80cb201
01130d4
ac07a92
13cf87d
1502a81
2c42905
aa1a258
62bcb29
51c7079
6f84e48
c5f9c72
9fe80bc
f75e3c3
20ca362
d7eb15f
5ad9748
5888204
d57bb3d
d8cd879
9080e70
26d960e
6babfb8
b3d50ff
e1665ad
fab6544
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,6 +5,9 @@ Sceptre implements resolvers, which can be used to resolve a value of a | |
CloudFormation ``parameter`` or ``sceptre_user_data`` value at runtime. This is | ||
most commonly used to chain the outputs of one Stack to the inputs of another. | ||
|
||
You can use resolvers with any resolvable property on a StackConfig, as well as in the arguments | ||
of hooks and other resolvers. | ||
|
||
If required, users can create their own resolvers, as described in the section | ||
on `Custom Resolvers`_. | ||
|
||
|
@@ -52,6 +55,27 @@ file_contents | |
|
||
**deprecated**: Consider using the `file`_ resolver instead. | ||
|
||
join | ||
~~~~ | ||
|
||
This resolver allows you to join multiple strings together to form a single string. This is great | ||
for combining the outputs of multiple resolvers. This resolver works just like CloudFormation's | ||
``!Join`` intrinsic function. | ||
|
||
The argument for this resolver should be a list with two elements: (1) A string to join the elements | ||
on and (2) a list of items to join. | ||
|
||
Example: | ||
|
||
.. code-block:: yaml | ||
|
||
parameters: | ||
BaseUrl: !join | ||
- ":" | ||
- - !stack_output my/app/stack.yaml::HostName | ||
- !stack_output my/other/stack.yaml::Port | ||
|
||
|
||
no_value | ||
~~~~~~~~ | ||
|
||
|
@@ -79,6 +103,48 @@ A resolver to execute any shell command. | |
|
||
Refer to `sceptre-resolver-cmd <https://github.com/Sceptre/sceptre-resolver-cmd/>`_ for documentation. | ||
|
||
select | ||
~~~~~~ | ||
|
||
This resolver allows you to select a specific index of a list of items. This is great for combining | ||
with the ``!split`` resolver to obtain part of a string. This function works almost the same as | ||
CloudFormation's ``!Select`` intrinsic function, **except you can use this with negative indices to | ||
select from the end of a list**. | ||
|
||
The argument for this resolver should be a list with two elements: (1) A numerical index and (2) a | ||
list of items to select out of. If the index is negative, it will select from the end of the list. | ||
For example, "-1" would select the last element and "-2" would select the second-to-last element. | ||
|
||
Example: | ||
|
||
.. code-block:: yaml | ||
|
||
sceptre_user_data: | ||
# This selects the last element after you split the connection string on "/" | ||
DatabaseName: !select | ||
- -1 | ||
- !split ["/", !stack_output my/database/stack.yaml::ConnectionString] | ||
|
||
split | ||
~~~~~ | ||
|
||
This resolver will split a value on a given delimiter string. This is great when combining with the | ||
``!select`` resolver. This function works the same as CloudFormation's ``!Split`` intrinsic function. | ||
|
||
Note: The return value of this resolver is a *list*, not a string. This will not work to set Stack | ||
configurations that expect strings, but it WILL work to set Stack configurations that expect lists. | ||
|
||
The argument for this resolver should be a list with two elements: (1) The delimiter to split on and | ||
(2) a string to split. | ||
|
||
Example: | ||
|
||
.. code-block:: yaml | ||
|
||
notifications: !split | ||
- ";" | ||
- !stack_output my/sns/topics.yaml::SemicolonDelimitedArns | ||
Comment on lines
+144
to
+146
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do you think it would make it easier for users to understand if these examples just used direct values? something like this..
result: the list ["note1", "note2", "note3"] is passed into the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, I don't really. !sub, !join, !select, and !split are ONLY useful when you have resolvers to string together; They're clumsy if you don't. It's better to use Jinja to do these things, if you can. But if you have resolvers whose resolved values you need to manipulate/interpolate/select from, they're really the only way to do that. As such, If we don't show how these resolvers can be used with other resolvers, we're not really showing how to use them well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ¿Por qué no los dos? :) notifications: !split
- ";"
- "note1; note2; note3" Resolves to: notifications: [ "note1", "note2", "note3" ] As a more practical example, the following will split a semicolon-delimited list exported by another stack: notifications: !split
- ";"
- !stack_output my/sns/topics.yaml::SemicolonDelimitedArns There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @dboitnot, sure, that works. But If you wanted to split a simple string, you'd just go like this: notifications: {{ "note1; note2; note3".split(";") }} I fear that if we show people examples of using these resolvers in cases where you don't actually need to use them (and where you really shouldn't use them), it will be more confusing. The only times you need to use these resolvers is when you need to combine/manipulate the outputs of other resolvers. I actually added documentation to that effect to make it absolutely clear. |
||
|
||
.. _stack_attr_resolver: | ||
|
||
stack_attr | ||
|
@@ -181,6 +247,67 @@ Example: | |
parameters: | ||
VpcIdParameter: !stack_output_external prj-network-vpc::VpcIdOutput prod | ||
|
||
|
||
sub | ||
~~~ | ||
|
||
This resolver allows you to create a string using Python string format syntax. This functions as a | ||
great way to combine together a number of resolver outputs into a single string. This functions | ||
similarly to Cloudformation's ``!Sub`` intrinsic function. | ||
|
||
It should be noted that Jinja2 syntax is far more capable of interpolating values than this resolver, | ||
so you should use Jinja2 if all you need is to interpolate raw values from environment variables, | ||
variables from stack group configs, var files, and ``--var`` arguments. **The one thing that Jinja2 | ||
interpolation can't do is interpolate resolver arguments into a string.** And that's what ``!sub`` | ||
can do. For more information on why Jinja2 can't reference resolvers directly, see | ||
:ref:`resolution_order`. | ||
|
||
The argument to this resolver should be a two-element list: (1) Is the format string, using | ||
curly-brace templates to indicate variables, and (2) a dictionary where the keys are the format | ||
string's variable names and the values are the variable values. | ||
|
||
Example: | ||
|
||
.. code-block:: yaml | ||
|
||
parameters: | ||
ConnectionString: !sub | ||
- "postgres://{username}:{password}@{hostname}:{port}/{database}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can the sub reference vars outside of it's scope? Can you set these to build-in vars like account id/region/etc.. What about references to vars from config.yaml If possible it might help users if there was an example or two of those use cases. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
!sub
# Note the double-curly braces is JINJA syntax to access variables within Jinja scope
- "postgres://{{ var.username }}:{password}@{hostname}:{port}/{{ database }}"
- password: !ssm /my/ssm/password
hostname: !stack_output my/database/stack.yaml::HostName
port: !stack_output my/database/stack.yaml::Port Notice how, if it's accessible within Jinja, you don't actually need to provide that to |
||
# Notice how we're interpolating a username and database via Jinja2? Technically it's not | ||
# necessary to pass them this way. They could be interpolated directly. But it might be | ||
# easier to read this way if you pass them explicitly like this. See example below for the | ||
# other way this can be done. | ||
- username: {{ var.username }} | ||
password: !ssm /my/ssm/password | ||
hostname: !stack_output my/database/stack.yaml::HostName | ||
port: !stack_output my/database/stack.yaml::Port | ||
database: {{var.database}} | ||
|
||
|
||
It's relevant to note that this functions similarly to the *more verbose* form of CloudFormation's | ||
``!Sub`` intrinsic function, where you use a list argument and supply the interpolated values as a | ||
second list item in a dictionary. **Important**: Sceptre's ``!sub`` resolver will not work without | ||
a list argument. It does **not** directly reference variables without you directly passing them | ||
in the second list item in its argument. | ||
|
||
You *can* combine Jinja2 syntax with this resolver if you want to interpolate in other variables | ||
that Jinja2 has access to. | ||
|
||
Example: | ||
|
||
.. code-block:: yaml | ||
|
||
parameters: | ||
ConnectionString: !sub | ||
# Notice the double-curly braces. That's Jinja2 syntax. Jinja2 will render the username into | ||
# the string even before the yaml is loaded. If you use Jinja2 to interpolate the value, then | ||
# it's not a template string variable you need to pass in the second list item passed to | ||
# !sub. | ||
- "postgres://{{ var.username }}:{password}@{hostname}:{port}/{{ stack_group_config.database }}" | ||
- password: !ssm /my/ssm/password | ||
hostname: !stack_output my/database/stack.yaml::HostName | ||
port: !stack_output my/database/stack.yaml::Port | ||
|
||
Custom Resolvers | ||
---------------- | ||
|
||
|
@@ -306,18 +433,22 @@ For details on calling AWS services or invoking AWS-related third party tools in | |
|
||
Resolver arguments | ||
^^^^^^^^^^^^^^^^^^ | ||
Resolver arguments can be a simple string or a complex data structure. | ||
Resolver arguments can be a simple string or a complex data structure. You can even use | ||
other resolvers in the arguments to resolvers! (Note: Other resolvers can only be passed in | ||
arguments when they're passed in lists and dicts.) | ||
|
||
.. code-block:: yaml | ||
|
||
template: | ||
path: <...> | ||
type: <...> | ||
parameters: | ||
Param1: !ssm "/dev/DbPassword" | ||
Param2: !ssm {"name": "/dev/DbPassword"} | ||
Param3: !ssm | ||
name: "/dev/DbPassword" | ||
parameters: | ||
Param1: !ssm "/dev/DbPassword" | ||
Param2: !ssm {"name": "/dev/DbPassword"} | ||
Param3: !ssm | ||
name: "/dev/DbPassword" | ||
Param4: !ssm | ||
name: !stack_output my/other/stack.yaml::MySsmParameterName | ||
|
||
.. _Custom Resolvers: #custom-resolvers | ||
.. _this is great place to start: https://docs.python.org/3/distributing/ | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,7 +2,7 @@ | |
from contextlib import contextmanager | ||
from datetime import datetime | ||
from os import sep | ||
from typing import Optional, Any, List | ||
from typing import Optional, Any, List, Tuple, Union | ||
|
||
import dateutil.parser | ||
import deprecation | ||
|
@@ -68,6 +68,28 @@ def func_on_instance(key): | |
return attr | ||
|
||
|
||
Container = Union[list, dict] | ||
Key = Union[str, int] | ||
|
||
|
||
def delete_keys_from_containers(keys_to_delete: List[Tuple[Container, Key]]): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is mostly just extracted out of the ResolvableProperty class and put here. I needed to also use this in the CustomYamlTagBase class. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm glad that this has been extracted but I'm a bit queasy about this function mutating collections it doesn't own. I got side-tracked by work but I feel this could be simplified (in the future) using list comprehensions and/or There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I posted a previous response to this and then I'm replacing it with this, because I thought you were talking about something else. If you look at how this is used, mutation is exactly the purpose of this function. Filtered lists and dicts will not suit the purpose of what this exists for and won't work. The whole point is in-place modification of containers because of the way that resolvers operate. |
||
"""Removes the indicated keys/indexes from their paired containers.""" | ||
list_items_to_delete = [] | ||
for container, key in keys_to_delete: | ||
if isinstance(container, list): | ||
# If it's a list, we want to gather up the items to remove from the list. | ||
# We don't want to modify the list length yet, since removals will change all the other | ||
# list indexes. Instead, we'll get the actual items at those indexes to remove later. | ||
list_items_to_delete.append((container, container[key])) | ||
else: | ||
del container[key] | ||
|
||
# Finally, now that we have all the items we want to remove the lists, we'll remove those | ||
# items specifically from the lists. | ||
for containing_list, item in list_items_to_delete: | ||
containing_list.remove(item) | ||
|
||
|
||
def normalise_path(path): | ||
""" | ||
Converts a path to use correct path separator. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,39 +1,21 @@ | ||
import abc | ||
import logging | ||
from functools import wraps | ||
from typing import TYPE_CHECKING | ||
|
||
from sceptre.helpers import _call_func_on_values | ||
from sceptre.logging import StackLoggerAdapter | ||
|
||
from typing import TYPE_CHECKING, Any | ||
from sceptre.resolvers import CustomYamlTagBase | ||
|
||
if TYPE_CHECKING: | ||
from sceptre.stack import Stack | ||
|
||
|
||
class Hook(abc.ABC): | ||
class Hook(CustomYamlTagBase, metaclass=abc.ABCMeta): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The big change here is that, since there's very little that differentiates a hook from a resolver, other than the "resolve" vs "run" public method, they now share a common base class. A lot of the functionality on Hooks has now been moved up into that base class since it was shared with Resolvers. |
||
""" | ||
Hook is an abstract base class that should be inherited by all hooks. | ||
|
||
:param argument: The argument of the hook. | ||
:param stack: The associated stack of the hook. | ||
Hook is an abstract base class that should be subclassed by all hooks. | ||
""" | ||
|
||
def __init__(self, argument: Any = None, stack: "Stack" = None): | ||
self.logger = logging.getLogger(__name__) | ||
|
||
if stack is not None: | ||
self.logger = StackLoggerAdapter(self.logger, stack.name) | ||
|
||
self.argument = argument | ||
self.stack = stack | ||
|
||
def setup(self): | ||
""" | ||
setup is a method that may be overwritten by inheriting classes. Allows | ||
hooks to run so initalisation steps when config is first read. | ||
""" | ||
pass # pragma: no cover | ||
logger = logging.getLogger(__name__) | ||
|
||
@abc.abstractmethod | ||
def run(self): | ||
|
@@ -43,15 +25,6 @@ def run(self): | |
""" | ||
pass # pragma: no cover | ||
|
||
def clone(self, stack: "Stack") -> "Hook": | ||
""" | ||
Produces a "fresh" copy of the Hook, with the specified stack. | ||
|
||
:param stack: The stack to set on the cloned resolver | ||
""" | ||
clone = type(self)(self.argument, stack) | ||
return clone | ||
|
||
Comment on lines
-46
to
-54
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is now defined on the base class |
||
|
||
class HookProperty(object): | ||
""" | ||
|
@@ -84,7 +57,7 @@ def __set__(self, instance: "Stack", value): | |
""" | ||
|
||
def setup(attr, key, value: Hook): | ||
attr[key] = clone = value.clone(instance) | ||
attr[key] = clone = value.clone_for_stack(instance) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
clone.setup() | ||
|
||
_call_func_on_values(setup, value, Hook) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think the before_update and after_update examples are wrong originally. shouldn't they be like this instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No.
!copy
in this imaginary example takes a dict as an argument, not a list. However,after_update
andbefore_update
should be lists. So it's correct in the docs, not in your suggestion.