Feature: String multiple resolvers together

[zaro0508]

This is a feature request. I would like to be able to string multiple sceptre resolvers together to return one value for a parameter. The idea here is similar to ls –al | more on linux systems.

Here's a trivial example of resolving a parameter by getting the contents of a file and then sorting it before assigning it to the SortedStaff parameter:

template:
  path: mystack.yaml
stack_name: mystack
parameters:
  SortedStaff: !file staff.txt | !rcmd sort

Assume staff.txt contains:

Peter
Andy
Norman
Kevin

Upon sceptre launch, SortedStaff would contain:

Andy
Kevin
Norman
Peter

[jfalkenstein]

So I was looking into something similar, using resolvers as arguments to other resolvers, but YAML explicitly doesn't let you do that.

I think the closest thing we could accomplish would be resolvers that operate a bit like CloudFormation resolvers that take list args, like how you can do something like this:

Conditions:
    MyCondition: !Equals [ !Ref FirstParam, !Ref SecondParam ]

To accomplish what you are suggesting in Sceptre, you could do something like:

parameters:
    SortedStaff: !chain
        - !file staff.txt
        - !sort. # The resolved value from !file would be passed as the argument to !sort. Notice how this isn't !rcmd

In order for this to work, however... the !chain resolver would need to:

1. Whenever setup() is called on it, it would have to also setup all nested resolvers in its arguments
2. When resolve() is called on it, it would need to resolve the first item in the list and then set that as the argument on the next.
3. It would need to continue to resolve and then set arguments down the list until the last resolver resolves
4. The last resolver's value would be returned.

The problem with your example is that you're passing sort as the argument to !rcmd, but also the value from !file. It isn't clear how you intend BOTH values to be passed as arguments.

[zaro0508]

You're right @jfalkenstein and you've read my mind. Your example is what I really meant, chaining multiple resolvers together. I like your idea however it seems like it's a lot of work though. I'm wondering if we can get this same functionality by creating and using custom jinja functions. I'm thinking it would be something like this..

parameters:
  SortedStaff: {{ GetFile(staff.txt) | sort }}

although i'm not sure how we would register and load that custom jinja based GetFile function.

[jfalkenstein]

Well, with the j2_environment parameter, I suppose we can load Jinja extensions. One way would be to just make an extension. I've thought about making resolvers available in the jinja environment before.

Something like {{ resolve('!stack_output blah/blah.yaml::MyOutput') }} or something like that.

However, the issue with using resolvers in ninja is that Jinja is rendered BEFORE the config is rendered and this before the stack exists. I don't think the timing would allow resolvers to operate before the stack exists.

[jfalkenstein]

Alternative, we could just prepopulate the Jinja environment with a module of useful utility functions that augment what Jinja already provides. I've really wanted functions for hashing, for example. We could also have some for getting file contents, stack outputs, and a few other useful ones too.

[jfalkenstein]

@zaro0508 I just had an idea on this one. Imagine this:

template:
  path: mystack.yaml
stack_name: mystack
parameters:
  SortedStaff: !pipe "!file staff.txt | !rcmd sort"

I think this could work. Since, by the time this is loaded, we would have already added the yaml resolver constructors to the loader, we could the !pipe resolver would essentially do something like this:

def resolve(self):
    piped_segments = self.argument.split('|')
    result = None
    for segment in piped_segments:
        resolver = yaml.safe_load(segment.strip())
        resolver.stack = self.stack
        if result and resolver.argument is None:
            resolver.argument = result
        elif result:
            resolver.argument += f' {result}'
        resolver.setup()
        result = resolver.resolve()
    return result

In other words, the result of prior resolvers is used as (or appended to) the arguments of the latter ones. The argument to !pipe is a string of piped resolvers that will be lazy-evaluated when it's resolved.

What do you think of this idea?

[jfalkenstein]

@zaro0508 You never did respond to this one. What do you think of my idea here ^^?

[zaro0508]

hmm, a !pipe resolver that executes other resolvers? I like this idea, especially if it's implemented as a non-core resolver.

[zaro0508]

This is fixed with PR #1313 and available in sceptre v4.1.0