Problem when launching a subset of an environment and using resolvers across profiles #483
Labels
Comments
|
@basvank sorry for the delay, this issue will be fixed for 2.0. |
|
@basvank would you be able to try and test this again with our v2 Release Candidate? You should be able to install it with |
|
@ngfgrant It still fails, unfortunately, and it now even fails when I try to launch an environment that does not depend on a stack somewhere else in the tree. That was not the case earlier (I was on master, commit The error now looks as follows: |
|
@basvank you will need to follow the migration guide: https://github.com/cloudreach/sceptre/wiki/Migration-Guide:-V1-to-V2 Specifically, StackOutput (non-external) resolver needs to specify the full relevant path from the project config to the stack file (including the extension) Assuming prod/B.yaml has this should now be You will need to ensure all your config files have this properly updated now for StackOutputs |
|
I followed the migration guide and it indeed seemed to have fixed the earlier problem, as I could get the However, it seemed that, even if I deployed a subset of the stack, it still tried to deploy to whole setup. Eventually, it failed with an enormous stack trace and a recursion error: We're using nested stacks up to a depth of about 4 folders. Any chance that is pushing the limits of sceptre, or do you have any idea of anything else that is going wrong? |
|
Nested stacks shouldn't be an issue, if possible could you post a full example tree of your current setup? Just the minimum set to make this scenario fail so I can test it out locally. Sceptre 2.0 will have changed some functionality regarding launching a stack, if you try to launch a stack which depends on other stacks then they will be launched as well (this also applies cross stack groups). If you want a single stack to launch then you will have to use |
|
@nabeelamjad It's a corporate project, so I would need to discuss if sharing the code is possible. We have about 200 stacks currently configured via sceptre, so creating a minimum set might be a challenge. |
|
@basvank sure thing - it doesn't have to be the actual structure or whatever. As long as I can some similar setup (as you said, 200 stacks, so might be more difficult) or a broad description so we can generate them ourselves to result in the same recursion error. We'll do some more playing with deeply nested stacks (we've tried about 3-4 levels deep ourselves). |
|
@nabeelamjad Playing around with it a bit more I get the impression that the issue might be in the dependency graph. In total, we have 507 Note: sometimes the recursion error is not thrown in sceptre itself, but rather the error |
|
Will do @basvank - are all the stack outputs connected in some way? Do you know the max depth of the "most connected" stack group? If we assume the following stacks: 1, 2, 3, 4, 5, 6, 7, 8, 9 where 9 is connected to 8, to 7, etc. Then that'd be a depth/level of 8/9 |
|
The deepest depth I have traced is about 9 levels deep. I cannot imagine it to be much deeper than that. |
|
@basvank I'm testing this out now with a recursion limit of 20 levels deep and a total of 200 stacks where each stack is dependent on the previous stack. The first stack sets an output value and the rest propagates it. This may take a while to launch on AWS, but so far I've had no problems generating the actual dependency graph nor launching it. Monitoring the progress on AWS shows everything seems to be going fine. I've attached a zip file with the setup, you will need to replace If this goes fine I may try different kind of complex stacks as well as multiple sibling stack groups rather than just a single nested stack group and multiple groups at the root level. I suppose it may hard to debug your actual problem as I'm just taking a wild guess here. Edit: If you see Here's the output of the nested stack group, no issues observed at all. All but the first stack receives its Sceptre List Outputs |
|
@nabeelamjad Your note about the ValueError is interesting. Any way to narrow down where the problem might be? I've checked the paths to the templates dir and the file extensions in the output_resolvers, but haven't found the issue yet. This is the complete stacktrace: Also, using the working version of sceptre, as of commit |
|
@basvank I've taken your launch order and converted them into templates/config files and had no issues attempting to launch it, slightly confused what it could be. I think I'm still edging towards some sort of configuration error. I've attached the stack zip with your launch order re-generated into mock templates (non-dependant stacks output their stack name and dependent templates use those as their output) |
|
@basvank have any of the subsequent full v2 releases resolved this for you? |
|
Closing as not heard back, reopen if the issue persists. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This problem arose after playing around with the implementation of #395. It concerns sceptre v2 (I am using the master branch).
Consider the following sceptre environment:
The
devandprodenvironments live in different accounts, so inconfig/dev/config.yamlwe setprofile: aws_test_accountand inconfig/prod/config.yamlwe useprofile: aws_prod_account. Both these profiles work as intended, as you will see in a minute.In template
config/prod/B.yamlwe use a resolver to an output of templateconfig/dev/A.yaml, so for instancetest_parameter: !stack_output dev/A::testOutput.If we now run
sceptre launch .everything works fine, so the setup looks good (as in; there is no problem with the profiles or AWS credentials).However, if we now run
sceptre launch prodto only launch a subset of the environment, we get an error. My whole setup is a bit more complex, but for reference this is my complete stack trace:Here,
xxxis the name of the stack that the resolver points to, so in the example this would bedev/A.So it looks that the resolvers don't use the profiles properly when only a subset of the stack is deployed. The workaround is simple; just launch the whole stack. But it would be nice if in this setup launching a subset of the stack would be supported as well.
The text was updated successfully, but these errors were encountered: