Exception thrown in stack results in resources marked for deletion #7050
Comments
|
@vipentti curious what you think of this. The first thing I see is traditional generic TStack programs run via this function: And notice they have a try/catch and pass exceptions into While the function that inline programs would use, directly beneath it: Doesn't do that. I don't know if that is important or not. |
|
@orionstudt I created two Automation.Api.Tests in #7057 which both fail similarly as reported here. So it happens with both inline program as well as the |
|
@vipentti you are correct it is both. @dferretti is a coworker of mine and when he initially described the issue to me I incorrectly interpreted that he was using Automation API and that it had been a bug I introduced with that implementation. But you're right, it's both, so that is probably worse 😬 |
|
I ran the tests against earlier versions of the cli and automation API and the tests fail there as well. Looks like this bug may have been there for a while. |
|
I think I hit this while messing with my stack outputs. I inadvertently introduced an exception and the whole stack got deleted. |
lukehoban
added
p1
lukehoban
added
emergent
and removed
kind/bug
|
I believe this behavior has existed since the very first release of the .NET plugin. The problem is that you return public static async Task<int> Main(string[] args)
{
return await Deployment.RunAsync<MyStack>();
}With this code, I agree this is a confusing and dangerous behavior, so I welcome any improvement suggestions. cc @t0yv0 for awareness. |
mikhailshilkov
removed
the
p1
|
Ah that is a tricky gotchya. That makes sense for the CLI / local programs. We should probably make sure that this is not happening with Automation API then, when we are controlling the program lifecycle |
|
Indeed! Do you have a repro for this issue with the automation API? |
|
Some off-the-wall ideas that could be helpful:
|
|
@mikhailshilkov I believe @vipentti wrote some tests in #7057 verifying that this behavior is occurring in Automation API as well |
|
Probably from doing silly things while playing around but I'm still hitting this every day. If I wasn't running preview first, or if I wasn't paying attention, then resources would be deleted. 😬 |
|
@gitfool Are you using the problematic pattern described in #7050 (comment) or what is your case? |
|
@mikhailshilkov I'm using the dotnet automation api via a generic host: public static class Program
{
public static Task<int> Main(string[] args) =>
CreateHostBuilder(args).RunCommandAsync(args);
}
...
protected override async Task<int> OnExecuteAsync(CommandContext context, Settings settings)
{
...
var stackName = $"{Config.Pulumi.Organization.Name}/{settings.Environment.ToLower()}";
var stackArgs = new InlineProgramArgs(info.ProjectName, stackName, PulumiFn.Create(ServiceProvider, info.StackType))
{
Logger = LoggerFactory.CreateLogger<Pulumi.Deployment>()
};
var stack = await LocalWorkspace.CreateOrSelectStackAsync(stackArgs);
...
var result = await stack.UpAsync(
new UpOptions
{
Diff = !settings.NoDiff,
ExpectNoChanges = settings.ExpectNoChanges,
OnStandardOutput = AnsiConsole.WriteLine
});
if (result.Summary.Result != UpdateState.Succeeded)
{
return -1;
}
...
return 0;
} |
|
@gitfool thanks for the confirmation and the code. @orionstudt @vipentti It sounds like this issue is especially bad for inline programs? Any thoughts on why? |
|
Possibly the LocalRuntimeService isn't returning to the CLI correctly |
|
@mikhailshilkov OK so #7299 should resolve this. I brought in one the tests that @vipentti wrote to confirm. |
If an exception happens while executing my stack code, it causes the program to exit, and any resource that would have been defined after the exception gets marked for deletion (or worse actually deleted). I have only tried this in the dotnet SDK.
Expected behavior
I would have expected the program exiting on error would cause pulumi to not move forward with the preview/up operation.
Current behavior
Currently, pulumi does see the exception, and logs it as a diagnostic. But it then continues as if your program was complete and any resources that weren't yet created act as if you were removing them from your source code / stack.
Steps to reproduce
successful
pulumi up:followed by a failed preview attempt:
or if you aren't careful, a problematic
up:Context (Environment)
This tripped me up as I was iterating on a stack I have been working on (luckily no real resources were harmed). Towards the beginning of my stack I usually
config.Requireall of my inputs. When I misspelled one of my config names, it failed super early and therefore pretty much every resource was marked for deletion.This test was run in the pulumi/pulumi:v3.2.1 docker container.
The text was updated successfully, but these errors were encountered: