Produce console output while waiting for slow-to-create resources

[floyd-may]

When resources are slow to create, the pulumi CLI produces no console output while waiting for those resources. When running pulumi in a hosted CI environment like, say, CircleCI, that can cause the CI job to be forcibly terminated. CircleCI, for instance, kills jobs that produce no console output after 10 minutes. Naturally, that means the Pulumi update needs to be cancelled via pulumi cancel because it was forcibly killed while it was still running. Therefore, subsequent CI jobs will fail immediately.

Proposed feature: --logwaitmessages[=<interval>] flag

I propose that a flag for the pulumi CLI be added so that a "waiting..." message (or similar) will be output to stdout at least every interval seconds, with a sane default (30?). This will prevent pulumi from being auto-killed for producing no output for too long.

[susanev]

@dixler heres a few recommendations, pls only take what is useful to you...

• consider displaying feedback messages by default and then having a flag to turn off the feedback messages instead of what is proposed in the issue
• use feedback messages that give as much context and information as possible (e.g, provisioning 2/4 resources...); note the ... is a standard pattern to communicate that something is happening so lets try and append that if possible
• display the feedback messages every 10 minutes even if another resources has not yet been provisioned (see comments, but this is important for liveness checks)

[Frassle]

if we cant provide a count, consider displaying feedback messages using the Fibonacci sequence for lengths of time, so after 1 minute, 2 minutes, 3 minutes, 5 minutes, 8 minutes, 13 minutes, etc... maxing out at 55 minutes. (e.g, provisioning...)

That's questionable. A lot of CI systems are going to use a fixed timespan for liveness checks and as soon as we are silent beyond that we'd get killed.

[susanev]

@Frassle okay, we can do a fixed time instead. @dixler and i were chatting about wanting to max out, do ya think that's not needed?

[floyd-may]

My preference would be to leave the behavior unchanged with regard to maxing out; i.e. if a slow-to-create resource currently causes Pulumi to time out and give up, I'd say preserve that behavior; otherwise, if Pulumi will currently wait indefinitely, keep that behavior.

[dixler]

Any ideas on why the spinner is basically disabled in non-interactive mode?

pulumi/pkg/backend/display/diff.go

Lines 55 to 60 in 9df8d3a

	if stdout == os.Stdout && stderr == os.Stderr && opts.IsInteractive {
	spinner, ticker = cmdutil.NewSpinnerAndTicker(prefix, nil, 8 /*timesPerSecond*/)
	} else {
	spinner = &nopSpinner{}
	ticker = time.NewTicker(math.MaxInt64)
	}

@Frassle

[Frassle]

Any ideas on why the spinner is basically disabled in non-interactive mode?

The spinner assumes it can mutate the screen buffer by sending backspace chars to Printf. That only works if stdout is pointing to a tty (Which is basically what IsInteractive is reporting).

Having said that NewSpinnerAndTicker already has code to check if IsInteractive is true and falls back to just printing plain dots if we don't have a tty. So I'm not sure why this outer check is also here.

[floyd-may]

@dixler @pgavlin @Frassle thank you, thank you, thank you for this! Can one of you possibly tell me when to expect this to be available in the Pulumi CLI?

[Frassle]

The next scheduled release is next Wednesday (23rd)