Use a per-stack PULUMI_HOME directory

[EronWright]

Proposed changes

Closes #483

This PR seeks to isolate the credentials associated with a given Stack, to solve the problem of credentials leaking across stacks. Some underlying details here:

1. Pulumi CLI stores login credentials in PULUMI_HOME (e.g. ~/.pulumi/credentials.json).
2. A side-effect of using PULUMI_ACCESS_TOKEN is that the CLI login credentials are set.
3. Pulumi CLI prefers the persisted login credentials to PULUMI_ACCESS_TOKEN.

This PR takes the conservative approach of encapsulating the PULUMI_HOME into a per-stack working directory, as opposed to reusing ~/.pulumi across stacks. The working directory is retained across reconciliation passes, and cleaned up during stack finalization. Note that the workspace directory is erased at the end of each reconciliation pass, as is the current behavior.

This PR does NOT solve the (lack of) mutability of PULUMI_ACCESS_TOKEN across stack updates.

Note that this PR contains some commits (related to hacking on the operator) that will be moved to a separate PR.

Technical Details

Relevant terminology used within the controller codebase:

• root directory - a temporary directory for each stack, retained until finalization
• home directory - the PULUMI_HOME directory, located within the stack's root directory
• workspace directory - the Pulumi workspace directory containing the program and stack configuration.

The current behavior of the operator is to erase the workspace directory on each reconciliation pass, e.g. to ensure a clean git checkout. This PR retains this behavior while keeping the home directory across passes, e.g. to reuse the providers.

Related issues (optional)

• Pulumi CLI shouldn't persist credentials when picked up via environment variables pulumi#13919

[rquitales]

It would be nice to also somehow include a check that the workspace folder existing when the stack is being reconciled to validate that we are indeed using a workspace folder.

[EronWright]

Sorry I don't see a decent way to do that.

[rquitales]

I was thinking of just spawning a goroutine that polls the existence of the workspace folder. Something like:

It("should purge the workspace dir", func() {
	rootDir := getRootDir(stack)
	Expect(k8sClient.Create(context.TODO(), stack)).To(Succeed())
	// Spy on the root directory in separate goroutine to make sure the workspace dir is created.
	var workspaceCreated bool
	done := make(chan struct{})
	go func() {
		for {
			select {
			case <-time.After(2 * time.Second):
				if exists(filepath.Join(rootDir, "workspace")) {
					workspaceCreated = true
					return
				}
			case <-done:
				return
			}
		}
	}()
	waitForStackSuccess(stack)
	close(done) // Clean-up spawned goroutine.
	Expect(workspaceCreated).To(BeTrue())
	Expect(exists(filepath.Join(rootDir, "workspace"))).To(BeFalse())
})

[EronWright]

I'm not thrilled at the lack of determinism, I think I'll try to make a hook for testing purposes.

[rquitales]

Let's expand on this in a follow-up PR.