Added ability for deferred mutator execution

[andrewnester]

Changes

Added DeferredMutator and bundle.Defer function which allows to always execute some mutators either in the end of execution chain or after error occurs in the middle of execution chain.

Usage as follows:

deferredMutator := bundle.Defer([]bundle.Mutator{
    lock.Acquire()
    transform.DoSomething(),
    //...
}, []bundle.Mutator{
    lock.Release(),
})

In such case lock.Release() will always be executed: either when all operations above succeed or when any of them fails

Tests

Before the change

andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-10 16:41:22.902659 +0200 CEST. Use --force to override

After the change

andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy 
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized

[pietern]

Nice work, thanks!

[pietern]

Existing mutators return nested mutators instead of executing them in place. I like this better, tbh, so once this is merged we should change the prototype of the mutator interface to no longer return []Mutator and make all mutators execute nested mutators in place.

[pietern]

Nice! This makes me think we also need (separate PR of course) a bundle.Seq to execute multiple mutators in sequence such that we can compose lock.Acquire with a bundle.Defer where the unlock happens. As it is here, lock.Release would always be called even if lock.Acquire fails.

[andrewnester]

@pietern just to clarify to make sure I understand correctly. You propose to have a new method bundle.Seq with API like this

func Seq(mutators []Mutator) Mutator { ... }

seq := bundle.Seq(
		bundle.Defer([]bundle.Mutator{
			files.Upload(),
			artifacts.UploadAll(),
			terraform.Interpolate(),
			terraform.Write(),
			terraform.StatePull(),
			terraform.Apply(),
			terraform.StatePush(),
		}, []bundle.Mutator{
			lock.Release(),
		}))

deployPhase := []bundle.Mutator{
    lock.Acquire(),
    seq,
}

bundle.Seq will essentially transform list of mutators into Mutator object with Apply method so it can be added to the list of mutators

In such case seq will only be executed once lock.Acquire succeeds

[pietern]

Yes to the proposed prototype, but the call site would be slightly different and call the Seq function to compose call lock.Acquire and the defer block in sequence.

[andrewnester]

So something like this then?

deferred := bundle.Defer([]bundle.Mutator{
			files.Upload(),
			artifacts.UploadAll(),
			terraform.Interpolate(),
			terraform.Write(),
			terraform.StatePull(),
			terraform.Apply(),
			terraform.StatePush(),
		}, []bundle.Mutator{
			lock.Release(),
		})

seq := bundle.Seq([lock.Acquire(), deferred])

deployPhase := []bundle.Mutator{
   seq
}

Do we then really need bundle.Seq? What if bundle.Defer just returns Mutator instead of []Mutator?
In that case we would just have it used like

deployPhase := []bundle.Mutator{
   lock.Acquire(),
   deferred
}

@pietern any thoughts?

[pietern]

We don't if we leave everything else the same, so yeah you could leave as is for this PR.

The comment on bundle.Seq is because we have mixed usage of singular Mutator and []Mutator in a couple places and formalizing []Mutator into a bundle.Seq would get rid of that.

[pietern]

This means file deletion will happen even if the user doesn't consent. You could move files.Delete into the main block for the time being. Then when we have a bundle.Seq (see other comment), we can sequence this defer block with a final delete to happen after lock release has completed.