fix: prevent terraform init races and corrupted cache directories

[deansheather]

Before we were creating a new initMu to guard terraform init calls for each executor, and a new executor for each job. This changes the initMut to be a global mutex to guard all terraform init calls.

In production we only use a single terraform cache directory shared for all provisioner daemons so there aren't any optimizations to this locking scheme that can be made that would help production deployments. I was going to have a global map from e.cachePath to a mutex but decided it wasn't worth it since we never use a different cache path between runs.

[mafredri]

@deansheather how does this change help? We were already sharing the server mutex with the executors.

If there were multiple servers sharing a cache path I’d understand the issue. I had always imagined we’d use subfolders for each server/provisionerd instance.

[deansheather]

All provisioners in the same process share the same cache directory, and each had it's own mutex.

[mafredri]

Ah I misunderstood your PR description I think. This change is fine for now but I do think we should just have IDs for each provisioner and create subpaths in the cache dir in the future.

[mafredri]

Ah right, I think one reason I implemented it like this was for better test concurrency. Right now I believe most tests using the provisions have their own cache path. So this global mutex might slow those down due to not betting able to run concurrently or reuse the cache.