engine/engine_impl.go: Add a mutex around pod image updates for step progress.

[reltuk]

When a pipeline fans out a DAG, concurrent calls are made to (*Kubernetes).Run
to start each step in parallel. But starting a step requires a
Read/Modify/Write loop to the Kubernetes API, and so updates are highly likely
to fail with a concurrent modification error. A user encountered this behavior
here[1], and it's easy to reproduce by creating a pipeline with high fan out.

This commit adds a mutex around the Read/Modify/Write loop in
(*Kubernetes).start. We know that concurrent modifications from the single
process will cause failures, so there is no reason to dispatch them
concurrently.

This commit also changes the current backoff parameters (5 max tries, .1
jitter factor) to be a little more robust. In local testing, even with the
mutex, concurrent modification errors were still possible but were much rarer.

1: https://discourse.drone.io/t/kube-runner-limitations/6853

[CLAassistant]

All committers have signed the CLA.

[reltuk]

A different approach is to use PATCH to update the fields we care about without risk of concurrent modification error. That approach is outlined here: reltuk#1. Objectively, it might be better — higher inherent parallelism and potentially less load on the API server if we do see concurrent modification errors. But it has downsides, not least of which is the RBAC requirements change mentioned in that PR. It's also a bigger change semantically compared to the approach taken here