Bugfix: Add image building job deletion delay

[deadlycoconuts]

Context

As of present, when the Turing API server triggers an image building job as part of launching a batch ensembling job/service, it checks for the presence of an existing image building job in the cluster with the same name. The status of the image building job found (if it is found), corresponds to a variety of situations:

• Found; running -> an existing image building job is already taking place and still running -> wait for the job to complete
• Found; failed -> an existing image building job has failed -> delete the old job -> begin a new one
• Not found -> begin a new one

In particular, we're interested in the second scenario, where there a new job is triggered right after the deletion of the old (failed) run (see this for more details):

err = ib.clusterController.DeleteJob(context.Background(), ib.imageBuildingConfig.BuildNamespace, job.Name)
if err != nil {
	log.Errorf("error deleting job: %v", err)
	return "", ErrDeleteFailedJob
}

job, err = ib.createKanikoJob(kanikoJobName, imageRef, request.ArtifactURI, request.BuildLabels,
	request.EnsemblerFolder, request.BaseImageRefTag)
if err != nil {
	log.Errorf("unable to build image %s, error: %v", imageRef, err)
	return "", ErrUnableToBuildImage
}

A potential problem can occur whereby the creation of the new job happens too quickly after the deletion of the job, due to the k8s resources having insufficient time to be removed from the cluster (the kube API server would return an error saying that the creation of the new job has failed due to the existence of another job with the very same name). When this happens, the (new) image building job fails immediately.

This PR thus introduces a new method to wait for the deletion to be complete before proceeding with the creation of another image builder job.

Main Modifications

• api/turing/imagebuilder/imagebuilder.go - Addition of an additional helper method to wait for image building jobs to be completely removed from the cluster

[krithika369]

LGTM. Thanks, @deadlycoconuts !