Fix k8s tag propagation and controller-uid detection

[sjawhar]

Purpose

• K8s tags are currently being pulled from the Task object (where they are always empty) instead of the correct Cloud object.
• When custom label are added to a k8s job, the automatically added controller-uid label disappears from the ObjectMeta, which then breaks things like waiting for the job to be created, etc.

Approach

• Pull the tags from Cloud
• Use the job's Spec.Selector.MatchLabels instead of ObjectMeta to retrieve controller-uid

Logs

This is what kubectl describe job shows on master:

Without custom labels/tags:

Labels:         controller-uid=5cf76b42-cbc8-484a-b496-6b867182e523
                job-name=tpi-3aw4vu6287ot1-2biu93iu-1tw3q6af
Annotations:    batch.kubernetes.io/job-tracking: 

With labels:

Labels:         app.kubernetes.io/instance=jenkins
Annotations:    app.kubernetes.io/instance: jenkins
                batch.kubernetes.io/job-tracking: 

[0x2b3bfa0]

Smoke tests are failing, can you please take a look?

[0x2b3bfa0]

It seems like smoke tests keep failing after retrying. 🤔

    task_smoke_test.go:133: 
        	Error Trace:	task_smoke_test.go:133
        	Error:      	Received unexpected error:
        	            	container "tpi-smoke-test-2996915558-5wp6qvfp-4rg3ne32" in pod "tpi-smoke-test-2996915558-5wp6qvfp-4rg3ne32-sxj56" is not available
        	Test:       	TestTaskSmoke/k8s

[0x2b3bfa0]

Finally tests are passing! 😅