Convert a Task to run as a Pod instead of a kubernetes job

[abhinavdas]

When we query the status of a task, we also care about Pod details, such as does the pod have some image pull failure and the phase of the pod (running, pending, etc). To do this, we need to query the Job status and then we also need to query the pod status of the pod that is created for the Job. It is also possible for the Pod to run into some error. In this case, the Job controller creates another Pod and restarts the Job. Thus, there can be many old Pods that had some error laying around. These pods should be deleted so that we are cleaning up resources.

These are some of the reasons why Job queries take long as we do the following:

1. Look up job status
2. Find the pod for this job
3. Delete old pods

There is no mapping that allows 0(1) lookup of Job to Pod that I am aware of. To find the Pod for a Job, we need to filter out Pods with label selector job-name=THE_NAME_OF_JOB. For n pods, this would be an O(n) lookup. For argo, each Task is mapped to a Job and each Job creates at least 1 Pod. So, for n Tasks, we will have at least n Pods. The status of each such Task is going to be periodically queried by the workflow executor and this makes the sum of these operations O(n^2).

Since, workflow executor already knows how to recreate the Pod on failure, node termination, etc, we do not need to use a Job controller. If we use Pods, then we avoid the O(n) iteration to look for pods. We also, avoid the cleanup required for old Pods created by Job controller.

[abhinavdas]

Upgrade Note

If you upgrade a cluster that was running workflows before this change then you may end up with kubernetes jobs and pods that are left around until they are garbage collected by kubernetes. To avoid this situation there are two choices:

1. You can finish all existing workflows (or cancel them) before starting the upgrade.
2. If you are an advanced argo user, then you can use console commands to delete jobs in the axuser namespace. The steps to do this are the following:
   2.1 source bash-helpers.sh from argo source code
   2.2 kdef CLUSTER-NAME to select the cluster that you want to work on
   2.3 kns axuser to change the default namespace to axuser
   2.4 k delete jobs --all to delete all existing kubernetes jobs created in axuser
   The delete jobs will be restarted as Pods in sometime.

[abhinavdas]

@cyee88 I have added a release note for this bug. This will be required for upgrade from any cluster that does not have this code to one that is getting upgraded to one that does.