Pod are repeatedly created and deleted

[Wang-Kai]

What happened:

Due to the bug, the job was deleted from etcd, but it still remains in the cache. This causes the Volcano controller to create a pod, and then the GC controller deletes the pod instantly, resulting the operation being executed repeatedly. This causes a lot of load on the apiserver.

What you expected to happen:

The volcano controller's cache keep same with etcd, and should not fight with the GC controller about pod.

How to reproduce it (as minimally and precisely as possible):

When a pod is updating, delete the owner job instantly.

Anything else we need to know?:

Environment: linux

• Volcano Version: v1.8.2
• Kubernetes version (use kubectl version): v1.20
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
• Kernel (e.g. uname -a): Linux 5.10.0-103-bili-colo x86_64
• Install tools:
• Others: