[pure k8s] K8s's pod eviction may not trigger when job container uses too much disk space

[wangdian]

K8s's pod eviction may not trigger when job container uses too much disk space.

According to k8s doc, there are two fs partitions that kubelet supports:

1. The nodefs filesystem that kubelet uses for volumes, daemon logs, etc.
2. The imagefs filesystem that container runtimes uses for storing images and container writable layers.

The imagefs limit is optional, so it may not configured by default. The result is if nodefs and imagefs are different, k8s may not evict pod that consumes large disk space.

Plus, if a pod is evicted by k8s, PAI treat it as system failure and retry without increase retry count. The job will always retry.

[fanyangCS]

closed and tracked in #4694?