Add more CRD validations markers to block invalid job on client apply

[Jeffwan]

A follow up of #1362

If required item missing, we should fail the job submission on the client instead of only reporting errors on the server side.
Since we don't have plan to use web hook, we can use CRD validations. Please double check if all our requirement can be satisfied by using these markers.

https://book.kubebuilder.io/reference/markers/crd-validation.html

[Jeffwan]

/kind enhancement

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.