ProwJob CRD is too large

[maboehm]

Hi,

the prowJob CR just recently crossed the threshold of being too large, for example running against a kind cluster:

$ kind create cluster
...
$ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

# this is the last commit this still works on
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/prow/e67659d368fd013492a9ce038d801ba8998b7d10/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml --server-side
customresourcedefinition.apiextensions.k8s.io/prowjobs.prow.k8s.io serverside-applied

# latest version fails
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/prow/main/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml --server-side
Error from server: etcdserver: request is too large

Am I doing something wrong when applying?

In terms of solving this issue, the main problem is that there is a lot of repetition, e.g. the taskPodTemplate appears several times. It would be possible to have some fields as e.g. apiextension.JSON or runtime.RawExtension and defer validation to a validatingwebhook. Which makes the output of e.g. kubectl explain less helpful.

[matthyx]

Hi @maboehm does it work if you kubectl create -f ... instead?
I think the main difference is that apply also saves the last applied configuration in the annotations, which doubles the size required and can more easily trigger the etcd limit.

[matthyx]

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/prow/main/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml --server-side

(apparently it doesn't, sorry)

[maboehm]

Either way, since tools like Helm or Flux use apply the size should work for that as well.

[lentzi90]

I have been using apply --server-side=true to avoid the annotation. It looks like the CRD itself is finally too large. With server side apply it worked up until e67659d (including). From 6405295 it is no longer possible to apply.

The only way to avoid this is to reduce the size of the CRD or configure etcd to allow a larger size. 🙁

[matthyx]

ok well I think we have to shrink it... @maboehm wanna have a try?