"runners.actions.summerwind.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

[ealasgarov]

Describe the bug

When I'm installing actions-runner-controller with:

"kubectl apply -f https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.22.2/actions-runner-controller.yaml" command

I am getting below error:

"Error from server (Invalid): error when creating "https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.22.2/actions-runner-controller.yaml": CustomResourceDefinition.apiextensions.k8s.io "runners.actions.summerwind.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes"

To Reproduce

Steps to reproduce the behavior:

1. Run 'kubectl apply -f https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.22.2/actions-runner-controller.yaml'

Expected behavior

Resources should be installed as described in the Readme

Screenshots

Environment (please complete the following information):

• actions-runner-controller 0.22.2 (latest)
• AKS (kubernetes) version 1.22.6 (latest stable version)
• cert-manager installed, latest version

Additional context

I have downloaded that actions-runner-controller.yaml and I can see that custom resource definition has more than 30k lines, is that right?

[mumoshu]

@ealasgarov Hey. This is a known issue in Kubernetes(and kubectl-apply) that results in that an annotation automatically generated for server-side apply results in exceeding the K8s' own limit. Try using kubectl create instead.

[KENNYSOFT]

Possibly duplicate of #1102

[ealasgarov]

thanks a lot for comments, will try that!

[jonathanbeber]

can we maybe change in the readme the kubectl apply by kubectl create just to avoid having to search for that?

[billimek]

can we maybe change in the readme the kubectl apply by kubectl create just to avoid having to search for that?

@jonathanbeber do you happen to know if the kubectl create approach will work for updating CRDs?

I'm thinking about a scenario where the CRDs are already installed and we are upgrading to a new version which has changes to the CRDs. From my experience, kubectl create will fail if the resource already exists. If this is the case, it would require a deletion of the CRDs first followed by a fresh create.

The consequence of deleting the CRDs is that all the RunnerDeployments, Runners, etc will also be subsequently removed unless we maybe also add the --cascade=orphan argument to the kubectl delete operation?

To your point, I think we need to update the docs given the size of the CRDs and apply no longer working.

[mumoshu]

@billimek In case you missed it, you can use kubectl replace for upgrading CRDs as documented in https://github.com/actions-runner-controller/actions-runner-controller/blob/master/charts/actions-runner-controller/docs/UPGRADING.md#steps

[zerodayyy]

Are there any solutions to work around this issue when using ArgoCD? Unfortunately it's not possible to change the underlying logic there and installing CRDs manually is not an option

[toast-gear]

I don't think so outside of keeping custom version of the chart and adding the the argocd.argoproj.io/sync-options: Replace=true annotation to the CRDs in your custom chart.

If we produced the crds.yaml manifest as a single manifest attached to the GitHub release would that help at all? You could use the --skip-crds flag and install the CRDs separately using something custom to grab them from the release as a pre-install step before running the helm upgrade? EDIT you could package the CRDs up as an application and set that application to use replace instead of apply? https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#replace-resource-instead-of-applying-changes

[mumoshu]

Also see #1391 and #1256 (comment), #1102 (comment)

[mumoshu]

To me prometheus-operator/prometheus-operator#4439 (comment) looks like the best way; TL;DR; you can set the sync option via ArgoCD Application spec.

[shadiramadan]

I do think it would help to distribute the CRDs separately.

[SebastianScherer88]

I had the same issue and can confirm that the workaround shown here:

kubectl create -f ...  // For creating the resource
kubectl replace -f ... // For updating the resource

worked for me

[yyvess]

Try the option server-side

kubectl apply -f xxx.yaml --server-side

https://kubernetes.io/docs/reference/using-api/server-side-apply/

[jspc]

Try the option server-side

kubectl apply -f xxx.yaml --server-side

https://kubernetes.io/docs/reference/using-api/server-side-apply/

This worked perfectly for me. In my particular use case, I was running a github actions job that was just running the line:

$ kubectl apply -f https://github.com/actions/actions-runner-controller/releases/download/v0.22.0/actions-runner-controller.yaml

And so following the instruction to use kubctl create and then kubectl update would have sucked, frankly, because I'd have had to put some logic in to check whether a/some/all the CRDs existed first to determine which command to run.

The --server-side flag is what got my over that hurdle.

A note, though; if you've already tried to install the CRDs via kubectl apply -f... in the past, you will see a series of errors about ownership of various types.

Personally, I just deleted them (kubectl delete -f https://github.com/actions/actions-runner-controller/releases/download/v0.22.0/actions-runner-controller.yaml) myself, then ran again with the --server-side flag, which worked perfectly.

Another approach is to run kubectl apply -f ... with the flag --force-conflicts too, but I wasn't massively in love with doing that.

YMMV

[mumoshu]

Thank you everyone for reporting! I can confirm that kubectl apply --server-side should be the best way nowadays, as you kindly shared!

[biclighter81]

Argocd users can follow this documentation entry: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#server-side-apply

This syncPolicy works for me:

syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true