Work Queue Latency increased 10 times

[eshikhov]

When applying Linstore CRDs (group internal.linstor.linbit.com) Work Queue Latency increased 10 times

I rolled these CRDs into different clusters with the same effect everywhere

Only after the CRDs are removed, the delay of the work queue returns to normal time. On the graph you can see the moment of installing and removing CRDs.

Environment
OS: Talos v1.8.3
Kubernetes version: v1.30.7
Linstore CRDs: v1_27_1

[ghernadi]

I am not fully sure if I understand the issue here. Since I am not a k8s expert, please correct me if I'm wrong, but this is how I understand what you are saying:

The pure existence of LINSTOR's CRDs make k8s itself slower (higher worker queue latency).

If my understanding is correct, I am not really sure what LINSTOR can do about this. Can you try to create some arbitrary CRDs and check if k8s also gets slower from those? If so, I'd suggest you open an issue (or question if this is a known behavior) in k8s GitHub. Unless someone can point us to something that LINSTOR is doing wrong (apart from "using CRDs"), I do not see how we can help here.

[eshikhov]

Yes, you are correct, installing LINSTOR's CRDs makes k8s itself slower (higher worker queue latency).
Installing other CRDs, even more than LINSTOR's, has no negative impact.

[vladimirfx]

The pure existence of LINSTOR's CRDs make k8s itself slower (higher worker queue latency).

Yes, you understand correctly. Only Linstore CRDs slow down openapi_v3_contoller in order of magnitude. Other CRDs (a few hundred) do not affect that controller. Unfortunately, I can't find relevant K8s issues.
What is interesting is that when CRDs are modified by removing old versions - the response time for openapi_v3_contoller is normalized.

[eshikhov]

Tried it in two different clusters
One distribution on talos Kubernetes version v1.30.7
the other on k3s Kubernetes version v1.23.16

On different distributions and versions of Kubernetes Work Queue Latency increased

[ghernadi]

Sorry for the silence.. we are investigating and will try to come up with a solution, but since our todo-lists are quite full, it might take a while.

Thanks anyways for reporting!