Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread spins logging "Error while waiting for resource to be scaled" #1856

Closed
k-wall opened this issue Nov 6, 2019 · 2 comments
Closed

Thread spins logging "Error while waiting for resource to be scaled" #1856

k-wall opened this issue Nov 6, 2019 · 2 comments

Comments

@k-wall
Copy link
Contributor

k-wall commented Nov 6, 2019

Using 4.6.2, I am seeing a situation where a kubernetes-client thread spins in a tight loop with the following exception logged, every 50ms:

2019-11-06T14:59:28.679Z ERROR [StatefulSetOperationsImpl] Error while waiting for resource to be scaled. java.lang.NullPointerException: null at io.fabric8.kubernetes.client.dsl.internal.StatefulSetOperationsImpl.getCurrentReplicas(StatefulSetOperationsImpl.java:72) at io.fabric8.kubernetes.client.dsl.internal.StatefulSetOperationsImpl.getCurrentReplicas(StatefulSetOperationsImpl.java:35) at io.fabric8.kubernetes.client.dsl.internal.RollableScalableResourceOperation.lambda$waitUntilScaled$0(RollableScalableResourceOperation.java:104) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 201

I notice that StatefulSetOperationsImpl#getObservedGeneration is careful to guard against a null current.getStatus(), but StatefulSetOperationsImpl.getCurrentReplicas does not.

k-wall added a commit to k-wall/kubernetes-client that referenced this issue Nov 6, 2019
k-wall added a commit to k-wall/kubernetes-client that referenced this issue Nov 6, 2019
@k-wall
Copy link
Contributor Author

k-wall commented Nov 7, 2019

@rohanKanojia

I understand more about this issue now. It comes about when you've io.fabric8.kubernetes.client.DefaultKubernetesClient#load a resource such as a statefulset then call io.fabric8.kubernetes.client.dsl.Deletable#delete on the returned object.

In this case, the object is never fetched from the server (reloadingFromServer is false so io.fabric8.kubernetes.client.dsl.base.BaseOperation#getMandatory returns the existing object). In the case of a scaleable resource, prior to deletion the code tries to scale to zero, but this fails with the NPE if the resource lacks a Status section.

My initial PR doesn't really help. The code still just spins comparing -1 != 0 rather than throwing the NPE.

I think the code ought to omit the waitUntilScaled part if reloadingFromServer is false.

k-wall added a commit to k-wall/kubernetes-client that referenced this issue Nov 7, 2019
k-wall added a commit to k-wall/kubernetes-client that referenced this issue Nov 7, 2019
@rohanKanojia
Copy link
Member

@k-wall : apologies for the late reply, let me go through your new PR and see if can be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants