Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scaling StatefulSets with waiting seems to fail #1626

Closed
scholzj opened this issue Jul 13, 2019 · 1 comment
Closed

Scaling StatefulSets with waiting seems to fail #1626

scholzj opened this issue Jul 13, 2019 · 1 comment

Comments

@scholzj
Copy link
Contributor

scholzj commented Jul 13, 2019

When trying to scale StatefulSet using the scale(...) method with the wait parameter set to true, I seem to always get exception similar to this:

2019-07-12 11:25:37 ERROR StatefulSetOperationsImpl:115 - Error while waiting for resource to be scaled.
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [StatefulSet]  with name: [my-cluster-kafka]  in namespace: [kafka-cluster-test]  failed.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:228) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:161) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.RollableScalableResourceOperation.lambda$waitUntilScaled$0(RollableScalableResourceOperation.java:94) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.io.InterruptedIOException: interrupted
        at okio.Timeout.throwIfReached(Timeout.java:146) ~[com.squareup.okio.okio-1.15.0.jar:?]
        at okio.Okio$2.read(Okio.java:137) ~[com.squareup.okio.okio-1.15.0.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[com.squareup.okio.okio-1.15.0.jar:?]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:354) ~[com.squareup.okio.okio-1.15.0.jar:?]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:226) ~[com.squareup.okio.okio-1.15.0.jar:?]
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$1(HttpClientUtils.java:100) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:92) ~[com.squareup.okhttp3.okhttp-3.12.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:330) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:311) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:749) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:217) ~[io.fabric8.kubernetes-client-4.3.0.jar:?]
        ... 9 more

And the scaling seems to always fail (i.e. print the error log message about giving up). I did a little bit of debugging and it seems that the problem is that rollingTimeout for which it is supposed to wait for the scaling to happen is always set to 0. This cause two things:

  • The waiting is never successful because it never waits
  • And since it never waits it interrupts the poller thread when it cancels it before the first run had a chance to finish.

The default config seems to set the default value to 900000ms. But that seems to be never transfered into the RollingOperationContext and used in the scale method. Is this some bug in the code where the rollingTimeout is not set properly? Or am I supposed to set it somewhere before doing the scaling? Thanks for your help!
@scholzj scholzj changed the title Scaling statefulsets with waiting seems to throw exception Scaling StatefulSets with waiting seems to fail Jul 13, 2019
scholzj added a commit to scholzj/kubernetes-client that referenced this issue Jul 23, 2019
@scholzj
Fix waiting for StatefulSet scaling - Closes fabric8io#1626
955bfcd
scholzj added a commit to scholzj/kubernetes-client that referenced this issue Jul 23, 2019
@scholzj
Fix waiting for StatefulSet scaling - Closes fabric8io#1626
663a64b
@scholzj scholzj mentioned this issue Jul 23, 2019
Merged
@sbaier1
Copy link

sbaier1 commented Aug 27, 2019

also seeing this issue, are you planning to update your PR? @scholzj

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@scholzj @sbaier1