-
Notifications
You must be signed in to change notification settings - Fork 351
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Devspace Disconnections #639
Comments
FYI, dqfan2012 is one of our developers who is using devspace on our cluster and running into this issue. I can answer any questions about the cluster set up. My theory is the connection is being closed if there is no communication passing through for a while and then once devspace tries to send something through the closed connection, it throws an error.
I would suggest detecting the error when it occurs and attempting to reconnect. If the reconnection is successful, then continuing from there if possible. If the reconnect fails, that indicates a connectivity problem larger than just a tunnel timeout and then it would be appropriate to fail I think. |
@dqfan2012 @KaelBaldwin thanks for reporting this issue! Yes this sounds like kubernetes is closing the idle connections which leads to the described errors. This is a tricky issue because of the following things:
I think the best approach to tackle this issue would be to restart the sync, terminal and portforwarding completely, if any of these encounter a closed connection |
That makes sense, thanks @FabianKramm ! |
@KaelBaldwin @dqfan2012 quick update on this issue. I investigated a little bit and found out that the timeout when kubernetes kills idle connections can be configured in the kubelet via the '--streaming-connection-idle-timeout' flag (see kubernetes docs). I could easily reproduce this issue in minikube by setting this flag to 30s. Furthermore, I found a way in kubernetes how the underlying connection to the api server can be pinged periodically and hence avoid idle connection closing (because essentially they will never be idle with the constant pinging). I will implement this for the sync, terminal and port-forwarding and this should solve this issue without the need of a complete devspace restart |
@dqfan2012 @KaelBaldwin was this issue solved with v3.5.17? |
@dqfan2012 started having a new issue after upgrading to v3.5.17 and had to revert. I'm going to be trying it out and see if I have the same problem. But I think it would belong in a new issue and this one can be closed. |
thanks! |
What happened?
I get disconnected a few times per day with the following error:
What did you expect to happen instead?
I expect to stay connected.
How can we reproduce the bug? (as minimally and precisely as possible)
I disconnect from devspace normally if I allow cmd to idle too long. I think the error occurs when I do work that triggers a sync after an idle disconnection.
Local Environment:
Kubernetes Cluster:
Anything else we need to know?
I can include the error log and sync log if you need them.
/kind bug
The text was updated successfully, but these errors were encountered: