Retrying to start the Informer on AKS never succeeds and ends up in immediate ECONNRESET

[ivanstanev]

Hey 👋

We've been users of the client library in our product for a while now. Recently we noticed that connections to AKS suddenly get interrupted (roughly 5 minutes after start) and we stop getting notified of new workloads in the cluster. We noticed it is because we didn't have an error handler (as defined in the example: https://github.com/kubernetes-client/javascript/blob/master/examples/typescript/informer/informer.ts#L16-L22) and that AKS has a Load Balancer for the K8s API server that interrupts long-running connections after 5 minutes by default.

So we added the setTimeout() + informer.start() to try and fix this.

However, we find that this does not help and the informer ends up in an infinite loop where the API server immediately returns ECONNRESET, the informer tries to re-start after 5 seconds (due to setTimeout()), and our app never recovers - stuck in receiving ECONNRESET and retrying infinitely. Killing our Pod and starting from scratch fixes this - until the API server stops the connection and again ending in a loop of ECONNRESET and trying to start the informer.

We are using version 0.13.2 of the library.

I noticed this recent PR #576 fixes a connection leak and ensures abort() is called on the connection. Do you think this is related and it would help once it lands in a new release?

[brendandburns]

@ivanstanev I think it should help. I just pushed a new release (0.14.0) with the fix, please try it out!

[ivanstanev]

Thanks for the super quick response, going to try it out now and let it run over the weekend and let you know how it goes!

[ivanstanev]

@brendandburns I think the recent 0.14.2 release with the Keep Alive may help with this, I haven't updated this issue because it wasn't resolved yet. Will let you know if it resolves this on AKS now!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.