pod watches stop sending data after a while, but remain connected in 1.18.4

[blushingpenguin]

What happened:

After upgrading from 1.16.10 to 1.18.4 watches stop working after a period of time. The connection to api server from the pod remains (i.e. it can be seen with lsof), but no further watch data is sent.

What you expected to happen:

With < 1.16.10 (back to 1.13), periodically the api server connection would drop after a period of time (which I believe was expected) -- in that case we'd just reconnect.

How to reproduce it (as minimally and precisely as possible):

I don't have an easy reproduction (the code is a custom job scheduler written in .NET using the k8s csharp client) -- I can probably boil it down if reproduction is necessary.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  1.18.4
• Size of cluster (how many worker nodes are in the cluster?)
  3 x b2ms
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
  dotnet / node microservices

Triage required from @Azure/aks-pm

Action required from @Azure/aks-pm

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.