Write rates of endpoints still high on 3.6.1

[smarterclayton]

API server write rates from 3.6.1 on a heavily loaded cluster.

{resource="endpoints",verb="PUT"} | 35.0125
{resource="pods",verb="PUT"} | 19.45416666666667
{resource="pods",verb="DELETE"} | 16.987499999999997
{resource="events",verb="POST"} | 16.841666666666665
{resource="pods",verb="POST"} | 16.104166666666664
{resource="nodes",verb="PATCH"} | 15.579166666666667
{resource="localsubjectaccessreviews",verb="POST"} | 8.433333333333334
{resource="builds",verb="PATCH"} | 8.108333333333333
{resource="daemonsets",verb="PUT"} | 6.304166666666666
{resource="replicasets",verb="POST"} | 4.279166666666667
{resource="subjectaccessreviews",verb="POST"} | 1.7291666666666667
{resource="tokenreviews",verb="POST"} | 0.9333333333333333
{resource="replicationcontrollers",verb="PUT"} | 0.4666666666666667
{resource="deploymentconfigs",verb="PUT"} | 0.3541666666666667
{resource="deploymentconfigs",verb="POST"} | 0.2583333333333333

@derekwaynecarr @openshift/sig-platform-management

[liggitt]

do we have large numbers of endpoints with no subsets? if so, kubernetes/kubernetes#50583 might be relevant

[smarterclayton]

Hrm, doubt it but maybe.

[smarterclayton]

I don't like seeing 16 pod deletes a second. That's probably bad.

[smarterclayton]

@openshift/sig-master

[smarterclayton]

Here's write volume to etcd before and after the 3.6.1 upgrade.

[smarterclayton]

The jump was Sept 26th 2:15 gmt. May be unrelated to update?

[liggitt]

what is the server version?