Kubernetes 1.8.3 - Error: etcdserver: mvcc: required revision has been compacted.

[ghost]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
New changes performed from the master respond with:
Error: etcdserver: mvcc: required revision has been compacted.

Existing calico pod can't be scheduled, we are trying to kill the pod to let it restart since other replica's do work as expected.

What you expected to happen:
Pod should be deleted and restarted on the node.

How to reproduce it (as minimally and precisely as possible):
Simply give the kube-apiserver the delete command for this pod in the namespace kube-system

Anything else we need to know?:
We have etcd clustered on virtual machines outside the cluster, this has always worked as expected.

Environment:

• Kubernetes version (use kubectl version):
  1.8.3
• Cloud provider or hardware configuration:
  vSphere Virtual Machines
• OS (e.g. from /etc/os-release):
  CentOS 7.4
• Kernel (e.g. uname -a):
  4.13.12-1.el7.elrepo.x86_64
• Install tools:
  Kubeadm
• Others:
  Etcd version 3.2.7-1

[liggitt]

"etcdserver: mvcc: required revision has been compacted." is not an error, it's an expected message when a watch is attempted to be established at a resource version that has already been compacted

without more information, this isn't recreatable or debuggable.

do you have etcd, apiserver, kubelet, or controller manager logs?

[ghost]

@liggitt thanks for your feedback, every action gives the following output;
Error from server: etcdserver: mvcc: database space exceeded

Apparently there is a 2gb quota by default which is exceeded by 2/3 of our etcd nodes (replication I assume) which causes these other issue's.

[ghost]

"Unable to register node"

[jennybuckley]

/cc @jpbetz

[lavalamp]

Something is wrong with your etcd cluster, this is almost certainly not an apiserver problem. There's an alarm you might need to disable.

[ghost]

There was indeed an alarm, solved this issue by compacting the latest revision, defrag and turn the alarm off.

[lijinghuatongxue]

There was indeed an alarm, solved this issue by compacting the latest revision, defrag and turn the alarm off.

how turn the alarm off ?