kube-apiserver[14274]: E1015 10:17:36.804307 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted

[luciferhs]

my kubernetes version 1.16.0 ,etcd version 3.3.13,binary deployment

After deployment  i foud here is somthing wrong with kube-apiserver and somthingelse

it shows  kube-apiserver[14274]: E1015 12:26:55.231545   14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted    and  When I perform   "kubectl get cs"
it shows "unknown"   both etcd and kube-controller-manager and kube-scheduler 

[tedyu]

Can you post more of the kube-apiserver log (prior to the error) ?

Thanks

[luciferhs]

It's been exporting like the following logs：

Oct 15 16:30:19 node2 kube-apiserver[14274]: I1015 16:30:19.873780 14274 aggregator.go:255] Updating OpenAPI spec because v1beta1.metrics.k8s.io is removed
Oct 15 16:30:20 node2 kube-apiserver[14274]: I1015 16:30:20.830058 14274 aggregator.go:258] Finished OpenAPI spec generation after 956.227762ms
Oct 15 16:30:20 node2 kube-apiserver[14274]: I1015 16:30:20.900320 14274 trace.go:116] Trace[1941197594]: "Delete" url:/api/v1/namespaces/monitoring/secrets (started: 2019-10-15 16:30:20.117717184 +0800 CST m=+90464.464151205) (total time: 782.559416ms):
Oct 15 16:30:20 node2 kube-apiserver[14274]: Trace[1941197594]: [782.559416ms] [782.559416ms] END
Oct 15 16:31:17 node2 kube-apiserver[14274]: I1015 16:31:17.277150 14274 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Oct 15 16:31:17 node2 kube-apiserver[14274]: I1015 16:31:17.277199 14274 controller.go:130] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Nothing (removed from the queue).
Oct 15 16:51:28 node2 kube-apiserver[14274]: I1015 16:51:28.358702 14274 cacher.go:771] cacher (*core.Pod): 1 objects queued in incoming channel.
Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.872782 14274 trace.go:116] Trace[1275503428]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager (started: 2019-10-15 17:12:02.568153143 +0800 CST m=+92966.914587163) (total time: 2.304580236s):
Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[1275503428]: [2.304488648s] [2.304387033s] About to write a response
Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.873225 14274 trace.go:116] Trace[80135351]: "GuaranteedUpdate etcd3" type:*coordination.Lease (started: 2019-10-15 17:12:03.74252003 +0800 CST m=+92968.088954056) (total time: 1.130671572s):
Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[80135351]: [1.130542609s] [1.130244463s] Transaction committed
Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.873373 14274 trace.go:116] Trace[1630583982]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node1 (started: 2019-10-15 17:12:03.742346764 +0800 CST m=+92968.088780788) (total time: 1.131003735s):
Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[1630583982]: [1.130912001s] [1.130792156s] Object stored in database
Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.878601 14274 trace.go:116] Trace[890369680]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 17:12:03.67061806 +0800 CST m=+92968.017052083) (total time: 1.207947031s):
Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[890369680]: [1.20786511s] [1.207767378s] About to write a response
Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.878850 14274 trace.go:116] Trace[581870170]: "Get" url:/api/v1/namespaces/kube-system (started: 2019-10-15 17:12:03.595704758 +0800 CST m=+92967.942138786) (total time: 1.283103337s):
Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[581870170]: [1.28304395s] [1.282994205s] About to write a response
Oct 15 17:16:12 node2 kube-apiserver[14274]: I1015 17:16:12.824757 14274 controller.go:606] quota admission added evaluator for: ingressroutes.traefik.containo.us
Oct 15 17:30:03 node2 kube-apiserver[14274]: E1015 17:30:03.290378 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 17:48:22 node2 kube-apiserver[14274]: E1015 17:48:22.343574 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 18:09:26 node2 kube-apiserver[14274]: E1015 18:09:26.434478 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 18:24:55 node2 kube-apiserver[14274]: E1015 18:24:55.506139 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 18:34:18 node2 kube-apiserver[14274]: E1015 18:34:18.603663 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 18:44:13 node2 kube-apiserver[14274]: E1015 18:44:13.683862 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 18:54:20 node2 kube-apiserver[14274]: I1015 18:54:20.659228 14274 trace.go:116] Trace[660470849]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 18:54:19.442949099 +0800 CST m=+99103.789383131) (total time: 1.216236442s):
Oct 15 18:54:20 node2 kube-apiserver[14274]: Trace[660470849]: [1.216117936s] [1.216013086s] About to write a response
Oct 15 18:57:05 node2 kube-apiserver[14274]: E1015 18:57:05.789816 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 19:06:04 node2 kube-apiserver[14274]: E1015 19:06:04.863936 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 19:12:23 node2 kube-apiserver[14274]: E1015 19:12:23.883327 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 19:15:58 node2 kube-apiserver[14274]: I1015 19:15:58.824401 14274 trace.go:116] Trace[18979956]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager (started: 2019-10-15 19:15:57.086358734 +0800 CST m=+100401.432792764) (total time: 1.737997065s):
Oct 15 19:15:58 node2 kube-apiserver[14274]: Trace[18979956]: [1.737904096s] [1.737798693s] About to write a response
Oct 15 19:15:58 node2 kube-apiserver[14274]: I1015 19:15:58.862034 14274 trace.go:116] Trace[1901767773]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 19:15:56.716418229 +0800 CST m=+100401.062852253) (total time: 2.145574634s):
Oct 15 19:15:58 node2 kube-apiserver[14274]: Trace[1901767773]: [2.145480381s] [2.145367208s] About to write a response
Oct 15 19:26:43 node2 kube-apiserver[14274]: E1015 19:26:43.953031 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 19:40:56 node2 kube-apiserver[14274]: E1015 19:40:56.047028 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 19:49:07 node2 kube-apiserver[14274]: E1015 19:49:07.105254 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 19:53:08 node2 kube-apiserver[14274]: I1015 19:53:08.653563 14274 trace.go:116] Trace[563357405]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 19:53:06.504319498 +0800 CST m=+102630.850753529) (total time: 2.149203504s):
Oct 15 19:53:08 node2 kube-apiserver[14274]: Trace[563357405]: [2.149134065s] [2.149002004s] About to write a response
Oct 15 19:53:08 node2 kube-apiserver[14274]: I1015 19:53:08.677481 14274 trace.go:116] Trace[1810182771]: "Get" url:/api/v1/namespaces/kube-system (started: 2019-10-15 19:53:07.485786405 +0800 CST m=+102631.832220428) (total time: 1.191654916s):
Oct 15 19:53:08 node2 kube-apiserver[14274]: Trace[1810182771]: [1.191600621s] [1.191552706s] About to write a response
Oct 15 20:04:25 node2 kube-apiserver[14274]: E1015 20:04:25.192296 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 20:27:01 node2 kube-apiserver[14274]: E1015 20:27:01.263609 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 20:35:04 node2 kube-apiserver[14274]: E1015 20:35:04.323793 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
Oct 15 20:47:22 node2 kube-apiserver[14274]: E1015 20:47:22.350260 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted

Can I ignore this mistake？ Some people say this is the etcd version problem，but when i deployed the kuberntes version 1.15+，there are no longer error logs.

Thank you for reply！

[neolit123]

/sig api-machinery

[fedebongio]

/assign @wenjiaswe

[wenjiaswe]

etcd is the Kubernetes’ backing store for all cluster data, it has a history compaction mechanism to avoid performance degradation and eventual storage space exhaustion, here are some docs on etcd repo.

"watch chan error: etcdserver: mvcc: required revision has been compacted" literally means the watched revision is compacted. This is working as designed, when attempts to re-establish a watch from a resourceVersion that is no longer available would prompt the caller to re-list objects and obtain a new current resourceVersion to watch from..

For your comment "and When I perform "kubectl get cs" it shows "unknown" both etcd and kube-controller-manager and kube-scheduler ", what does that mean? Could you paste the output of "kubectl get cs" here?

[luciferhs]

[root@master istio]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready 6d2h v1.16.0
node1 Ready 6d2h v1.16.0
node2 Ready 6d2h v1.16.0
node3 Ready 6d2h v1.16.0
[root@master istio]# kubectl get cs
NAME AGE
controller-manager
scheduler
etcd-2
etcd-1
etcd-0

When I perform "kubectl get cs" it shows "unknown" both etcd and kube-controller-manager and kube-scheduler "。 i have no idea,please give some sugesstions!thanks.

[databus23]

This is working as designed.

@wenjiaswe Whats confusing about this message then is that it is logged as an ERROR. If its part of normal operations it should not be logged as an error or probably not logged at all.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yacklebeam]

I'd like to /reopen this issue. I agree that behavior working as designed shouldn't be logged as an error. Errors are something that we set alarms on, to draw the attention of those who maintain the cluster.
Logging errors for expected behavior makes it MUCH harder to find and deal with real errors that are affecting availability.

Completely agree...the fact that something to be ignored (because it's "working as designed") is being logged as an error should be a red flag for a maintainer of any project, and something to /fix/.

EDIT: Appears to be fixed in 1.8 here, #84602

[demisx]

Still seeing the same logged as error in the Kube API Server log (kubernetes v1.18.6):

W0810 19:07:50.776360       1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

[Erick-shi]

It's been exporting like the following logs：

Oct 15 16:30:19 node2 kube-apiserver[14274]: I1015 16:30:19.873780 14274 aggregator.go:255] Updating OpenAPI spec because v1beta1.metrics.k8s.io is removed Oct 15 16:30:20 node2 kube-apiserver[14274]: I1015 16:30:20.830058 14274 aggregator.go:258] Finished OpenAPI spec generation after 956.227762ms Oct 15 16:30:20 node2 kube-apiserver[14274]: I1015 16:30:20.900320 14274 trace.go:116] Trace[1941197594]: "Delete" url:/api/v1/namespaces/monitoring/secrets (started: 2019-10-15 16:30:20.117717184 +0800 CST m=+90464.464151205) (total time: 782.559416ms): Oct 15 16:30:20 node2 kube-apiserver[14274]: Trace[1941197594]: [782.559416ms] [782.559416ms] END Oct 15 16:31:17 node2 kube-apiserver[14274]: I1015 16:31:17.277150 14274 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io Oct 15 16:31:17 node2 kube-apiserver[14274]: I1015 16:31:17.277199 14274 controller.go:130] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Nothing (removed from the queue). Oct 15 16:51:28 node2 kube-apiserver[14274]: I1015 16:51:28.358702 14274 cacher.go:771] cacher (*core.Pod): 1 objects queued in incoming channel. Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.872782 14274 trace.go:116] Trace[1275503428]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager (started: 2019-10-15 17:12:02.568153143 +0800 CST m=+92966.914587163) (total time: 2.304580236s): Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[1275503428]: [2.304488648s] [2.304387033s] About to write a response Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.873225 14274 trace.go:116] Trace[80135351]: "GuaranteedUpdate etcd3" type:*coordination.Lease (started: 2019-10-15 17:12:03.74252003 +0800 CST m=+92968.088954056) (total time: 1.130671572s): Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[80135351]: [1.130542609s] [1.130244463s] Transaction committed Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.873373 14274 trace.go:116] Trace[1630583982]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node1 (started: 2019-10-15 17:12:03.742346764 +0800 CST m=+92968.088780788) (total time: 1.131003735s): Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[1630583982]: [1.130912001s] [1.130792156s] Object stored in database Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.878601 14274 trace.go:116] Trace[890369680]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 17:12:03.67061806 +0800 CST m=+92968.017052083) (total time: 1.207947031s): Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[890369680]: [1.20786511s] [1.207767378s] About to write a response Oct 15 17:12:04 node2 kube-apiserver[14274]: I1015 17:12:04.878850 14274 trace.go:116] Trace[581870170]: "Get" url:/api/v1/namespaces/kube-system (started: 2019-10-15 17:12:03.595704758 +0800 CST m=+92967.942138786) (total time: 1.283103337s): Oct 15 17:12:04 node2 kube-apiserver[14274]: Trace[581870170]: [1.28304395s] [1.282994205s] About to write a response Oct 15 17:16:12 node2 kube-apiserver[14274]: I1015 17:16:12.824757 14274 controller.go:606] quota admission added evaluator for: ingressroutes.traefik.containo.us Oct 15 17:30:03 node2 kube-apiserver[14274]: E1015 17:30:03.290378 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 17:48:22 node2 kube-apiserver[14274]: E1015 17:48:22.343574 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 18:09:26 node2 kube-apiserver[14274]: E1015 18:09:26.434478 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 18:24:55 node2 kube-apiserver[14274]: E1015 18:24:55.506139 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 18:34:18 node2 kube-apiserver[14274]: E1015 18:34:18.603663 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 18:44:13 node2 kube-apiserver[14274]: E1015 18:44:13.683862 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 18:54:20 node2 kube-apiserver[14274]: I1015 18:54:20.659228 14274 trace.go:116] Trace[660470849]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 18:54:19.442949099 +0800 CST m=+99103.789383131) (total time: 1.216236442s): Oct 15 18:54:20 node2 kube-apiserver[14274]: Trace[660470849]: [1.216117936s] [1.216013086s] About to write a response Oct 15 18:57:05 node2 kube-apiserver[14274]: E1015 18:57:05.789816 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 19:06:04 node2 kube-apiserver[14274]: E1015 19:06:04.863936 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 19:12:23 node2 kube-apiserver[14274]: E1015 19:12:23.883327 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 19:15:58 node2 kube-apiserver[14274]: I1015 19:15:58.824401 14274 trace.go:116] Trace[18979956]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager (started: 2019-10-15 19:15:57.086358734 +0800 CST m=+100401.432792764) (total time: 1.737997065s): Oct 15 19:15:58 node2 kube-apiserver[14274]: Trace[18979956]: [1.737904096s] [1.737798693s] About to write a response Oct 15 19:15:58 node2 kube-apiserver[14274]: I1015 19:15:58.862034 14274 trace.go:116] Trace[1901767773]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 19:15:56.716418229 +0800 CST m=+100401.062852253) (total time: 2.145574634s): Oct 15 19:15:58 node2 kube-apiserver[14274]: Trace[1901767773]: [2.145480381s] [2.145367208s] About to write a response Oct 15 19:26:43 node2 kube-apiserver[14274]: E1015 19:26:43.953031 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 19:40:56 node2 kube-apiserver[14274]: E1015 19:40:56.047028 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 19:49:07 node2 kube-apiserver[14274]: E1015 19:49:07.105254 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 19:53:08 node2 kube-apiserver[14274]: I1015 19:53:08.653563 14274 trace.go:116] Trace[563357405]: "Get" url:/api/v1/namespaces/kube-system/endpoints/kube-scheduler (started: 2019-10-15 19:53:06.504319498 +0800 CST m=+102630.850753529) (total time: 2.149203504s): Oct 15 19:53:08 node2 kube-apiserver[14274]: Trace[563357405]: [2.149134065s] [2.149002004s] About to write a response Oct 15 19:53:08 node2 kube-apiserver[14274]: I1015 19:53:08.677481 14274 trace.go:116] Trace[1810182771]: "Get" url:/api/v1/namespaces/kube-system (started: 2019-10-15 19:53:07.485786405 +0800 CST m=+102631.832220428) (total time: 1.191654916s): Oct 15 19:53:08 node2 kube-apiserver[14274]: Trace[1810182771]: [1.191600621s] [1.191552706s] About to write a response Oct 15 20:04:25 node2 kube-apiserver[14274]: E1015 20:04:25.192296 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 20:27:01 node2 kube-apiserver[14274]: E1015 20:27:01.263609 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 20:35:04 node2 kube-apiserver[14274]: E1015 20:35:04.323793 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted Oct 15 20:47:22 node2 kube-apiserver[14274]: E1015 20:47:22.350260 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted

Can I ignore this mistake？ Some people say this is the etcd version problem，but when i deployed the kuberntes version 1.15+，there are no longer error logs.

Thank you for reply！

do you have do etcd compact Manual?

[eigood]

Still seeing the same logged as error in the Kube API Server log (kubernetes v1.18.6):

W0810 19:07:50.776360       1 watcher.go:199] watch chan error: etcdserver: mvcc: required revision has been compacted

That's a "warning", because it has a "W" as the first character. You can also see "I" and "E" lines.

[ale-telefonica]

I´m still seeing these logs in version 1.23.1

W0411 04:07:17.644840 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted

[nrktkt]

@ale-telefonica

That's a "warning", because it has a "W" as the first character. You can also see "I" and "E" lines [for info and error].