The resourceVersion for the provided list is too old

[apratina]

We get the following error msg at the time of executing kubect get pod command.

Error:

Error from server (Expired): The resourceVersion for the provided list is too old.\n

Here are the details of k8s version:

• Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
• Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

We would like to understand what does this error mean?

[pacoxu]

Would you run kubectl get pod --v=4 or --v=6 to get more info about the request to apiserver?

/kind support
/remove-kind bug
According to current descriptions, we cannot figure out whether this is a bug.
/priority awaiting-more-evidence

[apratina]

Let me try to gather some more detail. In the mean time could you share some more details on what does this msg mean and why we should see this?

[neolit123]

related: kubernetes/kubectl#965
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/errors.go#L35

IIUC, this can happen in a case with a lot of pods and slow etcd response times.

/sig cli api machinery

[k8s-ci-robot]

@neolit123: The label(s) sig/api, sig/machinery cannot be applied, because the repository doesn't have them.

In response to this:

related: kubernetes/kubectl#965
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/errors.go#L35

IIUC, this can happen in a case with a lot of pods and slow etcd response times.

/sig cli api machinery

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vsxen]

#72187

[lauchokyip]

Hi @apratina . The resourceVersion field is a way to keep track of the persisted version of the object. Every X minutes, the api server will make a call to etcd's database which will truncate old resourceVersion. You are getting the error because the resourceVersion has been deleted and the api server is returning HTTP code 140

A very easy way to reproduce this:

1. Let the api server listen to port X, willl use port 8080 as an example

kubectl proxy --port=8080

2. Open your browser and type

http://localhost:8080/api/v1/pods

You will see something like

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "264927"
  },

3. From the List chart over here (second row), if we set the limit=X and the resourceVersion=X then we can get the exact resourceVersion for the PodList. So type

http://localhost:8080/api/v1/pods?limit=50&resourceVersion=264927

and try decreasing the resourceVersion by 100 each time, for example

http://localhost:8080/api/v1/pods?limit=50&resourceVersion=264027

then

http://localhost:8080/api/v1/pods?limit=50&resourceVersion=263927

Eventually you will see the error message The resourceVersion for the provided list is too old.

Just like what @neolit123 suggested, it might be because you have too many pods and slow etcd response times

[fedebongio]

/triage accepted
/cc @yliaog to dig on a old bug related to this.

[yliaog]

i was thinking about #91073, it is about 'too large RV', not 'too old RV' though.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kundan2707]

/reopen

[k8s-ci-robot]

@kundan2707: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kundan2707]

Again reported here
projectcalico/calico#6032

[Vandit1604]

From what i could understand after reading @lauchokyip 's comment and Efficient detection of changes, I read this

If a client watch is disconnected then that client can start a new watch from the last returned resourceVersion; the client could also perform a fresh get / list request and begin again. See Resource Version Semantics for more detail.

and the first thought that came into my mind was

Why don't we automatically start a new watch from the last returned resourceVersion or a fresh get / list request when a client watch is disconnected?

IMHO new watch from the last returned resourceVersion should be the preferred approach.
If it is something we didn't needed before but could use now I would love to work on that.