Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"too old resource version" #383

Open
tiborsimko opened this issue Feb 7, 2023 · 0 comments
Open

"too old resource version" #383

tiborsimko opened this issue Feb 7, 2023 · 0 comments
Labels
priority/soon

Comments

@tiborsimko
Copy link
Member

Observed several times in the past the following problem in the job-controller container, when the pod is closing:

$ kubectl get pods | grep reana-run-batch-7f403e34-6513-4ff7-ba3b-8960a4bd7d2c-qtjn7
reana-run-batch-7f403e34-6513-4ff7-ba3b-8960a4bd7d2c-qtjn7        1/2     NotReady           0          15h

$ kubectl logs reana-run-batch-7f403e34-6513-4ff7-ba3b-8960a4bd7d2c-qtjn7 job-controller | tail -25
2023-02-06 17:03:32,462 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:32,500 | werkzeug | Thread-96 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:32] "POST /jobs HTTP/1.1" 201 -
2023-02-06 17:03:32,505 | werkzeug | Thread-97 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:32] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:32,509 | werkzeug | Thread-98 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:32] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:33,641 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:35,518 | werkzeug | Thread-99 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:35] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:38,526 | werkzeug | Thread-100 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:38] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:41,535 | werkzeug | Thread-101 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:41] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:44,543 | werkzeug | Thread-102 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:44] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:47,552 | werkzeug | Thread-103 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:47] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:50,560 | werkzeug | Thread-104 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:50] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:52,861 | root | kubernetes_job_monitor | INFO | Grabbing pod reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2-vbt6v logs ...
2023-02-06 17:03:52,951 | root | kubernetes_job_monitor | INFO | Storing job logs: ccb047dc-2df9-43c9-9a2a-4f7e9545596e
2023-02-06 17:03:52,991 | root | kubernetes_job_monitor | INFO | Cleaning Kubernetes job reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 ...
2023-02-06 17:03:53,072 | root | kubernetes_job_monitor | INFO | New Pod event received: MODIFIED
2023-02-06 17:03:53,072 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:53,077 | root | kubernetes_job_monitor | INFO | New Pod event received: DELETED
2023-02-06 17:03:53,077 | root | kubernetes_job_monitor | INFO | Kubernetes job id: reana-run-job-82214be2-2dbf-494c-b03f-a4eb82b161a2 failed.
2023-02-06 17:03:53,569 | werkzeug | Thread-105 | INFO | 127.0.0.1 - - [06/Feb/2023 17:03:53] "GET /jobs/ccb047dc-2df9-43c9-9a2a-4f7e9545596e HTTP/1.1" 200 -
2023-02-06 18:37:43,804 | root | kubernetes_job_monitor | ERROR | Error while connecting to Kubernetes API: (410)
Reason: Expired: too old resource version: 171643461 (171656967)

Let's investigate the root cause (and possibly upgrade Kubernetes cluster version?)
mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this issue Feb 27, 2023
@mdonadoni
k8s: use unique error messages
01c7ed9 
This commit changes the error messages used to log k8s API errors so
that each of them is unique. This will help to debug reanahub#383.
@mdonadoni mdonadoni mentioned this issue Feb 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/soon
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tiborsimko