Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow / Unresponsive Kubernetes API #4339

Closed
TekTimmy opened this issue Oct 21, 2020 · 3 comments · Fixed by #4340
Closed

Slow / Unresponsive Kubernetes API #4339

TekTimmy opened this issue Oct 21, 2020 · 3 comments · Fixed by #4340
Assignees
@alexec
Labels
type/bug
Milestone
v2.11

Comments

@TekTimmy
Copy link

TekTimmy commented Oct 21, 2020 •
edited

Summary

Not sure if this is Argo or AWS Kubernetes related... I would label it more as a "Problem" than a bug.

If we run ~300 Argo workflows (about 8 non parallel steps each) in parallel on our EKS Cluster which takes ~70 Nodes the Kubernetes API starts to get slow and in worst case we experience API timeouts.
The worst thing is that it also makes the workflows fail with different error messages:

failed to save outputs: the server was unable to return a response in the time allotted, but may still be processing the request (patch pods bx59k-552586920)

failed to save outputs: Patch https://10.100.0.1:443/api/v1/namespaces/v1-0/pods/5r49l-2608568875: net/http: TLS handshake timeout

failed to save outputs: Patch https://10.100.0.1:443/api/v1/namespaces/v1-0/pods/nx5b7-2156588813: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""

failed to save outputs: Patch https://10.100.0.1:443/api/v1/namespaces/v1-0/pods/9q96d-4078545465: stream error: stream ID 5; INTERNAL_ERROR

failed to save outputs: Patch https://10.100.0.1:443/api/v1/namespaces/v1-0/pods/ztglr-1531658454: dial tcp 10.100.0.1:443: i/o timeout

failed to save outputs: Patch https://10.100.0.1:443/api/v1/namespaces/v1-0/pods/wg9qh-2738839668: unexpected EOF

We have scaled up the workflow-controller with --workflow-workers 1024 --pod-workers 64 --qps 200 --burst 50.
We would expect Argo / Kubernetes to handle such workload without breaking.

Diagnostics

We are using AWS EKS 1.17 and Argo 2.11.5

time="2020-10-21T15:37:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-11-pair-X-v1-c9z2g
time="2020-10-21T15:37:22Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-B-v1-pair-1a2166-tn7-v1-fnhn4
time="2020-10-21T15:37:22Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-B-v1-pair-1a2166-tn7-v1-fnhn4
time="2020-10-21T15:37:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-C-v1-hq4sj
time="2020-10-21T15:37:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-annotation-B-v1-pair-1a2166-tn7-v1-fnhn4
time="2020-10-21T15:37:22Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-Z-v1-lane-hyltldsxx-003-lp67f
time="2020-10-21T15:37:22Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-Z-v1-lane-hyltldsxx-003-lp67f
time="2020-10-21T15:37:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-Z-v1-lane-hyltldsxx-003-lp67f
time="2020-10-21T15:37:22Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-D-v1-5gzg6
time="2020-10-21T15:37:22Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-D-v1-5gzg6
time="2020-10-21T15:37:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-D-v1-5gzg6
time="2020-10-21T15:37:23Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-E-v1-lane-hyllvdsxx-003-hzljh
time="2020-10-21T15:37:23Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-E-v1-lane-hyllvdsxx-003-hzljh
time="2020-10-21T15:37:23Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-E-v1-lane-hyllvdsxx-003-hzljh
time="2020-10-21T15:37:23Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-F-v1-lwgnt
time="2020-10-21T15:37:23Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-F-v1-lwgnt
time="2020-10-21T15:37:23Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-raw-data-F-v1-lwgnt
time="2020-10-21T15:37:23Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-9-v9r5v
time="2020-10-21T15:37:23Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-9-v9r5v
time="2020-10-21T15:37:23Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-9-v9r5v
time="2020-10-21T15:37:24Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-G-v1-nl5bk
time="2020-10-21T15:37:24Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-G-v1-nl5bk
time="2020-10-21T15:37:24Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-G-v1-nl5bk
time="2020-10-21T15:37:24Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-H-v1-48r8r
time="2020-10-21T15:37:24Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-H-v1-48r8r
time="2020-10-21T15:37:24Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-H-v1-48r8r
time="2020-10-21T15:37:24Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-I-v1-pair-Y-v1-88mb9
time="2020-10-21T15:37:24Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-I-v1-pair-Y-v1-88mb9
time="2020-10-21T15:37:24Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-annotation-I-v1-pair-Y-v1-88mb9
time="2020-10-21T15:37:24Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-4-pair-X-v1-9lvk6
time="2020-10-21T15:37:24Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-4-pair-X-v1-9lvk6
time="2020-10-21T15:37:24Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-4-pair-X-v1-9lvk6
time="2020-10-21T15:37:24Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-19-2lmbt
time="2020-10-21T15:37:24Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-19-2lmbt
time="2020-10-21T15:37:24Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-19-2lmbt
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-annotation-A-v1-chr-x-8slcb
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-annotation-A-v1-chr-x-8slcb
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-1a437-tp1-v1-lane-hyltldsxx-002-vp4ch
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-1a437-tp1-v1-lane-hyltldsxx-002-vp4ch
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-A-v1-chr-x-8slcb
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-18-4wgpk
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-18-4wgpk
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-1a437-tp1-v1-lane-hyltldsxx-002-vp4ch
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-1e273-tp1-v1-wzmtk
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-1e273-tp1-v1-wzmtk
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-18-4wgpk
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-1d806-tp1-v1-tw528
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-1d806-tp1-v1-tw528
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-1e273-tp1-v1-wzmtk
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-1d806-tp1-v1-tw528
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-varscan-A-v1-chr-10-pair-X-v1-w8lx2
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-varscan-A-v1-chr-10-pair-X-v1-w8lx2
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-Z-v1-lane-hyltldsxx-002-kn977
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-Z-v1-lane-hyltldsxx-002-kn977
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-4-rr5wx
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-4-rr5wx
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-qc-A-v1-47btt
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-qc-A-v1-47btt
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-qc-Y-v1-74pdr
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-qc-Y-v1-74pdr
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-varscan-A-v1-chr-10-pair-X-v1-w8lx2
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-Z-v1-pair-1c682-tn7-v1-rmvhj
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-Z-v1-pair-1c682-tn7-v1-rmvhj
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-raw-data-qc-A-v1-47btt
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-haplotype-caller-A-v1-chr-4-rr5wx
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-Z-v1-lane-hyltldsxx-002-kn977
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-raw-data-qc-Y-v1-74pdr
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-annotation-Z-v1-pair-1c682-tn7-v1-rmvhj
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-Y-v1-xrwc8
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-Y-v1-xrwc8
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-Y-v1-xrwc8
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-U-v1-lane-h2gfwdsxy-003-kdj99
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-U-v1-lane-h2gfwdsxy-003-kdj99
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-L-v1-pair-G-v1-gclvq
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-annotation-L-v1-pair-G-v1-gclvq
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-O-v1-lane-h2ckjdsxy-003-54wxz
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-alignment-O-v1-lane-h2ckjdsxy-003-54wxz
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-17-pair-X-v1-ptt9b
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-17-pair-X-v1-ptt9b
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-annotation-L-v1-pair-G-v1-gclvq
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-U-v1-lane-h2gfwdsxy-003-kdj99
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-gatk-mutect-A-v1-chr-17-pair-X-v1-ptt9b
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-O-v1-lane-h2ckjdsxy-003-54wxz
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-K-v1-dbxm7
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-K-v1-dbxm7
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-K-v1-dbxm7
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-X-v1-dqsxd
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-X-v1-dqsxd
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-X-v1-dqsxd
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-Y-v1-p2kzg
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-post-alignment-Y-v1-p2kzg
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-post-alignment-Y-v1-p2kzg
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-qc-M-v1-bllv4
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-raw-data-qc-M-v1-bllv4
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-raw-data-qc-M-v1-bllv4
time="2020-10-21T15:37:25Z" level=warning msg="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-varscan-A-v1-chr-7-pair-X-v1-ztdq7
time="2020-10-21T15:37:25Z" level=error msg="error in entry template execution" error="Deadline exceeded" namespace=v1-0 workflow=wgs-svc-varscan-A-v1-chr-7-pair-X-v1-ztdq7
time="2020-10-21T15:37:25Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-svc-varscan-A-v1-chr-7-pair-X-v1-ztdq7

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@alexec
Copy link
Contributor

alexec commented Oct 21, 2020

There is code in place to mitigate issues with updating the result of high load for the Kubernetes API (5 attempts over 500ms), so I believe you must be under extreme load.

I think this can be improved however (every 1s with exponential back-off over 10s say).

alexec added a commit to alexec/argo-workflows that referenced this issue Oct 21, 2020
@alexec
fix(executor): Increase backoff. Fixes argoproj#4339
a44ccd5
@alexec alexec mentioned this issue Oct 21, 2020
Merged
6 tasks
@alexec alexec self-assigned this Oct 21, 2020
@TekTimmy
Copy link
Author

Thanks a lot for that @alexec!
An argo list takes 4.5 seconds for me 😮 so those timeouts might even not be enough.
Yes there is a lot of load on every node but it should not influence the Kubernetes API, or does it?
Might the nodeStatusOffLoad: true help to reduce the load on the Kubernetes API server?

@TekTimmy
Copy link
Author

FYI EKS Cluster load:

NAME                                               CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-192-160-108-111.eu-central-1.compute.internal   16038m       100%   8814Mi          14%       
ip-192-160-151-227.eu-central-1.compute.internal   16006m       100%   8999Mi          14%       
ip-192-160-160-94.eu-central-1.compute.internal    16000m       100%   8563Mi          13%       
ip-192-160-70-63.eu-central-1.compute.internal     15998m       100%   8531Mi          13%       
ip-192-160-233-147.eu-central-1.compute.internal   15998m       100%   8253Mi          13%       
ip-192-160-83-151.eu-central-1.compute.internal    15895m       100%   8496Mi          13%       
ip-192-160-115-52.eu-central-1.compute.internal    15768m       99%    8438Mi          13%       
ip-192-160-122-147.eu-central-1.compute.internal   15760m       99%    8489Mi          13%       
ip-192-160-143-79.eu-central-1.compute.internal    15735m       99%    8712Mi          14%       
ip-192-160-247-16.eu-central-1.compute.internal    15701m       98%    8090Mi          13%       
ip-192-160-254-59.eu-central-1.compute.internal    15697m       98%    9060Mi          14%       
ip-192-160-191-114.eu-central-1.compute.internal   15549m       97%    8560Mi          13%       
ip-192-160-126-208.eu-central-1.compute.internal   15458m       97%    8852Mi          14%       
ip-192-160-100-45.eu-central-1.compute.internal    15376m       96%    8378Mi          13%       
ip-192-160-95-176.eu-central-1.compute.internal    15363m       96%    8613Mi          14%       
ip-192-160-159-236.eu-central-1.compute.internal   15100m       95%    8822Mi          14%       
ip-192-160-234-186.eu-central-1.compute.internal   15064m       94%    7700Mi          12%       
ip-192-160-187-216.eu-central-1.compute.internal   15054m       94%    8443Mi          13%       
ip-192-160-231-50.eu-central-1.compute.internal    15044m       94%    7732Mi          12%       
ip-192-160-139-201.eu-central-1.compute.internal   14968m       94%    9051Mi          14%       
ip-192-160-74-164.eu-central-1.compute.internal    14944m       94%    8677Mi          14%       
ip-192-160-153-158.eu-central-1.compute.internal   14906m       93%    8543Mi          13%       
ip-192-160-121-163.eu-central-1.compute.internal   14875m       93%    8721Mi          14%       
ip-192-160-243-53.eu-central-1.compute.internal    14861m       93%    8437Mi          13%       
ip-192-160-138-201.eu-central-1.compute.internal   14859m       93%    8480Mi          13%       
ip-192-160-99-211.eu-central-1.compute.internal    14847m       93%    8970Mi          14%       
ip-192-160-197-37.eu-central-1.compute.internal    14041m       88%    44900Mi         73%       
ip-192-160-118-113.eu-central-1.compute.internal   9855m        62%    32369Mi         52%       
ip-192-160-238-155.eu-central-1.compute.internal   9293m        58%    24423Mi         40%       
ip-192-160-187-27.eu-central-1.compute.internal    8324m        52%    24761Mi         40%       
ip-192-160-165-242.eu-central-1.compute.internal   8239m        51%    10529Mi         17%       
ip-192-160-134-148.eu-central-1.compute.internal   8220m        51%    22916Mi         37%       
ip-192-160-202-85.eu-central-1.compute.internal    8157m        51%    25154Mi         41%       
ip-192-160-83-52.eu-central-1.compute.internal     8005m        50%    7652Mi          12%       
ip-192-160-208-190.eu-central-1.compute.internal   7920m        49%    14572Mi         24%       
ip-192-160-236-237.eu-central-1.compute.internal   7862m        49%    4366Mi          7%        
ip-192-160-168-171.eu-central-1.compute.internal   7828m        49%    18522Mi         30%       
ip-192-160-163-248.eu-central-1.compute.internal   7826m        98%    1533Mi          4%        
ip-192-160-153-68.eu-central-1.compute.internal    7806m        49%    26194Mi         42%       
ip-192-160-122-29.eu-central-1.compute.internal    7264m        45%    27842Mi         45%       
ip-192-160-105-161.eu-central-1.compute.internal   7092m        44%    13808Mi         22%       
ip-192-160-102-200.eu-central-1.compute.internal   7011m        44%    30923Mi         50%       
ip-192-160-228-199.eu-central-1.compute.internal   6804m        42%    18768Mi         31%       
ip-192-160-80-22.eu-central-1.compute.internal     6769m        42%    21980Mi         35%       
ip-192-160-157-154.eu-central-1.compute.internal   6635m        41%    22254Mi         36%       
ip-192-160-85-65.eu-central-1.compute.internal     6605m        41%    23688Mi         38%       
ip-192-160-255-245.eu-central-1.compute.internal   6452m        40%    23672Mi         39%       
ip-192-160-83-142.eu-central-1.compute.internal    6320m        39%    24583Mi         39%       
ip-192-160-115-27.eu-central-1.compute.internal    6050m        38%    28229Mi         45%       
ip-192-160-166-162.eu-central-1.compute.internal   6011m        37%    27847Mi         45%       
ip-192-160-239-240.eu-central-1.compute.internal   5905m        37%    4023Mi          6%        
ip-192-160-86-78.eu-central-1.compute.internal     5897m        37%    25831Mi         42%       
ip-192-160-123-147.eu-central-1.compute.internal   5499m        34%    22836Mi         37%       
ip-192-160-95-42.eu-central-1.compute.internal     5382m        33%    29233Mi         47%       
ip-192-160-176-153.eu-central-1.compute.internal   4693m        29%    18352Mi         29%       
ip-192-160-242-57.eu-central-1.compute.internal    4511m        28%    22126Mi         36%       
ip-192-160-155-58.eu-central-1.compute.internal    4236m        26%    40483Mi         65%       
ip-192-160-100-120.eu-central-1.compute.internal   4214m        53%    435Mi           1%        
ip-192-160-154-142.eu-central-1.compute.internal   4194m        53%    428Mi           1%        
ip-192-160-160-54.eu-central-1.compute.internal    4111m        25%    30794Mi         50%       
ip-192-160-104-215.eu-central-1.compute.internal   4094m        51%    433Mi           1%        
ip-192-160-125-112.eu-central-1.compute.internal   4059m        51%    432Mi           1%        
ip-192-160-99-67.eu-central-1.compute.internal     4040m        51%    432Mi           1%        
ip-192-160-159-52.eu-central-1.compute.internal    3795m        23%    28421Mi         46%       
ip-192-160-138-135.eu-central-1.compute.internal   3726m        23%    27062Mi         44%       
ip-192-160-227-23.eu-central-1.compute.internal    3403m        21%    25298Mi         41%       
ip-192-160-249-245.eu-central-1.compute.internal   3321m        20%    20741Mi         34%       
ip-192-160-98-147.eu-central-1.compute.internal    3129m        19%    26855Mi         43%       
ip-192-160-177-201.eu-central-1.compute.internal   3109m        39%    7194Mi          23%       
ip-192-160-156-22.eu-central-1.compute.internal    2995m        18%    38119Mi         62%       
ip-192-160-169-174.eu-central-1.compute.internal   2863m        36%    10635Mi         34%       
ip-192-160-190-7.eu-central-1.compute.internal     2776m        17%    35956Mi         58%       
ip-192-160-147-188.eu-central-1.compute.internal   2720m        69%    3884Mi          25%       
ip-192-160-190-162.eu-central-1.compute.internal   2568m        16%    42094Mi         68%       
ip-192-160-170-159.eu-central-1.compute.internal   2494m        31%    7207Mi          23%       
ip-192-160-86-23.eu-central-1.compute.internal     2155m        27%    10081Mi         32%       
ip-192-160-66-108.eu-central-1.compute.internal    2125m        26%    10246Mi         32%       
ip-192-160-209-238.eu-central-1.compute.internal   2112m        13%    38163Mi         63%       
ip-192-160-99-246.eu-central-1.compute.internal    2101m        26%    9164Mi          29%       
ip-192-160-97-66.eu-central-1.compute.internal     1870m        47%    3372Mi          22%       
ip-192-160-194-83.eu-central-1.compute.internal    1858m        11%    45576Mi         75%       
ip-192-160-109-112.eu-central-1.compute.internal   1588m        20%    19161Mi         61%       
ip-192-160-181-39.eu-central-1.compute.internal    1464m        37%    426Mi           2%        
ip-192-160-179-239.eu-central-1.compute.internal   1403m        17%    15801Mi         50%       
ip-192-160-87-23.eu-central-1.compute.internal     1396m        17%    10656Mi         34%       
ip-192-160-77-172.eu-central-1.compute.internal    1387m        17%    11195Mi         35%       
ip-192-160-86-14.eu-central-1.compute.internal     1378m        17%    10531Mi         33%       
ip-192-160-84-112.eu-central-1.compute.internal    1360m        17%    10578Mi         33%       
ip-192-160-163-45.eu-central-1.compute.internal    1354m        17%    8885Mi          28%       
ip-192-160-247-2.eu-central-1.compute.internal     1331m        8%     422Mi           0%        
ip-192-160-118-67.eu-central-1.compute.internal    1291m        16%    7487Mi          24%       
ip-192-160-85-135.eu-central-1.compute.internal    1273m        16%    7227Mi          23%       
ip-192-160-83-59.eu-central-1.compute.internal     1268m        16%    10747Mi         34%       
ip-192-160-69-107.eu-central-1.compute.internal    1264m        15%    7459Mi          23%       
ip-192-160-152-70.eu-central-1.compute.internal    1262m        15%    16540Mi         53%       
ip-192-160-90-28.eu-central-1.compute.internal     1231m        15%    10127Mi         32%       
ip-192-160-154-236.eu-central-1.compute.internal   1000m        12%    551Mi           1%        
ip-192-160-116-177.eu-central-1.compute.internal   61m          0%     402Mi           1%

@alexec alexec added this to the v2.11 milestone Oct 22, 2020
alexec added a commit that referenced this issue Oct 23, 2020
@alexec
fix(executor): Increase pod patch backoff. Fixes #4339 (#4340)
b76246e
alexec added a commit that referenced this issue Oct 23, 2020
@alexec
fix(executor): Increase pod patch backoff. Fixes #4339 (#4340)
a4186df
alexcapras pushed a commit to alexcapras/argo that referenced this issue Nov 12, 2020
@alexec @alexcapras
fix(executor): Increase pod patch backoff. Fixes argoproj#4339 (argop…
5ca685c 
…roj#4340)

Signed-off-by: Alex Capras <alexcapras@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexec alexec

Labels
type/bug
Projects
None yet
Milestone
v2.11
Development

Successfully merging a pull request may close this issue.

fix(executor): Increase pod patch backoff. Fixes #4339 alexec/argo-workflows
2 participants
@alexec @TekTimmy