Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OneClickUpdate Script Stuck #1

Closed
grv18 opened this issue Aug 29, 2022 · 1 comment
Closed

OneClickUpdate Script Stuck #1

grv18 opened this issue Aug 29, 2022 · 1 comment

Comments

@grv18
Copy link

grv18 commented Aug 29, 2022

I tried to update my EKS cluster using the given script. Below are the steps that I did on my Cloud9 Instance.


git clone https://github.com/aws-samples/amazon-eks-one-click-cluster-upgrade.git
cd amazon-eks-one-click-cluster-upgrade/
python installer.py 
python eks_updater.py <name-of-cluster> <Updated Version> <region>

I can see that the Control Plane is updated to 1.21 version. However, the eks_updater script is still running. I initiated the process some 2 hours ago and the Control Plane is running now the specified version.

I could see that the script is stuck at the update AddOn stage. Below is the last message on the CW logs

The Addons Found = ['aws-load-balancer-controller-6db9694d6b-58gcd', 'aws-load-balancer-controller-6db9694d6b-pmpsb', 'aws-node-5t9x9', 'aws-node-rlkdt', 'aws-node-vr4df', 'coredns-765545c8b8-7zhsn', 'coredns-765545c8b8-jlr7w', 'kube-proxy-b9gtd', 'kube-proxy-kbjvf', 'kube-proxy-x4gzt', 'metrics-server-9f459d97b-wqrcn']

Here is what i see on my Command line from where i ran the Python Updater script.

The Cluster eksworkshop-eksctl is Still Updating to 1.21 ..... 00:11:54.12
The eksworkshop-eksctl Updated to 1.21
The Time Taken For the Cluster to Upgrade  00:12:20.41
 The add-ons Update has been initiated.... 
The Addons Found =  aws-load-balancer-controller-6db9694d6b-58gcd aws-load-balancer-controller-6db9694d6b-pmpsb aws-node-5t9x9 aws-node-rlkdt aws-node-vr4df coredns-765545c8b8-7zhsn coredns-765545c8b8-jlr7w kube-proxy-b9gtd kube-proxy-kbjvf kube-proxy-x4gzt metrics-server-9f459d97b-wqrcn
aws-node-5t9x9 Current Version =  v1.7.5-eksbuild.1 Updating To =  v1.9.3
Total Pods With aws-node = 3
old vpc cni Pod aws-node-5t9x9   new vpc cni aws-node-xq957
aws-node-rlkdt Current Version =  v1.7.5-eksbuild.1 Updating To =  v1.9.3
Total Pods With aws-node = 3
old vpc cni Pod aws-node-rlkdt   new vpc cni aws-node-xq957
aws-node-vr4df Current Version =  v1.7.5-eksbuild.1 Updating To =  v1.9.3
Total Pods With aws-node = 3
old vpc cni Pod aws-node-vr4df   new vpc cni aws-node-5v8dr
coredns-765545c8b8-7zhsn Current Version =  v1.8.3-eksbuild.1 Updating To =  v1.8.4-eksbuild.1
Total Pods With kube-dns = 3
old CoreDNs Pod coredns-765545c8b8-7zhsn         new CoreDnsPod coredns-59d47d99dc-xxvs5
coredns-765545c8b8-jlr7w Current Version =  v1.8.3-eksbuild.1 Updating To =  v1.8.4-eksbuild.1
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/amazon-eks-one-click-cluster-upgrade/eksupdate/src/k8s_client.py", line 26, in run
    x=addon_status(cluster_name=cluster_name,new_pod_name=new_pod_name,podName=podName,regionName=regionName,nameSpace=nameSpace)
  File "/home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/amazon-eks-one-click-cluster-upgrade/eksupdate/src/k8s_client.py", line 219, in addon_status
    if response.status.container_statuses[0].ready and response.status.container_statuses[0].started:
TypeError: 'NoneType' object is not subscriptable

Total Pods With kube-dns = 3
old CoreDNs Pod coredns-765545c8b8-jlr7w         new CoreDnsPod coredns-59d47d99dc-xxvs5
kube-proxy-b9gtd Current Version =  v1.20.7-eksbuild.1 Updating To =  v1.21.2-eksbuild.2
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/amazon-eks-one-click-cluster-upgrade/eksupdate/src/k8s_client.py", line 26, in run
    x=addon_status(cluster_name=cluster_name,new_pod_name=new_pod_name,podName=podName,regionName=regionName,nameSpace=nameSpace)
  File "/home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/amazon-eks-one-click-cluster-upgrade/eksupdate/src/k8s_client.py", line 219, in addon_status
    if response.status.container_statuses[0].ready and response.status.container_statuses[0].started:
TypeError: 'NoneType' object is not subscriptable

Total Pods With kube-proxy = 3
old KubProxy Pod kube-proxy-b9gtd        new KubeProxyPod kube-proxy-jh955
kube-proxy-kbjvf Current Version =  v1.20.7-eksbuild.1 Updating To =  v1.21.2-eksbuild.2
Total Pods With kube-proxy = 3
old KubProxy Pod kube-proxy-kbjvf        new KubeProxyPod kube-proxy-tkd98
kube-proxy-x4gzt Current Version =  v1.20.7-eksbuild.1 Updating To =  v1.21.2-eksbuild.2
Total Pods With kube-proxy = 3
old KubProxy Pod kube-proxy-x4gzt        new KubeProxyPod kube-proxy-tkd98

I took the strace and lsof O/p of this process and seems like it is waiting on

Admin:environment $ ps -eFL | grep python
ec2-user  3605 25204  3605  0    1 29855   920   1 04:19 pts/4    00:00:00 grep --color=auto python
root     17476 17472 17476  0    1 91216 29912   1 03:37 ?        00:00:00 /usr/bin/python -tt /usr/sbin/yum-cron
ec2-user 22850 26543 22850  0   19 385452 136508 0 02:37 pts/3    00:00:05 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28775  0   19 385452 136508 1 02:50 pts/3    00:00:09 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28776  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28777  0   19 385452 136508 1 02:50 pts/3    00:00:19 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28780  0   19 385452 136508 0 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28781  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28782  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28783  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28784  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28785  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28786  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28787  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28788  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28789  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28790  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28791  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28792  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28793  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
ec2-user 22850 26543 28794  0   19 385452 136508 1 02:50 pts/3    00:00:00 python3 eks_updater.py eksworkshop-eksctl 1.21 us-west-2
Admin:environment $ 

Admin:environment $ lsof -p 26543
COMMAND   PID     USER   FD   TYPE DEVICE  SIZE/OFF     NODE NAME
bash    26543 ec2-user  cwd    DIR  259,1       190   913808 /home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/a azon-eks-one-click-cluster-upgrade
bash    26543 ec2-user  rtd    DIR  259,1       257       96 /
bash    26543 ec2-user  txt    REG  259,1    935976  4195692 /usr/bin/bash
bash    26543 ec2-user  mem    REG  259,1     71160  8410842 /usr/lib64/libnss_files-2.26.so
bash    26543 ec2-user  mem    REG  259,1     37032  8789607 /usr/lib64/libnss_sss.so.2
bash    26543 ec2-user  mem    REG  259,1 113049440 13072173 /usr/lib/locale/locale-archive
bash    26543 ec2-user  mem    REG  259,1   2021864  8410826 /usr/lib64/libc-2.26.so
bash    26543 ec2-user  mem    REG  259,1     19208  8410830 /usr/lib64/libdl-2.26.so
bash    26543 ec2-user  mem    REG  259,1    179264  8410886 /usr/lib64/libtinfo.so.6.0
bash    26543 ec2-user  mem    REG  259,1    174280  8410819 /usr/lib64/ld-2.26.so
bash    26543 ec2-user  mem    REG  259,1     26370     1393 /usr/lib64/gconv/gconv-modules.cache
bash    26543 ec2-user    0u   CHR  136,3       0t0        6 /dev/pts/3
bash    26543 ec2-user    1u   CHR  136,3       0t0        6 /dev/pts/3
bash    26543 ec2-user    2u   CHR  136,3       0t0        6 /dev/pts/3
bash    26543 ec2-user    6u   CHR  136,3       0t0        6 /dev/pts/3
bash    26543 ec2-user  255u   CHR  136,3       0t0        6 /dev/pts/3
Admin:environment $ 
Admin:environment $ lsof -p 22850
COMMAND   PID     USER   FD   TYPE   DEVICE  SIZE/OFF     NODE NAME
python3 22850 ec2-user  cwd    DIR    259,1       190   913808 /home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/amazon-eks-one-click-cluster-upgrade
python3 22850 ec2-user  rtd    DIR    259,1       257       96 /
python3 22850 ec2-user  txt    REG    259,1      7048  4249794 /usr/bin/python3.7
python3 22850 ec2-user  mem    REG    259,1     88640  8410801 /usr/lib64/libgcc_s-7-20180712.so.1
python3 22850 ec2-user  mem    REG    259,1     17008 12709186 /usr/lib64/python3.7/lib-dynload/_multiprocessing.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     26976  8410840 /usr/lib64/libnss_dns-2.26.so
python3 22850 ec2-user  mem    REG    259,1     71160  8410842 /usr/lib64/libnss_files-2.26.so
python3 22850 ec2-user  mem    REG    259,1   1832344  8711818 /home/ec2-user/.local/lib/python3.7/site-packages/yaml/_yaml.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     43504    95974 /usr/lib64/python3.7/site-packages/simplejson/_speedups.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     49984 12709185 /usr/lib64/python3.7/lib-dynload/_multibytecodec.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1   1072304 12709220 /usr/lib64/python3.7/lib-dynload/unicodedata.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     66640 12709202 /usr/lib64/python3.7/lib-dynload/array.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    128248 12709188 /usr/lib64/python3.7/lib-dynload/_pickle.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     25944 12709219 /usr/lib64/python3.7/lib-dynload/termios.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     20056  8411057 /usr/lib64/libuuid.so.1.3.0
python3 22850 ec2-user  mem    REG    259,1      7216 12709201 /usr/lib64/python3.7/lib-dynload/_uuid.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    320048 12668088 /usr/lib64/python3.7/lib-dynload/_decimal.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     13208 12709207 /usr/lib64/python3.7/lib-dynload/grp.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    153248  8411025 /usr/lib64/liblzma.so.5.2.2
python3 22850 ec2-user  mem    REG    259,1     39024 12668095 /usr/lib64/python3.7/lib-dynload/_lzma.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     68128  8411050 /usr/lib64/libbz2.so.1.0.6
python3 22850 ec2-user  mem    REG    259,1     23080 12668073 /usr/lib64/python3.7/lib-dynload/_bz2.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     63360 12709213 /usr/lib64/python3.7/lib-dynload/pyexpat.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    197960  8461231 /usr/lib64/libexpat.so.1.6.0
python3 22850 ec2-user  mem    REG    259,1     72512 12668089 /usr/lib64/python3.7/lib-dynload/_elementtree.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     38712 12709222 /usr/lib64/python3.7/lib-dynload/zlib.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     18016 12709190 /usr/lib64/python3.7/lib-dynload/_queue.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    131696 12709198 /usr/lib64/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     30992 12709204 /usr/lib64/python3.7/lib-dynload/binascii.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    120040 12668086 /usr/lib64/python3.7/lib-dynload/_datetime.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1      7304 12709187 /usr/lib64/python3.7/lib-dynload/_opcode.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     76176 12668093 /usr/lib64/python3.7/lib-dynload/_json.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     20552 12709191 /usr/lib64/python3.7/lib-dynload/_random.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     13432 12668071 /usr/lib64/python3.7/lib-dynload/_bisect.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     98352 12709194 /usr/lib64/python3.7/lib-dynload/_sha3.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     47504 12668072 /usr/lib64/python3.7/lib-dynload/_blake2.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    410400  8410985 /usr/lib64/libpcre.so.1.2.0
python3 22850 ec2-user  mem    REG    259,1    155680  8410984 /usr/lib64/libselinux.so.1
python3 22850 ec2-user  mem    REG    259,1     94200  8410846 /usr/lib64/libresolv-2.26.so
python3 22850 ec2-user  mem    REG    259,1     15616  8461240 /usr/lib64/libkeyutils.so.1.5
python3 22850 ec2-user  mem    REG    259,1     62880  8509861 /usr/lib64/libkrb5support.so.0.1
python3 22850 ec2-user  mem    REG    259,1     85984  8410997 /usr/lib64/libz.so.1.2.7
python3 22850 ec2-user  mem    REG    259,1    202472  8509853 /usr/lib64/libk5crypto.so.3.1
python3 22850 ec2-user  mem    REG    259,1     15768  8411005 /usr/lib64/libcom_err.so.2.1
python3 22850 ec2-user  mem    REG    259,1    947152  8509859 /usr/lib64/libkrb5.so.3.3
python3 22850 ec2-user  mem    REG    259,1    315672  8509849 /usr/lib64/libgssapi_krb5.so.2.2
python3 22850 ec2-user  mem    REG    259,1   2467296  8463015 /usr/lib64/libcrypto.so.1.0.2k
python3 22850 ec2-user  mem    REG    259,1    457928  8463017 /usr/lib64/libssl.so.1.0.2k
python3 22850 ec2-user  mem    REG    259,1     30936 12668091 /usr/lib64/python3.7/lib-dynload/_hashlib.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     53160 12709199 /usr/lib64/python3.7/lib-dynload/_struct.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1    132176 12709196 /usr/lib64/python3.7/lib-dynload/_socket.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     52536 12709208 /usr/lib64/python3.7/lib-dynload/math.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     37600 12709216 /usr/lib64/python3.7/lib-dynload/select.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     16680 12709189 /usr/lib64/python3.7/lib-dynload/_posixsubprocess.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1     22896 12668092 /usr/lib64/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
python3 22850 ec2-user  mem    REG    259,1 113049440 13072173 /usr/lib/locale/locale-archive
python3 22850 ec2-user  mem    REG    259,1   2021864  8410826 /usr/lib64/libc-2.26.so
python3 22850 ec2-user  mem    REG    259,1   1414728  8410832 /usr/lib64/libm-2.26.so
python3 22850 ec2-user  mem    REG    259,1     14304  8410852 /usr/lib64/libutil-2.26.so
python3 22850 ec2-user  mem    REG    259,1     19208  8410830 /usr/lib64/libdl-2.26.so
python3 22850 ec2-user  mem    REG    259,1    149416  8410844 /usr/lib64/libpthread-2.26.so
python3 22850 ec2-user  mem    REG    259,1     41032  8410998 /usr/lib64/libcrypt-2.26.so
python3 22850 ec2-user  mem    REG    259,1   3551352  8510815 /usr/lib64/libpython3.7m.so.1.0
python3 22850 ec2-user  mem    REG    259,1    174280  8410819 /usr/lib64/ld-2.26.so
python3 22850 ec2-user  mem    REG    259,1     26370     1393 /usr/lib64/gconv/gconv-modules.cache
python3 22850 ec2-user    0u   CHR    136,3       0t0        6 /dev/pts/3
python3 22850 ec2-user    1u   CHR    136,3       0t0        6 /dev/pts/3
python3 22850 ec2-user    2u   CHR    136,3       0t0        6 /dev/pts/3
python3 22850 ec2-user    3u  IPv4 30677781       0t0      TCP ip-192-168-78-154.us-west-2.compute.internal:38320->ip-192-168-137-239.us-west-2.compute.internal:https (CLOSE_WAIT)
python3 22850 ec2-user    4r   REG    259,1      5754 13110333 /home/ec2-user/environment/K8s-Manifest/EKS-Playground/UpdateCluster/amazon-eks-one-click-cluster-upgrade/eksupdate/src/S3Files/vpc-cni.yaml
python3 22850 ec2-user    5u  IPv4 30677789       0t0      TCP ip-192-168-78-154.us-west-2.compute.internal:54562->ip-192-168-129-144.us-west-2.compute.internal:https (CLOSE_WAIT)
python3 22850 ec2-user    6u   CHR    136,3       0t0        6 /dev/pts/3




Admin:environment $ strace -p 22850
strace: Process 22850 attached
futex(0x1878590, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY


^Cstrace: Process 22850 detached
 <detached ...>

Admin:environment $ strace -p 26543                                                                                                
strace: Process 26543 attached
wait4(-1, 


^Cstrace: Process 26543 detached
 <detached ...>

Why is the update script stuck?
@nayanen nayanen self-assigned this Sep 13, 2022
@mbeacom
Copy link
Contributor

mbeacom commented Jan 31, 2023

@grv18 We just released 0.1.0 of this module to pypi. Please feel free to re-open this if the issue persists.

@mbeacom mbeacom closed this as completed Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mbeacom @nayanen @grv18