cks: use HttpsURLConnection for checking api server #4639

shwstppr · 2021-02-02T09:25:07Z

Description

Frobable fix for #4146 , #4637

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2021-02-02T09:25:29Z

@blueorangutan package

blueorangutan · 2021-02-02T09:26:31Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-02T09:52:57Z

Packaging result: ✔centos7 ✖centos8 ✔debian. JID-2629

rohityadavcloud · 2021-02-02T10:36:51Z

@blueorangutan test

blueorangutan · 2021-02-02T10:37:36Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

rohityadavcloud · 2021-02-02T10:40:00Z

...bernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java

@@ -218,7 +221,10 @@ public static boolean isKubernetesClusterServerRunning(final KubernetesCluster k
        boolean k8sApiServerSetup = false;
        while (System.currentTimeMillis() < timeoutTime) {
            try {
-                String versionOutput = IOUtils.toString(new URL(String.format("https://%s:%d/version", ipAddress, port)), StringUtils.getPreferredCharset());
+                URL url = new URL(String.format("https://%s:%d/version", ipAddress, port));
+                HttpsURLConnection con = (HttpsURLConnection)url.openConnection();


should we configure it to ignore ssl cert validation?

and get more static analysis reports as issues?

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2021-02-02T12:15:25Z

@blueorangutan package

blueorangutan · 2021-02-02T12:16:32Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-02T12:45:02Z

Packaging result: ✔centos7 ✖centos8 ✔debian. JID-2637

nxsbi · 2021-02-02T18:45:43Z

Packaging result: ✔centos7 ✖centos8 ✔debian. JID-2637

If I want to test this, how do I do so? You will have to give instructions to obtain/install the packages in my environment.

blueorangutan · 2021-02-02T21:43:59Z

Trillian test result (tid-3471)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38279 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4639-t3471-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_07_deploy_kubernetes_ha_cluster	`Failure`	3612.86	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	112.05	test_kubernetes_clusters.py

shwstppr · 2021-02-03T06:49:11Z

Packaging result: heavy_check_markcentos7 heavy_multiplication_xcentos8 heavy_check_markdebian. JID-2637

If I want to test this, how do I do so? You will have to give instructions to obtain/install the packages in my environment.

@nxsbi do you want to test the fix in 4.15 or 4.14 or it doesn't matter?

nxsbi · 2021-02-03T06:52:41Z

Packaging result: heavy_check_markcentos7 heavy_multiplication_xcentos8 heavy_check_markdebian. JID-2637

If I want to test this, how do I do so? You will have to give instructions to obtain/install the packages in my environment.

@nxsbi do you want to test the fix in 4.15 or 4.14 or it doesn't matter?

@shwstppr I want to test in 4.15.

shwstppr · 2021-02-03T09:07:59Z

okay @nxsbi, I'll share the 4.15 packages with the fix soon

shwstppr · 2021-02-03T10:25:29Z

@nxsbi please find build packages here, http://download.cloudstack.org/testing/is-4637/
They are built against current 4.15 branch.

blueorangutan · 2021-02-03T21:14:10Z

Trillian test result (tid-3482)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40661 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4639-t3482-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_kubernetes_cluster	`Failure`	3605.90	test_kubernetes_clusters.py
test_02_invalid_upgrade_kubernetes_cluster	`Failure`	3622.01	test_kubernetes_clusters.py
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	0.08	test_kubernetes_clusters.py
test_04_deploy_and_scale_kubernetes_cluster	`Failure`	0.08	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.08	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.07	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.07	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.07	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	107.84	test_kubernetes_clusters.py

nxsbi · 2021-02-04T04:13:20Z

Tried to do a quick deployment with the new packages --
It failed again... Below are relevant lines from Management Server log.
Are there any additional log files I can look into or additional checks I can do? If yes, please advise..

2021-02-03 17:44:42,964 INFO [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-1:ctx-0078ab8c job-12195 ctx-65cc2bf7) (logid:4ac48e69) Waiting for Kubernetes cluster : k8test master node VMs to be accessible 2021-02-03 17:44:52,965 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-0078ab8c job-12195 ctx-65cc2bf7) (logid:4ac48e69) Failed to setup Kubernetes cluster : k8test3 in usable state as unable to access master node VMs of the cluster 2021-02-03 17:44:59,052 INFO [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-0078ab8c job-12195 ctx-65cc2bf7) (logid:4ac48e69) Detached Kubernetes binaries from VM : k8test3-master in the Kubernetes cluster : k8test3 2021-02-03 17:45:02,995 INFO [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-0078ab8c job-12195 ctx-65cc2bf7) (logid:4ac48e69) Detached Kubernetes binaries from VM : k8test3-node-1 in the Kubernetes cluster : k8test3 2021-02-03 17:45:02,998 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-0078ab8c job-12195) (logid:4ac48e69) Complete async job-12195, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to setup Kubernetes cluster : k8test3 in usable state as unable to access master node VMs of the cluster"}

NOTE -- I can login into the Master node via command line with SSH Key, and I can see that the cluster is actually functional, however it does nto report so in the UI (I ssh using the user Core, and then sudo su to root). IF I run the commands as core, I get access denied, as all files are owned by root with user/group as no access.

nxsbi · 2021-02-04T04:45:21Z

@shwstppr - Tested inside Master node, with curl just to see if the certificate issue persists -- Yes it does
Note this is with the new package build you provided

shwstppr · 2021-02-04T07:10:42Z

@nxsbi can you please share output of,
cat /var/log/cloudstack/management/management-server.log | grep -i 4ac48e69

nxsbi · 2021-02-04T08:11:53Z

@shwstppr

cat /var/log/cloudstack/management/management-server.log | grep -i 4ac48e69

Here it is. I have cleansed the log for IP, passwords, Keys etc. so you may see XXXXX.. For IP I only changed the management LAN IPs and external IPs. You will see them as 192.168.100.XXX where XXX is the actual last bit of correct IP or as 172.100.100.100
mgmt-4ac48e69cleansed.log

nxsbi · 2021-02-04T16:23:40Z

@shwstppr
Management server is able to curl -k the public IP of the VR. But it throws error if using just curl (without -k)

shwstppr · 2021-02-04T16:55:05Z

@nxsbi service is not being able to connect to SSH with VR public IP and forwarded port (must be 2222), failing here

cloudstack/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java

Lines 247 to 253 in 280c13a

    
           try (Socket socket = new Socket()) { 
        
               socket.connect(new InetSocketAddress(ipAddress, port), 10000); 
        
               masterVmRunning = true; 
        
           } catch (IOException e) { 
        
               if (LOGGER.isInfoEnabled()) { 
        
                   LOGGER.info(String.format("Waiting for Kubernetes cluster : %s master node VMs to be accessible", kubernetesCluster.getName())); 
        
               }

Are you able to manually do SSH from the management server?
ssh -i /root/.ssh/id_rsa.cloud -p 2222 core@public_ip_of_network

Firewall and port forwarding rules must be automatically provisioned by the service in the cluster's network.
Service uses SSH over 2222 to 2222+n ports. SSH on worker nodes are done only during k8s version upgrade

blueorangutan · 2021-02-22T10:00:03Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-23T08:19:35Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2803

rohityadavcloud · 2021-02-23T10:58:04Z

LGTM

rohityadavcloud · 2021-02-24T10:58:30Z

@shwstppr is this good to go?

DaanHoogland

clgtm

shwstppr · 2021-02-25T09:31:04Z

@

shwstppr · 2021-02-25T09:32:18Z

@weizhouapache possible for you to test this and see if this fixes curl -L issue you highlighted in #4146?

shwstppr · 2021-02-26T07:03:55Z

@blueorangutan package

blueorangutan · 2021-02-26T07:05:15Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-26T08:16:45Z

Packaging result: ✖centos7 ✖centos8 ✔debian. JID-2833

rohityadavcloud · 2021-02-26T08:52:38Z

@blueorangutan package

blueorangutan · 2021-02-26T08:53:15Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-26T09:45:04Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2839

rohityadavcloud · 2021-02-26T13:17:07Z

@blueorangutan test

blueorangutan · 2021-02-26T13:18:16Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-02-26T23:51:45Z

Trillian test result (tid-3625)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36155 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4639-t3625-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_affinity_groups.py
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_nic	`Error`	49.50	test_nic.py
test_01_migrate_VM_and_root_volume	`Error`	60.96	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	50.01	test_vm_life_cycle.py

cks: use HttpsURLConnection for checking api server

84d68a8

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr mentioned this pull request Feb 2, 2021

Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network #4637

Closed

rohityadavcloud approved these changes Feb 2, 2021

View reviewed changes

rohityadavcloud added this to the 4.14.1.0 milestone Feb 2, 2021

rohityadavcloud reviewed Feb 2, 2021

View reviewed changes

ignore ssl certificate validation

a31487c

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr linked an issue Feb 3, 2021 that may be closed by this pull request

Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network #4637

Closed

rohityadavcloud modified the milestones: 4.14.1.0, 4.15.1.0 Feb 4, 2021

shwstppr marked this pull request as ready for review February 22, 2021 10:58

shwstppr closed this Feb 22, 2021

shwstppr reopened this Feb 22, 2021

shwstppr closed this Feb 23, 2021

shwstppr reopened this Feb 23, 2021

apache deleted a comment from blueorangutan Feb 23, 2021

rohityadavcloud requested a review from DaanHoogland February 23, 2021 10:57

DaanHoogland approved these changes Feb 24, 2021

View reviewed changes

shwstppr closed this Feb 25, 2021

shwstppr reopened this Feb 25, 2021

weizhouapache approved these changes Feb 26, 2021

View reviewed changes

rohityadavcloud closed this Feb 26, 2021

rohityadavcloud reopened this Feb 26, 2021

rohityadavcloud merged commit f8ad3ad into apache:4.15 Mar 2, 2021

rohityadavcloud mentioned this pull request Mar 2, 2021

Kubernetes Service: Docs do not describe Kubernetes service network requirements #4146

Closed

cks: use HttpsURLConnection for checking api server #4639

cks: use HttpsURLConnection for checking api server #4639

Conversation

shwstppr commented Feb 2, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

shwstppr commented Feb 2, 2021

blueorangutan commented Feb 2, 2021

blueorangutan commented Feb 2, 2021

rohityadavcloud commented Feb 2, 2021

blueorangutan commented Feb 2, 2021

rohityadavcloud Feb 2, 2021

Choose a reason for hiding this comment

DaanHoogland Feb 23, 2021

Choose a reason for hiding this comment

shwstppr commented Feb 2, 2021

blueorangutan commented Feb 2, 2021

blueorangutan commented Feb 2, 2021

nxsbi commented Feb 2, 2021

blueorangutan commented Feb 2, 2021

shwstppr commented Feb 3, 2021

nxsbi commented Feb 3, 2021

shwstppr commented Feb 3, 2021

shwstppr commented Feb 3, 2021

blueorangutan commented Feb 3, 2021

nxsbi commented Feb 4, 2021 • edited

nxsbi commented Feb 4, 2021

shwstppr commented Feb 4, 2021

nxsbi commented Feb 4, 2021

nxsbi commented Feb 4, 2021

shwstppr commented Feb 4, 2021

blueorangutan commented Feb 22, 2021

blueorangutan commented Feb 23, 2021

rohityadavcloud commented Feb 23, 2021

rohityadavcloud commented Feb 24, 2021

DaanHoogland left a comment

Choose a reason for hiding this comment

shwstppr commented Feb 25, 2021

shwstppr commented Feb 25, 2021 • edited

shwstppr commented Feb 26, 2021

blueorangutan commented Feb 26, 2021

blueorangutan commented Feb 26, 2021

rohityadavcloud commented Feb 26, 2021

blueorangutan commented Feb 26, 2021

blueorangutan commented Feb 26, 2021

rohityadavcloud commented Feb 26, 2021

blueorangutan commented Feb 26, 2021

blueorangutan commented Feb 26, 2021

nxsbi commented Feb 4, 2021 •

edited

shwstppr commented Feb 25, 2021 •

edited