Failed VM customization on cse template install

[sachinphogat]

Describe the bug

While executing cse template install command, its failing with "Failed VM Customization error"

Required Python version: >= 3.7.3
Installed Python version: 3.7.11 (default, Aug 29 2022, 18:09:28)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Validating config file 'config.yaml'
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised.
Connected to vCloud Director (100.121.43.146:443)
Connected to vCenter Server 'PRODUCTION-VC' as 'catalystadmin@vsphere.local' (chnvprodvc01.chnvsblab.com)
Config file 'config.yaml' is valid
Installing template 'ubuntu-20.04_k8-1.23_weave-2.8.1' at revision '*' on vCloud Director using config file 'config.yaml'
Connected to vCD as system administrator: 100.121.43.146:443
Downloaded remote template cookbook from https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh' (file already exists)
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh' (file already exists)
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/nfsd.sh' (file already exists)
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/docker-upgrade.sh' (file already exists)
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-cni-apply.sh' (file already exists)
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-k8s-upgrade.sh' (file already exists)
Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/worker-k8s-upgrade.sh' (file already exists)
Found ova file 'ubuntu-20.04-server-cloudimg-amd64.ova' in catalog 'cse'
Deleting temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
Deleted temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
Creating vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh
Created vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
Customizing vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp', vm 'ubuntu-2004-k8s1233-weave281-vm'
Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh
Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning
Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning
Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning
Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsRunning
Failed VM customization. Check CSE install log
Template Installation Error. Check CSE install logs
Error while uploading file: 400

cse-install.log
22-08-30 15:30:30 | remote_template_manager:269 - get_unfiltered_remote_template_cookbook | DEBUG :: Downloaded remote template cookbook from https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml
22-08-30 15:30:30 | remote_template_manager:279 - get_unfiltered_remote_template_cookbook | DEBUG :: Template cookbook version: 2.0.0
22-08-30 15:30:30 | remote_template_manager:194 - _validate_remote_template_cookbook | DEBUG :: Template cookbook https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml is valid
22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-20.04_k8-1.23_weave-2.8.1 revision 1 is supported by CSE 3.1.4
22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.21_weave-2.8.1 revision 1 is supported by CSE 3.1.4
22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.20_weave-2.6.5 revision 3 is supported by CSE 3.1.4
22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.19_weave-2.6.5 revision 3 is supported by CSE 3.1.4
22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.18_weave-2.6.5 revision 3 is supported by CSE 3.1.4
22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template photon-v2_k8-1.14_weave-2.5.2 revision 5 is supported by CSE 3.1.4
22-08-30 15:30:31 | remote_template_manager:159 - _filter_unsupported_templates | DEBUG :: Successfully filtered unsupported templates.
22-08-30 15:30:32 | configure_cse:490 - install_template | INFO :: Installing template 'ubuntu-20.04_k8-1.23_weave-2.8.1' at revision '*' on vCloud Director using config file 'config.yaml'
22-08-30 15:30:32 | configure_cse:544 - install_template | INFO :: Connected to vCD as system administrator: 100.121.43.146:443
22-08-30 15:30:32 | remote_template_manager:269 - get_unfiltered_remote_template_cookbook | DEBUG :: Downloaded remote template cookbook from https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml
22-08-30 15:30:32 | remote_template_manager:279 - get_unfiltered_remote_template_cookbook | DEBUG :: Template cookbook version: 2.0.0
22-08-30 15:30:32 | remote_template_manager:194 - _validate_remote_template_cookbook | DEBUG :: Template cookbook https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml is valid
22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-20.04_k8-1.23_weave-2.8.1 revision 1 is supported by CSE 3.1.4
22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.21_weave-2.8.1 revision 1 is supported by CSE 3.1.4
22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.20_weave-2.6.5 revision 3 is supported by CSE 3.1.4
22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.19_weave-2.6.5 revision 3 is supported by CSE 3.1.4
22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.18_weave-2.6.5 revision 3 is supported by CSE 3.1.4
22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template photon-v2_k8-1.14_weave-2.5.2 revision 5 is supported by CSE 3.1.4
22-08-30 15:30:32 | remote_template_manager:159 - _filter_unsupported_templates | DEBUG :: Successfully filtered unsupported templates.
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh' (file already exists)
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh' (file already exists)
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/nfsd.sh' (file already exists)
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/docker-upgrade.sh' (file already exists)
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-cni-apply.sh' (file already exists)
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-k8s-upgrade.sh' (file already exists)
22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/worker-k8s-upgrade.sh' (file already exists)
22-08-30 15:30:33 | template_builder:210 - _upload_source_ova | INFO :: Found ova file 'ubuntu-20.04-server-cloudimg-amd64.ova' in catalog 'cse'
22-08-30 15:30:33 | template_builder:188 - _delete_temp_vapp | INFO :: Deleting temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
22-08-30 15:30:58 | template_builder:196 - _delete_temp_vapp | INFO :: Deleted temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
22-08-30 15:30:58 | template_builder:258 - _create_temp_vapp | INFO :: Creating vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
22-08-30 15:30:58 | core_utils:331 - read_data_file | DEBUG :: Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh
22-08-30 15:31:40 | template_builder:286 - _create_temp_vapp | INFO :: Created vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp'
22-08-30 15:31:40 | template_builder:301 - _customize_vm | INFO :: Customizing vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp', vm 'ubuntu-2004-k8s1233-weave281-vm'
22-08-30 15:31:40 | core_utils:331 - read_data_file | DEBUG :: Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh
22-08-30 15:31:40 | vsphere_utils:74 - get_vsphere | DEBUG :: VM ID: urn:vcloud:vm:f1ba8786-a8f1-4667-8eb9-f5b4ce7270b4, Hostname: chnvprodvc01.chnvsblab.com
22-08-30 15:31:40 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning
22-08-30 15:31:46 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning
22-08-30 15:31:51 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning
22-08-30 15:31:56 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsRunning
22-08-30 15:33:11 | template_builder:351 - _customize_vm | ERROR :: Failed VM customization with error: Error while uploading file: 400
Traceback (most recent call last):
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/templates/template_builder.py", line 343, in _customize_vm
msg_update_callback=self.msg_update_callback))
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 203, in execute_script_in_guest
self.upload_file_to_guest(vm, user, password, content, target)
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 147, in upload_file_to_guest
'Error while uploading file: %s' % resp.status_code)
Exception: Error while uploading file: 400
22-08-30 15:33:11 | configure_cse:633 - install_template | ERROR :: Template Installation Error. Check CSE install logs
Traceback (most recent call last):
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/configure_cse.py", line 599, in install_template
msg_update_callback=msg_update_callback)
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/configure_cse.py", line 2201, in _install_single_template
retain_temp_vapp=retain_temp_vapp
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/templates/template_builder.py", line 491, in build
self._customize_vm(vapp, self.temp_vm_name)
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/templates/template_builder.py", line 343, in _customize_vm
msg_update_callback=self.msg_update_callback))
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 203, in execute_script_in_guest
self.upload_file_to_guest(vm, user, password, content, target)
File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 147, in upload_file_to_guest
'Error while uploading file: %s' % resp.status_code)
Exception: Error while uploading file: 400

Reproduction steps

Executing below command as part of cse installation guide(https://cloudsolutions.vmware.com/content/dam/digitalmarketing/microsites/en/images/cloud-solutions/pdfs/CSE_CDS_DeploymentGuide.pdf).
 cse template install ubuntu-20.04_k8-1.23_weave-2.8.1 -s

Expected behavior

.

Additional context

No response