TGK 1.6.0 not deployable - guestinfo.postcustomization.networkconfiguration.status is None

[benedikt-haug]

Describe the bug

Dear developers,

TKG 1.6.0 deployments with kubernetes 1.23 don't work due to "Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None" timeout.
The issue appears in both ubuntu and photon releases, same log output.
I can't connect to the machine via SSH, given SSH is configured directly after this step in the setup.
1.5.4 deploys fine, but kubernetes 1.22 reached end of life on the 28 Oct 2022, so isn't really an option

Could it be that its currently not yet supported, given the documentation only states 1.5.4?

CSE version 3.1.4

If there is additional info you could want, please tell me <3

Deployed config (grabbed from the webinterfaces POST call and redacted orgName, virtualDataCenterName and site):

{"apiVersion":"cse.vmware.com/v2.0","kind":"TKGm","metadata":{"additionalProperties":true,"orgName":"test","virtualDataCenterName":"test-dc","name":"version160test","site":"https://myemployer.xyz"},"spec":{"additionalProperties":true,"topology":{"controlPlane":{"count":1,"sizingClass":"System Default","storageProfile":"mass_FTT-2","cpu":null,"memory":null},"workers":{"count":2,"sizingClass":"System Default","storageProfile":"mass_FTT-2","cpu":null,"memory":null},"nfs":{"count":0,"sizingClass":null,"storageProfile":null}},"settings":{"ovdcNetwork":"vmwareKannKeinEscaping","sshKey":"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOULTplVav8t/WR5ZVvZQWkTRBeCR9OQQ0WnCr+TTHXD ubuntu@cse1","rollbackOnFailure":true,"network":{"expose":true,"pods":{"cidrBlocks":["100.96.0.0/11"]},"services":{"cidrBlocks":["100.64.0.0/13"]}}},"distribution":{"templateName":"ubuntu-2004-kube-v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981","templateRevision":1}}}

How this looks ~/.cse-logs/cse-server-debug.log:

22-10-31 09:35:34 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:39 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:44 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:49 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:54 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:00 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:05 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:10 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:15 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:20 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:25 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:30 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:35 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:40 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:46 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status
 is None
22-10-31 09:36:51 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:56 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:37:01 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:37:06 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:37:06 | pyvcloud_utils:776 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | ERROR :: VM Post guest customization failed due to timeout(600 sec)
22-10-31 09:37:06 | cluster_service_2_x_tkgm:2516 - _add_control_plane_nodes | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | ERROR ::
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/rde/backend/cluster_service_2_x_tkgm.py", line 2499, in _add_control_plane_nodes
    vcd_utils.wait_for_completion_of_post_customization_procedure(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/common/utils/pyvcloud_utils.py", line 777, in wait_for_completion_of_post_customization_procedure
    raise exceptions.PostCustomizationTimeoutError
container_service_extension.exception.exceptions.PostCustomizationTimeoutError
22-10-31 09:37:06 | cluster_service_2_x_tkgm:922 - _create_cluster_async | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | ERROR :: failure on creating nodes ['mstr-32t7']
Error:
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/rde/backend/cluster_service_2_x_tkgm.py", line 2499, in _add_control_plane_nodes
    vcd_utils.wait_for_completion_of_post_customization_procedure(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/common/utils/pyvcloud_utils.py", line 777, in wait_for_completion_of_post_customization_procedure
    raise exceptions.PostCustomizationTimeoutError
container_service_extension.exception.exceptions.PostCustomizationTimeoutError

During handling of the above exception, another exception occurred:

Reproduction steps

1. Start a ubuntu-2004-kube-v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981 (TKG 1.6.0) deployment. Same issue appears with the v1.23 photon image.
2. It gets deleted after a few minutes without succeeding
3. Check logs
   ...

Expected behavior

It should finish successfully and provide a useable cluster.

Additional context

No response

[kutukcub]

I have an same issue in CSE 3.1.3 environment with VCD 10.3.3. While i was creating TKGm cluster, I was facing an issue the cluster creation failed in a minutes and deleting immediately itself. Also I can create Native cluster and there is no issue. I checked the FW there is no drop or disable rule, actually I can not see any action in FW while TKGm cluster was creating.

Error : Error adding control plane node: failure on creating nodes [‘mstr-d69s’]
Error: /root/control_plane.sh: wget -nv github.com/vmware-tanzu/carvel-imgpkg/releases/download/v0.24.0/imgpkg-linux-amd64 -O $imgpkg_path

[benedikt-haug]

It would be nice if one could check if compatibility is intended: