VR is stuck at Starting

[weizhouapache]

This could be same as #8055

When VR is started, there should be some logs like

2023-11-07 21:22:01,197 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-356:ctx-dc3231a7 10.0.34.0, job-745/job-746, cmd: AggregationControlCommand) (logid:107ab608) Run command on VR: 10.0.41.155, script: vr_cfg.sh with args: -c /var/cache/cloud/VR-14e05e67-d677-4a3d-9adc-72e30b19ea3c.cfg

2023-11-07 21:22:01,827 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-356:ctx-dc3231a7 10.0.34.0, job-745/job-746, cmd: AggregationControlCommand) (logid:107ab608) vr_cfg.sh execution result: true

However, sometimes VR is stuck at Starting, in that case, there is only first log, but no second log. There are many (~100) firewall rules /port forwarding rules with the network. The vr_cfg.sh seems to be executed without error (checked /var/log/cloud.log).

The issue seems to be caused by

cloudstack/plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

Lines 1854 to 1855 in 3d7281d

	result = SshHelper.sshExecute(routerIP, DefaultDomRSshPort, "root", getSystemVmKeyFile(), null, "/opt/cloud/bin/" + script + " " + args,
	VRScripts.CONNECTION_TIMEOUT, VRScripts.CONNECTION_TIMEOUT, timeout);

this issue needs to be reproduced and the fix needs to be verified.

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR

CLOUDSTACK VERSION

4.17.2.0

CONFIGURATION

OS / ENVIRONMENT

vmware

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[weizhouapache]

it looks we have faced the same issue today

2023-12-19 19:22:28,043 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-1001:ctx-10021003 kvm2.local, job-10021004/job-10021005, cmd: CheckSshCommand) (logid:981bae10) Run command on VR: x.x.x.x, script: netusage.sh with args: -c
2023-12-19 19:22:28,286 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-1001:ctx-10021003 kvm2.local, job-10021004/job-10021005, cmd: CheckSshCommand) (logid:981bae10) netusage.sh execution result: true

[DaanHoogland]

@JoaoJandre , this one was already merged; setting the milestone back to 4.18.2

[weizhouapache]

@DaanHoogland
they are not same issue.
let's close this one, and reopen it if it happens again