occasional exc for ssh and rsync after VM deploying

[atortosa]

Sometimes the VM deploying can take the following exception indipendently from the fact the VM is deployed alone or in a virtual application:

status: Trying to establish network connectivity to vm_3 (cloud-assigned uuid 8644ff02-3f5e-419e-a48f-67b8668dce78), on IP address 172.18.16.16...
status: Trying to establish network connectivity to vm_1 (cloud-assigned uuid dabbbcff-1f10-44e6-868f-a3a8998da1c6), on IP address 172.18.16.17...
status: Trying to establish network connectivity to vm_2 (cloud-assigned uuid 4ad0054f-921b-406f-8e1a-0a179abfa84b), on IP address 172.18.16.18...
status: Trying to establish network connectivity to vm_4 (cloud-assigned uuid 8b3e1a34-8430-481d-bdc4-86ae24b54168), on IP address 172.18.16.19...
status: Sending a copy of the code tree to vm_1 (172.18.16.17)...
status: Sending a termination request for Instance "vm_1" (cloud-assigned uuid dabbbcff-1f10-44e6-868f-a3a8998da1c6)....
status: Sending a copy of the code tree to vm_4 (172.18.16.19)...
status: Sending a copy of the code tree to vm_3 (172.18.16.16)...
status: Sending a copy of the code tree to vm_2 (172.18.16.18)...
status: Sending a termination request for Instance "vm_2" (cloud-assigned uuid 4ad0054f-921b-406f-8e1a-0a179abfa84b)....
status: Sending a termination request for Instance "vm_3" (cloud-assigned uuid 8644ff02-3f5e-419e-a48f-67b8668dce78)....
status: Sending a termination request for Instance "vm_4" (cloud-assigned uuid 8b3e1a34-8430-481d-bdc4-86ae24b54168)....
AI object A9CEAE4C-2BC8-5F6F-AE1B-735AC607471A (named "ai_1") could not be attached to this experiment: AI pre-attachment operations failure: Parallel object operation failure: VM object 97D32262-EB76-52D3-AD28-95A8F92A40CF (named "vm_1") could not be attached to this experiment: VM post-attachment operations failure: Error while executing the command line "ssh -i /home/cbuser/cbtool/lib/auxiliary//../../credentials/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null cbuser@172.18.16.17 "mkdir -p ~/cbtool;echo '#OSKN-redis' > ~/cb_os_parameters.txt;echo '#OSHN-172.18.144.101' >> ~/cb_os_parameters.txt;echo '#OSPN-6379' >> ~/cb_os_parameters.txt;echo '#OSDN-0' >> ~/cb_os_parameters.txt;echo '#OSTO-240' >> ~/cb_os_parameters.txt;echo '#OSCN-MYOPENSTACK' >> ~/cb_os_parameters.txt;echo '#OSMO-controllable' >> ~/cb_os_parameters.txt;echo '#OSOI-TEST_cbuser:MYOPENSTACK' >> ~/cb_os_parameters.txt;echo '#VMUUID-97D32262-EB76-52D3-AD28-95A8F92A40CF' >> ~/cb_os_parameters.txt;sudo chown -R cbuser /cbtool";rsync -e "ssh -o StrictHostKeyChecking=no -l cbuser -i /home/cbuser/cbtool/lib/auxiliary//../../credentials/id_rsa" --exclude-from '/home/cbuser/cbtool/lib/auxiliary//../../exclude_list.txt' -az --delete --no-o --no-g --inplace -O /home/cbuser/cbtool/lib/auxiliary//../../* 172.18.16.17:/cbtool/" (returncode = 27228) :ssh: connect to host 172.18.16.17 port 22: Connection refused
ssh: connect to host 172.18.16.17 port 22: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]
A rollback might be needed (only for VMs).

[bowers]

I often got this error in one cloud that takes a long time (several minutes) to create a new VM, Sometimes, instead of "connection refused", I got a "Broken pipe (32) rsync: connection unexpectedly closed".

Marcio just enhanced the active_operations.py code to include retries on the rsync call. I'd implemented a similar fix in my own fork of cloudbench, and it cured the rsync failures entirely for me.

[ibmcb]

Fixed (apparently) in commit id 2e6c95f. Feel free to re-open if needed.