Problems with attaching tinyvm instance to experiment in GCE

[marcinkaczor]

Hello. As part of a master's thesis, I would like to perform some experiments in GCE cloud via CBTOOL. My first aim is to run nullworkload appliction. I took the following steps:

• performed initial installation of CBTOOL
• configured CBTOOL orchestrator node to be run outside a cloud (on my PC)
  (steps detailed here: https://github.com/ibmcb/cbtool/wiki/FAQ-S#sq6)
• prepared my GCE cloud to be drived by CBTOOL
  (according to steps: https://github.com/ibmcb/cbtool/wiki/HOWTO:-Preparing-your-cloud-to-be-driven-by-CBTOOL#gce)
• successfully prepared an image to be used with CBTOOL in my cloud
  • vmattach check:ubuntu-1804-bionic-v20200626:cbuser:nullworkload
  • vmcapture youngest cb_nullworkload

Unfortunately, when I am trying to attach the instance to my experiment by the command:

• vmattach tinyvm

I get the following error (as I guess it is at the end of a pipeline...):

• status: Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 35.229.69.110, port 22 after attempt 0. Will try 36 more times.

The full log is visible in the above screenshot.

Have you got any ideas what is causing this problem that post boot script cannot be executed on VM?

I would be really glad if you could help me with that.

Thank you.

[maugustosilva]

oops, apologies for the long delay. We were busy finishing the conversion of the whole code for python3 :-). I recommend you start by trying what is described under Debugging single-VM attachment errors in https://github.com/ibmcb/cbtool/wiki/HOWTO:-Debug-initial-setup. This should allow you to login on the VM and run ~/cbtool/scripts/common/cb_post_boot.sh directly there...

[Giuseppe1992]

There is the same problem using Amazon EC2
this is the log that I have:

Aug  4 14:40:33 ip-172-31-29-18.eu-central-1.compute.internal cloudbench [DEBUG] process_management.py/ProcessManagement.retriable_run_os_command process_management - Command "~/cbtool/scripts/common/cb_post_boot.sh" failed to execute on hostname 172.31.29.217, port 22 after attempt 14. Will try 22 more times.

[mraygalaxy]

@marcinkaczor This is a very common networking problem for new users. cbtool has to be installed in a location that is reachable by the the cloud (on all ports). If it is not, things will not work.

You have many options here: 1) Put cbtool in the cloud itself 2) Use our VPN support and run it outside of the cloud or 3) Use a jumpbox

There are instructions on the wiki for doing both 2) and 3) if you don't want to do 1)

[marcinkaczor]

@maugustosilva, @mraygalaxy:

After issuing the command vmattach tinyvm on the orchestrator, I opened a second terminal view, logged on the VM (tinyvm) and ran ~/cbtool/scripts/common/cb_post_boot.sh directly there. It turned out that the script execution was interrupted, because there was a problem with cb_common.sh script with ntp service: Failed to restart ntp.service: Unit ntp.service not found. I installed ntp by executing sudo apt install ntp and succesfully ran cb_post_boot.sh script once again on this VM. Then, I came back to my orchestrator and it turned out that VM was succesfully attached to the experiment.

Summarizing, in the end, I succesfully attached the VM to my experiment but it required my manual intervention... I see I am not the only one who struggles with this problem :(

Could you please look at it, guys?

I attach the log from unsuccessful cb_post_boot.sh script execution on VM:

cbuser@cb-marcin-mygce-vm2-tinyvm:~$ ./cbtool/scripts/common/cb_post_boot.sh
open
port checker: host 10.142.0.45 is open.
cb_common.sh (40400): Starting generic VM post_boot configuration
cb_common.sh (40400): VMs need to be able to perform passwordless SSH between each other. Updating ~/.ssh/id_rsa with the contents from "~/cbtool/credentials/cbtool_rsa" to be the same on all VMs..
cb_common.sh (40400): Relaxing all security configurations
cb_common.sh (40400): Stopping service "ufw" with command "sudo systemctl stop ufw"...
cb_common.sh (40400): Disabling service "ufw" with command "sudo systemctl disable ufw"...
Synchronizing state of ufw.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ufw
cb_common.sh (40400): Disabling Apparmor...
cb_common.sh (40400): Stopping service "apparmor" with command "sudo systemctl stop apparmor"...
cb_common.sh (40400): Disabling service "apparmor" with command "sudo systemctl disable apparmor"...
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable apparmor
 * Unloading AppArmor profiles                                                                                                                                [ OK ] 
cb_common.sh (40400): Done
cb_common.sh (40400): Starting (AI) Log store...
cb_common.sh (40400): Local (AI) Log store started
cb_common.sh (40400): Starting (AI) Object store...
cb_common.sh (40400): Stopping service "redis-server" with command "sudo systemctl stop redis-server"...
cb_common.sh (40400): Disabling service "redis-server" with command "sudo systemctl disable redis-server"...
Synchronizing state of redis-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable redis-server
cb_common.sh (40400): Updating object store configuration template
cb_common.sh (40400): Restarting service "redis-server", with command "sudo service redis-server restart", attempt 1 of 7...
cb_common.sh (40400): Service "redis-server" was successfully restarted
cb_common.sh (40400): Enabling service "redis-server", with command "sudo update-rc.d -f redis-server defaults"...
cb_common.sh (40400): Local (AI) Object store started
cb_common.sh (40400): Making sure all NICs on this instance have IPs configured ...
cb_common.sh (40400): Refreshing hosts file ... 
cb_common.sh (40400): Enabling firewall via ufw commands...
Skipping adding existing rule
Skipping adding existing rule
Skipping adding existing rule
Skipping adding existing rule
Skipping adding existing rule
Skipping adding existing rule
Skipping adding existing rule
cb_common.sh (40400): Executing "post_boot_steps" function
cb_common.sh (40400): Nested container not requested.
cb_common.sh (40400): Stopping service "ganglia-monitor" with command "sudo systemctl stop ganglia-monitor"...
cb_common.sh (40400): Disabling service "ganglia-monitor" with command "sudo systemctl disable ganglia-monitor"...
ganglia-monitor.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ganglia-monitor
cb_common.sh (40400): Stopping service "gmetad" with command "sudo service gmetad stop"...
Failed to stop gmetad.service: Unit gmetad.service not loaded.
cb_common.sh (40400): Disabling service "gmetad" with command "sudo update-rc.d -f gmetad remove"...
cb_common.sh (40400): Killing previously running ganglia monitoring processes on cb-marcin-mygce-vm2-tinyvm
cb_common.sh (40400): Ganglia monitoring processes killed successfully on cb-marcin-mygce-vm2-tinyvm
cb_common.sh (40400): Previously running ganglia monitoring processes killed cb-marcin-mygce-vm2-tinyvm
cb_common.sh (40400): Stopping service "ntp" with command "sudo service ntp stop"...
Failed to stop ntp.service: Unit ntp.service not loaded.
cb_common.sh (40400): Disabling service "ntp" with command "sudo update-rc.d -f ntp remove"...
cb_common.sh (40400): Creating ntp (ntp.conf) file
open
port checker: host 10.142.0.45 is open.
cb_common.sh (43282): Force ntp time synchronization with server 10.142.0.45
sudo: ntpdate: command not found
/bin/cp: cannot create regular file '/var/lib/ntp/ntp.conf.dhcp': No such file or directory
cb_common.sh (40400): Forcing clock update from ntp
b''
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 1 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 2 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 3 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 4 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 5 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 6 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Restarting service "ntp", with command "sudo service ntp restart", attempt 7 of 7...
Failed to restart ntp.service: Unit ntp.service not found.
cb_common.sh (40400): Service "ntp" failed to restart after 7 attempts

[marcinkaczor]

Hello,

I added ntp to the cloud-init packages:

...
[VM_DEFAULTS : GCE_CLOUDCONFIG]
CLOUDINIT_PACKAGES = ntp
...

And the problem no loger exists.