Can't create instance after upgrade from 4.2.1 to 4.11.2

[velinbudinov]

ISSUE TYPE

• Bug Report

CLOUDSTACK VERSION

CS version 4.11.2 (both manager and agents)

CONFIGURATION

basic networking

OS / ENVIRONMENT

OS Ubuntu 16.04
Java 8

SUMMARY

After successful upgrade from 4.2.1 to 4.11.2, can't create new instances.
Error: "Error while starting VM"

agent.log
manager.log

[yadvr]

Can you describe your env, hypervisors, networking etc? Did you seed the 4.11.2 systemvmtemplate? Did you perform in place mgmt server upgrade or used a new mgmt server host?

[velinbudinov]

Because of 4.11.2 requirements, I had to upgrade OS of the mgmt and agents. I did clean Ubuntu 16.04 installation on mgmt, restored database and /etc/cloudstack directory and installed 4.11.2 packages after that. On agents, OS upgrade with clean Ubuntu 16.04 installation, CS agent 4.11.2 packages and restored agent.properties! Everything is working (start/stop instance, migrate instance between hosts) except create instance!
Environment is: basic networking, 1 mgmt Ubuntu 16.04 with CS version 4.11.2 with 1 cluster KVM - 4 hosts with Ubuntu 16.04 and CS agent 4.11.2 java 8. Virtual router and SSVM and CPVM have been updated with 4.11.2 systemvmtemplate.

[kioie]

From your logs, seems there's a conflict in your host and cluster setup

2019-07-12 14:55:30,721 WARN [o.a.c.alerts] (API-Job-Executor-5:ctx-c22c483d job-4033 ctx-d82ec66b) (logid:c73f95c5) AlertType:: 8 | dataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Failed to deploy Vm with Id: 3206, on Host with Id: null

The reason why start/stop instance, migrate instance between hosts seems to work is because they can find a cluster and host to migrate to as you can see

2019-07-12 14:53:25,926 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-e53ceb96 job-4033/job-4034 ctx-4b496990) (logid:c73f95c5) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId":1,"podId":1,"clusterId":1,"hostId":37,"rawParams":{"VmPassword":"rO0ABXQADnNhdmVkX3Bhc3N3b3Jk"},"userId":2,"accountId":2,"vmId":3206,"handlerName":"VirtualMachineManagerImpl"}

Maybe try and check if your service offerings or maybe storage offering could be limited to a certain zone/cluster or host

[ustcweizhou]

@velinbudinov can you post the result of mysql queries below ?

select broadcast_uri from cloud.networks where id=<network_id>;
select broadcast_uri from cloud.nics where instance_id=<instance id>;

[velinbudinov]

@velinbudinov can you post the result of mysql queries below ?

select broadcast_uri from cloud.networks where id=<network_id>;
select broadcast_uri from cloud.nics where instance_id=<instance id>;

@ustcweizhou both returns NULL

[ustcweizhou]

@velinbudinov can you post the result of mysql queries below ?

select broadcast_uri from cloud.networks where id=<network_id>;
select broadcast_uri from cloud.nics where instance_id=<instance id>;

@ustcweizhou both returns NULL

@velinbudinov can you change broadcast_uri of network to "vlan://untagged" and retry ?

[velinbudinov]

@velinbudinov can you post the result of mysql queries below ?

select broadcast_uri from cloud.networks where id=<network_id>;
select broadcast_uri from cloud.nics where instance_id=<instance id>;

@ustcweizhou both returns NULL

@velinbudinov can you change broadcast_uri of network to "vlan://untagged" and retry ?

@ustcweizhou thanks a lot! That fixed it! :)