no vCenter server visible in list while present in DB #103

bigtrodat · 2019-05-07T10:54:52Z

Expected Behavior

After entering vCenter Server Config (hostname, username, password,...) the vCenter server should appear in list and connected VMWare Hosts and VMs shoudl be visible as well.

Current Behavior

Although config is present, no vCenter Server is shown in the list.

FunFact1: Data is visible in icinga2_vsphere MariaDB:

MariaDB [icinga2_vsphere]> select * from vcenter_server\G
*************************** 1. row ***************************
             id: 2
     vcenter_id: 1
           host: srv-vcenter01.company.localdom
         scheme: https
       username: monitoring.readonly@VSPHERE.LOCAL
       password: SECRET
     proxy_type: NULL
  proxy_address: NULL
     proxy_user: NULL
     proxy_pass: NULL
ssl_verify_peer: y
ssl_verify_host: y
        enabled: y
1 row in set (0.00 sec)

MariaDB [icinga2_vsphere]> select * from vcenter\G
*************************** 1. row ***************************
                     id: 1
          instance_uuid: ▒▒t ▒@Ń▒˟C▒▒v
         trust_store_id: NULL
                   name: VMware vCenter Server
                version: 6.7.0
                os_type: linux-x64
               api_type: VirtualCenter
            api_version: 6.7.1
                  build: 11727113
                 vendor: VMware, Inc.
           product_line: vpx
   license_product_name: VMware VirtualCenter Server
license_product_version: 6.0
           locale_build: 000
         locale_version: INTL
1 row in set (0.01 sec)

FunFact 2: Debug shows sync seems to run:

# /usr/bin/icingacli vspheredb daemon run --trace --debug
Ready to run
DB configuration loaded
Reconnecting to DB
Connected to the database
vCenter ID=2 is now starting
Constructing ServerRunner
Starting for vCenterID=1
Initializing vCenter for srv-vcenter01.company.localdom
SOAPClient: got 4.33 KiB response in 112.57ms
vCenter hasn't been changed
Running vCenter Sync for ID=1
Pid 22544 stopped
Ready to fetch id/name/parent list
SOAPClient: got 4.33 KiB response in 112.94ms
SOAPClient: got 20.77 KiB response in 1299.77ms
Got id/name/parent for 61 objects
Task moRefs failed: Undefined property: stdClass::$overallStatus
Loading existing HostSystem from DB
Got 4 existing HostSystem
SOAPClient: got 22.12 KiB response in 1186.80ms
Got 4 HostSystem from VCenter
Ready to store HostSystem
HostSystem: 0 new, 0 modified, 0 deleted (got 4 from API)
Task "Host Systems" took 1192.04ms on srv-vcenter01.company.localdom
Loading existing VirtualMachine from DB
Got 33 existing VirtualMachine
SOAPClient: got 81.44 KiB response in 1177.72ms
Got 33 VirtualMachine from VCenter
Ready to store VirtualMachine
VirtualMachine: 0 new, 1 modified, 0 deleted (got 33 from API)
Task "Virtual Machines" took 1199.50ms on srv-vcenter01.company.localdom
Ready to sync QuickStats for Hosts and VMs
Loading existing HostSystem from DB
Got 4 existing HostSystem
SOAPClient: got 2.62 KiB response in 1122.26ms
Got 4 HostSystem from VCenter
Ready to store HostSystem
HostSystem: 0 new, 0 modified, 0 deleted (got 4 from API)
Loading existing VirtualMachine from DB
Got 33 existing VirtualMachine
SOAPClient: got 67.96 KiB response in 1407.68ms
Got 33 VirtualMachine from VCenter
Ready to store VirtualMachine
VirtualMachine: 0 new, 33 modified, 0 deleted (got 33 from API)
Task "Quick Stats" took 2563.84ms on srv-vcenter01.company.localdom
Loading existing Datastore from DB
Got 8 existing Datastore
SOAPClient: got 4.93 KiB response in 1136.15ms
Got 8 Datastore from VCenter
Ready to store Datastore
Datastore: 0 new, 0 modified, 0 deleted (got 8 from API)
Task "Data Stores" took 1139.18ms on srv-vcenter01.company.localdom
SOAPClient: got 18.55 KiB response in 1276.73ms
Got VirtualMachine perDatastoreUsage
0 created, 2 changed, 0 deleted
Task "VM DataStore Usage" took 1309.00ms on srv-vcenter01.company.localdom
SOAPClient: got 14.88 KiB response in 1197.57ms
Got 33 VirtualMachines with guest.disk
Got 68 vm_disk_usage objects from DB
0 created, 12 changed, 0 deleted out of 68 objects (API: 68)
Task "VM Disk Usage" took 1229.48ms on srv-vcenter01.company.localdom
SOAPClient: got 2.38 KiB response in 1038.72ms
Got 33 VirtualMachines with snapshot
Got 0 vm_snapshot objects from DB
0 created, 0 changed, 0 deleted out of 0 objects (API: 0)
Task "VM Snapshots" took 1040.87ms on srv-vcenter01.company.localdom
SOAPClient: got 224.12 KiB response in 1280.15ms
Got 33 VirtualMachines with config.hardware
Got 458 vm_hardware objects from DB
Got 59 vm_disk objects from DB
Got 34 vm_network_adapter objects from DB
0 created, 0 changed, 0 deleted out of 458 objects (API: 458)
0 created, 0 changed, 0 deleted out of 59 objects (API: 458)
0 created, 0 changed, 0 deleted out of 34 objects (API: 458)
Task "VM Hardware" took 1313.24ms on srv-vcenter01.company.localdom
SOAPClient: got 283.60 KiB response in 1992.52ms
Got 4 HostSystems with hardware.pciDevice
Got 812 host_pci_device objects from DB
0 created, 48 changed, 0 deleted out of 812 objects (API: 812)
Task "Host Hardware" took 2056.57ms on srv-vcenter01.company.localdom
SOAPClient: got 259.11 KiB response in 2048.16ms
Got 0 HostSensors with 4
Got 567 host_sensor objects from DB
0 created, 0 changed, 0 deleted out of 567 objects (API: 567)
Task "Host Sensors" took 2079.08ms on srv-vcenter01.company.localdom
SOAPClient: got 547.00 B response in 5424.88ms
SOAPClient: got 396.00 B response in 161.40ms
SOAPClient: got 2.28 KiB response in 698.35ms
1557225570121 <= 1557225570121 & 89161 <= 89161 skipped
Fetched 1 events to skip
Got 2 events
Task "Event Stream" took 6296.13ms on srv-vcenter01.company.localdom
SOAPClient: got 394.00 B response in 413.68ms
Task "Event Stream" took 413.93ms on srv-vcenter01.company.localdom
SOAPClient: got 394.00 B response in 358.36ms
Task "Event Stream" took 358.59ms on srv-vcenter01.company.localdom
SOAPClient: got 394.00 B response in 298.11ms
Task "Event Stream" took 298.41ms on srv-vcenter01.company.localdom
SOAPClient: got 394.00 B response in 466.02ms
Task "Event Stream" took 466.29ms on srv-vcenter01.company.localdom

Possible Solution

Maybe the "@" sign in vCenter username is a problem?
Maybe the message "Task moRefs failed: Undefined property: stdClass::$overallStatus" in trace is an indicator?
Just a guess.

Steps to Reproduce (for bugs)

Create a new vCenter Server in icingaweb2 UI

Your Environment

VMware vCenter®/ESXi™-Version: 6.7.0
Version/GIT-Hash of this module: 1.1.0
Operating System and version: Debian 9
Webserver, PHP versions:
Icinga Web 2 version: 2.6.3
vspheredb: 1.0.3
ipl: v0.2.1
incubator: v0.2.0
reactbundle: v0.5.0 (dev-master)

The text was updated successfully, but these errors were encountered:

Thomas-Gelf · 2019-05-07T10:56:27Z

Your issue should have been fixed, could you please try the current master branch?

bigtrodat · 2019-05-07T11:49:28Z

That was a quick answer. Unfortunately I'm already at the latest version because I did installation today the first time. But did the git pull again:

# cd /usr/share/icingaweb2/modules/vspheredb
/usr/share/icingaweb2/modules/vspheredb# git pull
Bereits aktuell.
/usr/share/icingaweb2/modules/vspheredb# git status
Auf Branch master
Ihr Branch ist auf dem selben Stand wie 'origin/master'.
nichts zu committen, Arbeitsverzeichnis unverändert

bigtrodat · 2019-05-07T12:10:58Z

Just realized, that objects table is empty while other tables (vcenter, vcenter_server, host_system, virtual_server, ...) have valid content. Hopefully this helps to find the reason.

MariaDB [icinga2_vsphere]> select * from object\G
Empty set (0.00 sec)

MariaDB [icinga2_vsphere]>

Thomas-Gelf · 2019-05-08T06:41:56Z

Please stop the daemon and try the following command:

icingacli vspheredb task sync --vCenterId 1 --debug --trace

Thomas-Gelf · 2019-05-08T06:46:37Z

After that, you might want to give the pull request #98 a try. In case that works for you please let me know. I'll then provide you with some more debugging code, as I'm pretty curious to figure out what kind of object comes with no status.

bigtrodat · 2019-05-10T09:48:20Z

Hi Thomas, sorry for my late response. With your suggestion
icingacli vspheredb task sync --vCenterId 1 --debug --trace
my colleague was able to fix the issue. Everything works now as expected.

Thomas-Gelf · 2019-05-10T09:54:09Z

@bigtrodat: glad to hear that! How did you fix the issue?

fnorded · 2019-07-19T08:49:30Z

Hi Thomas, I can confirm the issue - same here.
Sync seems to work, configuration is stored in the database but the objects table is empty.
There is nothing rendered in the web-ui under the categories: Virtual Machines, Hosts and Datastores.
There is content under Alarm History and Event History though i can't be matched to other objects.

Output of your suggestion:

#icingacli vspheredb task sync --vCenterId 1 --debug --trace
Ready to fetch id/name/parent list
SOAPClient: got 4.33 KiB response in 131.86ms
SOAPClient: got 40.12 KiB response in 1010.57ms
Got id/name/parent for 116 objects
Storing object tree to DB
Created 116 Managed Objects, 0 modified, 0 deleted
Task "Managed Object References" took 1328.77ms on vcenter.local.dom
Loading existing HostSystem from DB
Got 4 existing HostSystem
SOAPClient: got 16.89 KiB response in 1007.30ms
Got 4 HostSystem from VCenter
Ready to store HostSystem
HostSystem: 0 new, 0 modified, 0 deleted (got 4 from API)
Task "Host Systems" took 1010.68ms on vcenter.local.dom
Loading existing VirtualMachine from DB
Got 40 existing VirtualMachine
SOAPClient: got 98.78 KiB response in 1020.10ms
Got 40 VirtualMachine from VCenter
Ready to store VirtualMachine
VirtualMachine: 0 new, 0 modified, 0 deleted (got 40 from API)
Task "Virtual Machines" took 1030.76ms on vcenter.local.dom
Ready to sync QuickStats for Hosts and VMs
Loading existing HostSystem from DB
Got 4 existing HostSystem
SOAPClient: got 2.64 KiB response in 1007.10ms
Got 4 HostSystem from VCenter
Ready to store HostSystem
HostSystem: 0 new, 4 modified, 0 deleted (got 4 from API)
Loading existing VirtualMachine from DB
Got 40 existing VirtualMachine
SOAPClient: got 82.08 KiB response in 1012.13ms
Got 40 VirtualMachine from VCenter
Ready to store VirtualMachine
VirtualMachine: 0 new, 40 modified, 0 deleted (got 40 from API)
Task "Quick Stats" took 2175.32ms on vcenter.local.dom
Loading existing Datastore from DB
Got 10 existing Datastore
SOAPClient: got 6.34 KiB response in 1003.34ms
Got 10 Datastore from VCenter
Ready to store Datastore
Datastore: 0 new, 0 modified, 0 deleted (got 10 from API)
Task "Data Stores" took 1006.00ms on vcenter.local.dom
SOAPClient: got 21.92 KiB response in 1005.90ms
Got VirtualMachine perDatastoreUsage
0 created, 0 changed, 0 deleted
Task "VM DataStore Usage" took 1157.26ms on vcenter.local.dom
SOAPClient: got 18.60 KiB response in 1006.60ms
Got 40 VirtualMachines with guest.disk
Got 86 vm_disk_usage objects from DB
0 created, 0 changed, 0 deleted out of 86 objects (API: 86)
Task "VM Disk Usage" took 1010.79ms on vcenter.local.dom
SOAPClient: got 4.80 KiB response in 1004.26ms
Got 40 VirtualMachines with snapshot
Got 4 vm_snapshot objects from DB
0 created, 0 changed, 0 deleted out of 4 objects (API: 4)
Task "VM Snapshots" took 1007.01ms on vcenter.local.dom
SOAPClient: got 296.37 KiB response in 1021.54ms
Got 40 VirtualMachines with config.hardware
Got 587 vm_hardware objects from DB
Got 63 vm_disk objects from DB
Got 62 vm_network_adapter objects from DB
0 created, 0 changed, 0 deleted out of 587 objects (API: 587)
0 created, 0 changed, 0 deleted out of 63 objects (API: 587)
0 created, 0 changed, 0 deleted out of 62 objects (API: 587)
Task "VM Hardware" took 1062.85ms on vcenter.local.dom
SOAPClient: got 215.28 KiB response in 1015.25ms
Got 4 HostSystems with hardware.pciDevice
Got 576 host_pci_device objects from DB
0 created, 0 changed, 0 deleted out of 576 objects (API: 576)
Task "Host Hardware" took 1051.91ms on vcenter.local.dom
SOAPClient: got 180.10 KiB response in 1015.84ms
Got 0 HostSensors with 4
Got 396 host_sensor objects from DB
0 created, 0 changed, 0 deleted out of 396 objects (API: 396)
Task "Host Sensors" took 1035.81ms on vcenter.local.dom
SOAPClient: got 547.00 B response in 1334.20ms
SOAPClient: got 396.00 B response in 1.92ms
SOAPClient: got 1000.00 B response in 257.73ms
1563222911670 <= 1563222911670 & 3810945 <= 3810945 skipped
Fetched 1 events to skip
Got 1 events
Task "Event Stream" took 1598.10ms on vcenter.local.dom

I tried pull request #98 - no change

fnorded · 2019-09-12T17:23:35Z

Came back to this today, pulled current master, deleted and recreated the database but still the same issue. Log shows sync activity, object table stays empty and no vCenter, Hosts, Datastores, VMs in the GUI.

firatalkis · 2019-11-12T08:50:09Z

hello everybody,

I'm having the same problem.

We have two vcenter but I'm having the problem only in one

@Thomas-Gelf, output of your suggestion:

-bash-4.2$ icingacli vspheredb task sync --vCenterId 1 --debug --trace
Ready to fetch id/name/parent list
Task moRefs failed: CURL ERROR: Operation timed out after 5001 milliseconds with 0 out of 0 bytes received
#0 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/CurlLoader.php(213): Icinga\Module\Vspheredb\CurlLoader->request('POST', 'https://vcenter...', '<?xml version="...', Array)
#1 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/SoapClient.php(60): Icinga\Module\Vspheredb\CurlLoader->post('https://vcenter...', '<?xml version="...', Array)
#2 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/SoapClient.php(41): Icinga\Module\Vspheredb\SoapClient->doCurlRequest('<?xml version="...', 'https://vcenter...', 'urn:vim25/6.7.3', 1, 0)
#3 [internal function]: Icinga\Module\Vspheredb\SoapClient->__doRequest('<?xml version="...', 'https://vcenter...', 'urn:vim25/6.7.3', 1, 0)
#4 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Api.php(200): SoapClient->__soapCall('RetrieveService...', Array)
#5 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Api.php(302): Icinga\Module\Vspheredb\Api->soapCall('RetrieveService...', Object(SoapVar))
#6 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/LazyApiHelpers.php(108): Icinga\Module\Vspheredb\Api->fetchServiceInstance()
#7 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/PropertyCollector.php(22): Icinga\Module\Vspheredb\Api->getServiceInstance()
#8 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/LazyApiHelpers.php(92): Icinga\Module\Vspheredb\PropertyCollector->__construct(Object(Icinga\Module\Vspheredb\Api))
#9 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Sync/SyncManagedObjectReferences.php(165): Icinga\Module\Vspheredb\Api->propertyCollector()
#10 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Sync/SyncManagedObjectReferences.php(35): Icinga\Module\Vspheredb\Sync\SyncManagedObjectReferences->fetchNames()
#11 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Daemon/SyncRunner.php(89): Icinga\Module\Vspheredb\Sync\SyncManagedObjectReferences->sync()
#12 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Daemon/SyncRunner.php(206): Icinga\Module\Vspheredb\Daemon\SyncRunner->Icinga\Module\Vspheredb\Daemon{closure}()
#13 /usr/share/icingaweb2/modules/vspheredb/library/Vspheredb/Daemon/SyncRunner.php(188): Icinga\Module\Vspheredb\Daemon\SyncRunner->runNextImmediateTask()
#14 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/Timer/Timers.php(96): Icinga\Module\Vspheredb\Daemon\SyncRunner->Icinga\Module\Vspheredb\Daemon{closure}(Object(React\EventLoop\Timer\Timer))
#15 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(183): React\EventLoop\Timer\Timers->tick()
#16 /usr/share/icingaweb2/modules/vspheredb/application/clicommands/TaskCommand.php(109): React\EventLoop\StreamSelectLoop->run()
#17 /usr/share/php/Icinga/Cli/Loader.php(265): Icinga\Module\Vspheredb\Clicommands\TaskCommand->syncAction()
#18 /usr/share/php/Icinga/Application/Cli.php(152): Icinga\Cli\Loader->dispatch()
#19 /usr/share/php/Icinga/Application/Cli.php(142): Icinga\Application\Cli->dispatchOnce()
#20 /usr/bin/icingacli(7): Icinga\Application\Cli->dispatch()
#21 {main}

but adata available on the database

firatalkis · 2019-11-12T09:05:55Z

@bigtrodat can you tell How did you fix the issue?

fnorded · 2020-03-16T12:45:52Z

Just to keep this updated:
I added a second vcenter a few weeks ago and for this one everything works perfectly.
So it seems that it is some kind of configuration / content issue of the vcenter that leads to the objects table not being written for the first vcenter.

skupjoe · 2020-05-20T21:00:59Z

We are facing this issue exactly. The first daemon run/sync goes through fine--albeit slowly, with some tasks taking over a few minutes--and all the tables in the vspheredb seem to populate, but the objects list is empty, and so nothing renders and we cannot source this data with our import rules.

We don't have this issue on our vcenter 6.5 server, and now we are trying on 6.7u3b. I've tried v1.1.0 release & the latest master branch.

I've also tried adding the center entry twice, once with hostname and other with IP, but this doesn't seem to fix it either. @fnorded were you able to fix both centers that you added, is only the 2nd one working?

skupjoe · 2020-06-03T03:14:31Z

Turns out my issue was due to an unsupported OpaqueNetwork object type and I fixed by updating the schema:

ALTER TABLE vspheredb.`object` MODIFY COLUMN object_type enum('ComputeResource','ClusterComputeResource','Datacenter','Datastore','DatastoreHostMount','DistributedVirtualPortgroup','DistributedVirtualSwitch','Folder','HostMountInfo','HostSystem','Network','OpaqueNetwork','ResourcePool','StoragePod','VirtualApp','VirtualMachine','VmwareDistributedVirtualSwitch') CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL;

I've opened up #167 to better handle unsupported object types during sync.

Since @bigtrodat's issue and mine are resolved, I think this issue can be closed.

Thomas-Gelf · 2020-08-14T14:14:19Z

#167 has been scheduled for the next release, closing here as requested.

skupjoe mentioned this issue Jun 3, 2020

Handle/add increased logging for unsupported object types #167

Closed

Thomas-Gelf closed this as completed Aug 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no vCenter server visible in list while present in DB #103

no vCenter server visible in list while present in DB #103

bigtrodat commented May 7, 2019 •

edited

Thomas-Gelf commented May 7, 2019

bigtrodat commented May 7, 2019 •

edited

bigtrodat commented May 7, 2019

Thomas-Gelf commented May 8, 2019

Thomas-Gelf commented May 8, 2019

bigtrodat commented May 10, 2019

Thomas-Gelf commented May 10, 2019

fnorded commented Jul 19, 2019

fnorded commented Sep 12, 2019

firatalkis commented Nov 12, 2019

firatalkis commented Nov 12, 2019

fnorded commented Mar 16, 2020

skupjoe commented May 20, 2020 •

edited

skupjoe commented Jun 3, 2020 •

edited

Thomas-Gelf commented Aug 14, 2020

no vCenter server visible in list while present in DB #103

no vCenter server visible in list while present in DB #103

Comments

bigtrodat commented May 7, 2019 • edited

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Your Environment

Thomas-Gelf commented May 7, 2019

bigtrodat commented May 7, 2019 • edited

bigtrodat commented May 7, 2019

Thomas-Gelf commented May 8, 2019

Thomas-Gelf commented May 8, 2019

bigtrodat commented May 10, 2019

Thomas-Gelf commented May 10, 2019

fnorded commented Jul 19, 2019

fnorded commented Sep 12, 2019

firatalkis commented Nov 12, 2019

firatalkis commented Nov 12, 2019

fnorded commented Mar 16, 2020

skupjoe commented May 20, 2020 • edited

skupjoe commented Jun 3, 2020 • edited

Thomas-Gelf commented Aug 14, 2020

bigtrodat commented May 7, 2019 •

edited

bigtrodat commented May 7, 2019 •

edited

skupjoe commented May 20, 2020 •

edited

skupjoe commented Jun 3, 2020 •

edited