ovirt_nic errors with exception "Network 'infra-prod' was not found in datacenter 'BUA'."

[Timoses]

SUMMARY

ovirt_nic reports exception "Network 'infra-prod' was not found in datacenter 'BUA'.".
However, the VM resides in cluster "Default" which is located in datacenter "BAF".
The datacenter "BAF" does have a network named "infra-prod".
Hence, the ovirt_nic module should not throw the exception.

It is possible to manually assign the correct network to an interface for the VM via the GUI (in fact, the drop-down list only lists those networks which are available in datacenter "BAF")

ISSUE TYPE

• Bug Report

COMPONENT NAME

ovirt_nic

ANSIBLE VERSION

ansible 2.9.1
  config file = /home/myuser/code/ansible/project/ansible.cfg
  configured module search path = [u'/home/myuser/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/myuser/.local/lib/python2.7/site-packages/ansible
  executable location = /home/myuser/.local/bin/ansible
  python version = 2.7.15+ (default, Oct  7 2019, 17:39:04) [GCC 7.4.0]

CONFIGURATION

DEFAULT_HOST_LIST(/home/myuser/code/ansible/project/ansible.cfg) = [u'/etc/ansible/inventory/hosts']
DEFAULT_INVENTORY_PLUGIN_PATH(/home/myuser/code/ansible/project/ansible.cfg) = [u'/home/myuser/code/ansible/project/plugins/inventory']
DEFAULT_LOG_PATH(/home/myuser/code/ansible/project/ansible.cfg) = /var/log/ansible.log
DEFAULT_VAULT_PASSWORD_FILE(/home/myuser/code/ansible/project/ansible.cfg) = /home/myuser/.ansible/vault_pass
ENABLE_TASK_DEBUGGER(env: ANSIBLE_ENABLE_TASK_DEBUGGER) = True
HOST_KEY_CHECKING(/home/myuser/code/ansible/project/ansible.cfg) = False
INVENTORY_ENABLED(/home/myuser/code/ansible/project/ansible.cfg) = [u'host_list', u'script', u'auto', u'yaml', u'ini', u'toml']
RETRY_FILES_ENABLED(/home/myuser/code/ansible/project/ansible.cfg) = False

OS / ENVIRONMENT

RHEV-Manager version: 4.3.4.3-0.1.el7

STEPS TO REPRODUCE

- hosts: localhost
  tasks:
  - ovirt_vm:
      auth: "{{ ovirt_auth }}"
      state: "present"
      name: "myvmname"
      cluster: "BAF"
  - ovirt_nic:
      auth: "{{ ovirt_auth }}"
      vm: "myvmname"
      name: "testif"
      interface: "virtio"
      profile: "infra-prod"
      network: "infra-prod"

EXPECTED RESULTS

NIC 'testif' should be created with profile 'infra-prod' in network 'infra-prod'.

ACTUAL RESULTS

Module ovirt_nic errors with exception.

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: Network 'infra-prod' was not found in datacenter 'BA-PERF'.
failed: [myvmname -> localhost] {"ansible_loop_var": "item", "changed": false, "item": [{"name": "myvmname", "profile": {"boot_devices": "hd", "cluster": "Default", "cores": 2, "disks": [{"activate": true, "bootable": true, "format": "raw", "interface": "virtio_scsi", "name": "infra_sys", "size": "50GiB", "storage_domain": "rhev_infra_data_10"}], "io_threads": 0, "memory": "4GiB", "operating_system": "rhel_7x64", "state": "running", "type": "server"}}, {"interface": "virtio", "name": "prod", "network": "infra-prod", "profile": "infra-prod"}], "msg": "Network 'infra-prod' was not found in datacenter 'BUA'."}

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/ovirt/ovirt_nic.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @machacekondra @mnecas @mwperina
click here for bot help

[mnecas]

Hi @Timoses,
I have tried to reproduce the issue but it worked without a problem.
I know you said that the VM is in the data center with network infra-prod, but could you please check it once again. Because we get the data center from the VMs cluster and then get the networks in it.

[Timoses]

Thanks for the quick response @mnecas .

I went ahead and tested below reduced example against our RHEV:

  - ovirt_vm:
      auth: "{{ ovirt_auth }}"
      name: "testvm"
      state: present
      cluster: Default
    delegate_to: localhost
  - ovirt_nic:
      auth: "{{ ovirt_auth }}"
      vm: "testvm"
      name: "testif"
      interface: "virtio"
      profile: "infra-prod"
      network: "infra-prod"
    delegate_to: localhost

Result:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: Network 'infra-prod' was not found in datacenter 'BUA-PERF'.
fatal: [testvm -> localhost]: FAILED! => {"changed": false, "msg": "Network 'infra-prod' was not found in datacenter 'BUA-PERF'."}

On the VM when selecting "Edit" in RHEV-M it shows:

Cluster: Default
(Data Center: BAF-INFRA)

In RHEV-M under Compute -> Clusters:

Name |   | Comment | Data Center | Compatibility Version |
Default |   |   | BAF-INFRA | 4.3 |

Under Network -> Networks / vNIC Profiles:
Both list infra-prod under datacenter BAF-INFRA.

Even when creating the VM in another cluster (e.g. F-INFRA) then it complains with the same error (looking for the infra-prod network in datacenter BUA-PERF).

I've checked at this point: https://github.com/ansible/ansible/blob/v2.9.1/lib/ansible/modules/cloud/ovirt/ovirt_nic.py#L259 and it correctly sets the cluster name.
However: https://github.com/ansible/ansible/blob/v2.9.1/lib/ansible/modules/cloud/ovirt/ovirt_nic.py#L261
dc.name prints the BUA-PERF datacenter... Which is wrong.

ovirt-engine-sdk-python version is 4.3.3 .

Edit:
Seems like this line:

dc = dcs_service.list(search='Clusters.name=%s' % cluster_name)[0]

retrieves more than one result, with BUA-PERF being the first in the list (I disguised the names of our datacenters and clusters. 1 Datacenter = 1 Cluster and both have the same name. BUA-PERF is alphabetically the first). So it seems to simply pick the first Datacenter from the list, even though the cluster name does not match.

[mnecas]

Sorry for the late response.
it cannot be here:
dc = dcs_service.list(search='Clusters.name=%s' % cluster_name)[0]
only if multiple datacenters would have a cluster with the same name and its not possible to creat it. I talked with other devs and they were not sure either.
I tried to reproduce the issue as you described but still nothing I might misunderstood some parts. So I ask if you could give share the infrastructure like.

dc1:
   cluster1:
     ...
dc2:
   cluster2:
      ...

[Timoses]

@mnecas : This is the structure (names changed slightly):

# <datacenter>:
#  - <cluster>

AL-INFRA:
 - AL-INFRA
	
AL-LB-1:
 - AL-LB-1
		
AL-LB-2:
 - AL-LB-2
	
AL-MGMT:
 - AL-MGMT
	
AL-PER:
 - PER
	
AL-VA:
 - VA
	
SUP-INFRA:
 - Default

---

it cannot be here:
dc = dcs_service.list(search='Clusters.name=%s' % cluster_name)[0]

What does "cannot" mean? I'm quite sure when I debugged it that it returned multiple results.
So it's either a bug in the SDK or something else is going on.

I'll try to reproduce above once again (earliest on Monday/Tuesday next week)

Please let me know if you need any more information.

[Timoses]

@mnecas

I tried it once more. I added the following to https://github.com/ansible/ansible/blob/v2.9.2/lib/ansible/modules/cloud/ovirt/ovirt_nic.py#L261 :

cls_service = connection.system_service().clusters_service()
cls = cls_service.list()
dcs = dcs_service.list(search='Clusters.name=%s' % cluster_name)
dcs_all = dcs_service.list()
raise Exception("Available Clusters: %s | Searched cluster: %s | DCs found: %s | DCs: %s" % ([cl.name for cl in cls], cluster_name, len(dcs), [{'name': dc.name, 'clusters': dc.clusters} for dc in dcs_all]))

Result:

TraceALck (most recent call last):
  File "/tmp/ansible_ovirt_nic_payload_v5U_Z9/ansible_ovirt_nic_payload.zip/ansible/modules/cloud/ovirt/ovirt_nic.py", line 266, in main
Exception: Available Clusters: ['PERF', 'AL-INFRA', 'AL-LB-1', 'AL-LB-2', 'Default', 'AL-MGMT', 'VAL'] | Searched cluster: Default | DCs found: 7 | DCs: [{'clusters': [], 'name': 'AL-PERF'}, {'clusters': [], 'name': 'AL-MGMT'}, {'clusters': [], 'name': 'SUP-INFRA'}, {'clusters': [], 'name': 'AL-VAL'}, {'clusters': [], 'name': 'AL-INFRA'}, {'clusters': [], 'name': 'AL-LB-1'}, {'clusters': [], 'name': 'AL-LB-2'}]

I'm not sure why the dcs do not contain any clusters. Is that intended?
It looks like dcs = dcs_service.list(search='Clusters.name=%s' % cluster_name) returns all datacenters.

Bug in ovirt SDK? Something else happening?

[machacekondra]

Can you please try to run following script:

cls_service = connection.system_service().clusters_service()
dcs_service = connection.system_service().data_centers_service()
cls = cls_service.list()
dcs = dcs_service.list(search='Clusters.name=%s' % cluster_name)
dcs_all = dcs_service.list()
raise Exception("Available Clusters: %s | Searched cluster: %s | DCs found(%s): %s | DCs: %s" % ([cl.name for cl in cls], cluster_name, len(dcs), [{'name': dc.name, 'clusters': [c.name for c in connection.follow_link(dc.clusters)]} for dc in dcs], [{'name': dc.name, 'clusters': [c.name for c in connection.follow_link(dc.clusters)]} for dc in dcs_all]))

Where cluster_name is the cluster which doesn't properly work for you (the one which return mutiple results.

dcs variable contains list of datacenters, which has cluster with name cluster_name. The reason you didn't see any values in 'clusters' ouput in your Exception, is because it's link, so you need to use connection.follow_link to get the value out of it, as you can see in code above.

[Timoses]

@machacekondra
The output is:

Traceback (most recent call last):
  File "/tmp/ansible_ovirt_nic_payload_ff9CGr/ansible_ovirt_nic_payload.zip/ansible/modules/cloud/ovirt/ovirt_nic.py", line 267, in main
Exception: Available Clusters: ['PER', 'AL-INFRA', 'AL-LB-1', 'AL-LB-2', 'Default', 'AL-MGMT', 'VA'] | Searched cluster: Default | DCs found(7): [{'clusters': ['PER'], 'name': 'AL-PER'}, {'clusters': ['AL-MGMT'], 'name': 'AL-MGMT'}, {'clusters': ['Default'], 'name': 'SUP-INFRA'}, {'clusters': ['VA'], 'name': 'AL-VA'}, {'clusters': ['AL-INFRA'], 'name': 'AL-INFRA'}, {'clusters': ['AL-LB-1'], 'name': 'AL-LB-1'}, {'clusters': ['AL-LB-2'], 'name': 'AL-LB-2'}] | DCs: [{'clusters': ['PER'], 'name': 'AL-PER'}, {'clusters': ['AL-MGMT'], 'name': 'AL-MGMT'}, {'clusters': ['Default'], 'name': 'SUP-INFRA'}, {'clusters': ['VA'], 'name': 'AL-VA'}, {'clusters': ['AL-INFRA'], 'name': 'AL-INFRA'}, {'clusters': ['AL-LB-1'], 'name': 'AL-LB-1'}, {'clusters': ['AL-LB-2'], 'name': 'AL-LB-2'}]

[machacekondra]

@Timoses Hmm, as you can see both outputs for:

dcs = dcs_service.list(search='Clusters.name=%s' % cluster_name)

and for

dcs_all = dcs_service.list()

are the same. So for cluster Default it doesn't return multiple DCs. Which cluster didin't work for you and returned mutiple DCs?

[Timoses]

are the same.

@machacekondra : Isn't exactly that the problem? (That it returns all dcs for the search?):

dcs_found:

[{'clusters': ['PER'], 'name': 'AL-PER'}, {'clusters': ['AL-MGMT'], 'name': 'AL-MGMT'}, {'clusters': ['Default'], 'name': 'SUP-INFRA'}, {'clusters': ['VA'], 'name': 'AL-VA'}, {'clusters': ['AL-INFRA'], 'name': 'AL-INFRA'}, {'clusters': ['AL-LB-1'], 'name': 'AL-LB-1'}, {'clusters': ['
AL-LB-2'], 'name': 'AL-LB-2'}]

dcs_all:

[{'clusters': ['PER'], 'name': 'AL-PER'}, {'clusters': ['AL-MGMT'], 'name': 'AL-MGMT'}, {'clusters': ['Default'], 'name': 'SUP-INFRA'}, {'clusters': ['VA'], 'name': 'AL-VA'}, {'clusters': ['AL-INFRA'], 'name': 'AL-INFRA'}, {'clusters': ['AL-LB-1'], 'name': 'AL-LB-1'}, {'clusters': ['
AL-LB-2'], 'name': 'AL-LB-2'}]

According to https://github.com/ansible/ansible/blob/v2.9.2/lib/ansible/modules/cloud/ovirt/ovirt_nic.py#L261 we would expect to return only 1 DC (the DC which contains the relevant cluster). Then we can load the correct network interfaces.

!!! However: It appears that dcs_found returns ALL DCs. And line https://github.com/ansible/ansible/blob/v2.9.2/lib/ansible/modules/cloud/ovirt/ovirt_nic.py#L261 simply takes the first list item (as it seems to expect that the returned list should only contain the DC which has the relevant cluster).

So:

• either ovirt-sdk has a bug (in that it returns all DCs instead only the DC which contains the cluster searched for
• or ovirt_nic module uses the ovirt-sdk in a wrong manner.

According to ovirt-sdk documentation the search parameter is used to "restrict the returned data centers".

So for cluster Default it doesn't return multiple DCs.

On the contrary, it appears to return all DCs although the search result should have been restricted to only the DC containing the cluster with the searched name.

Am I missing something?

[Timoses]

Created a bug report at Red Hat Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1792209

[Timoses]

It turns out that the RHEVM user used with the ovirt_nic module (during ovirt_auth) did not have sufficient permissions (see here for more details).