Re-run fails with "RTNETLINK answers: File exists" when setting routing rules

[ganto]

I'm trying to setup a oVirt 4.4-alpha hosted-engine with ovirt-ansible-hosted-engine-setup-1.0.35-1.el8.noarch which is part of the official ovirt-4.4-pre repository.

If I re-run engine-setup after a failed installation attempt the Ansible role aborts with the following error:

2020-02-18 16:17:29,895+0100 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Get routing rules, IPv4'}
2020-02-18 16:17:29,896+0100 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Get routing rules, IPv4 kwargs is_conditional:False 
2020-02-18 16:17:29,896+0100 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Get routing rules, IPv4 kwargs 
2020-02-18 16:17:30,441+0100 DEBUG var changed: host "localhost" var "route_rules_ipv4" type "<class 'dict'>" value: "{
    "changed": true,
    "cmd": [
        "ip",
        "rule"
    ],
    "delta": "0:00:00.003368",
    "end": "2020-02-18 16:17:30.106158",
    "failed": false,
    "rc": 0,
    "start": "2020-02-18 16:17:30.102790",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "0:\tfrom all lookup local \n100:\tfrom all to 192.168.222.1/24 lookup main \n100:\tfrom all to 192.168.1.1/24 lookup main \n101:\tfrom 192.168.222.1/24 lookup main \n101:\tfrom 192.168.1.1/24 lookup main \n32766:\tfrom all lookup main \n32767:\tfrom all lookup default ",
    "stdout_lines": [
        "0:\tfrom all lookup local ",
        "100:\tfrom all to 192.168.222.1/24 lookup main ",
        "100:\tfrom all to 192.168.1.1/24 lookup main ",
        "101:\tfrom 192.168.222.1/24 lookup main ",
        "101:\tfrom 192.168.1.1/24 lookup main ",
        "32766:\tfrom all lookup main ",
        "32767:\tfrom all lookup default "
    ]
}"
2020-02-18 16:17:30,441+0100 INFO ansible ok {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_host': 'localhost', 'ansible_task': 'Get routing rules, IPv4', 'task_duration': 0}
2020-02-18 16:17:30,442+0100 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7fb2cdbf1c18> kwargs 
2020-02-18 16:17:30,779+0100 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : debug'}
[...]
2020-02-18 16:17:35,914+0100 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Fetch IPv4 CIDR for virbr0'}
2020-02-18 16:17:35,914+0100 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Fetch IPv4 CIDR for virbr0 kwargs is_conditional:False 
2020-02-18 16:17:35,915+0100 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Fetch IPv4 CIDR for {{ virbr_default }} kwargs 
2020-02-18 16:17:36,305+0100 DEBUG var changed: host "localhost" var "virbr_cidr_ipv4" type "<class 'ansible.utils.unsafe_proxy.AnsibleUnsafeText'>" value: ""192.168.222.1/24""
2020-02-18 16:17:36,305+0100 INFO ansible ok {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_host': 'localhost', 'ansible_task': 'Fetch IPv4 CIDR for virbr0', 'task_duration': 0}
[...]
2020-02-18 16:17:38,664+0100 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Add IPv4 outbound route rules'}
2020-02-18 16:17:38,664+0100 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Add IPv4 outbound route rules kwargs is_conditional:False 
2020-02-18 16:17:38,665+0100 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Add IPv4 outbound route rules kwargs 
2020-02-18 16:17:39,214+0100 DEBUG var changed: host "localhost" var "result" type "<class 'dict'>" value: "{
    "changed": true,
    "cmd": [
        "ip",
        "rule",
        "add",
        "from",
        "192.168.222.1/24",
        "priority",
        "101",
        "table",
        "main"
    ],
    "delta": "0:00:00.002805",
    "end": "2020-02-18 16:17:38.875350",
    "failed": true,
    "msg": "non-zero return code",
    "rc": 2,
    "start": "2020-02-18 16:17:38.872545",
    "stderr": "RTNETLINK answers: File exists",
    "stderr_lines": [
        "RTNETLINK answers: File exists"
    ],
    "stdout": "",
    "stdout_lines": []
}"
2020-02-18 16:17:39,214+0100 DEBUG var changed: host "localhost" var "ansible_play_hosts" type "<class 'list'>" value: "[]"
2020-02-18 16:17:39,214+0100 DEBUG var changed: host "localhost" var "ansible_play_batch" type "<class 'list'>" value: "[]"
2020-02-18 16:17:39,214+0100 DEBUG var changed: host "localhost" var "play_hosts" type "<class 'list'>" value: "[]"
2020-02-18 16:17:39,215+0100 ERROR ansible failed {
    "ansible_host": "localhost",
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
    "ansible_result": {
        "_ansible_no_log": false,
        "changed": true,
        "cmd": [
            "ip",
            "rule",
            "add",
            "from",
            "192.168.222.1/24",
            "priority",
            "101",
            "table",
            "main"
        ],
        "delta": "0:00:00.002805",
        "end": "2020-02-18 16:17:38.875350",
        "invocation": {
            "module_args": {
                "_raw_params": "ip rule add from 192.168.222.1/24 priority 101 table main",
                "_uses_shell": false,
                "argv": null,
                "chdir": null,
                "creates": null,
                "executable": null,
                "removes": null,
                "stdin": null,
                "stdin_add_newline": true,
                "strip_empty_ends": true,
                "warn": true
            }
        },
        "msg": "non-zero return code",
        "rc": 2,
        "start": "2020-02-18 16:17:38.872545",
        "stderr": "RTNETLINK answers: File exists",
        "stderr_lines": [
            "RTNETLINK answers: File exists"
        ],
        "stdout": "",
        "stdout_lines": []
    },
    "ansible_task": "Add IPv4 outbound route rules",
    "ansible_type": "task",
    "status": "FAILED",
    "task_duration": 0
}

I quickly check the involved tasks/bootstrap_local_vm/01_prepare_routing_rules.yml and it seems that there is actually an intended condition that should skip this task when the rule is already present. This however doesn't seem to work.

The role is run with:

# ansible --version
ansible 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
# rpm -q ansible
ansible-2.9.4-1.el8.noarch

[eslutsky]

its fixed it in ovirt-ansible-hosted-engine-setup-1.1.1-0.1 for 4.4 (on EL8) [0],
@ganto , are you running /usr/sbin/ovirt-hosted-engine-cleanup between your reinstallion attempts?

[0] #294

[ganto]

No, I wasn't. I first didn't realize that there is such a command and the Ansible code made me assume that this case is handled idempotent.

You can close this issue if you think it's fixed. Atm I cannot easily verify the fix because my oVirt 4.4-alpha is up and running.