Failed TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] #25

mleone87 · 2020-02-05T19:22:37Z

Hi, i'm following the tutorial but the ./dev/seed-deploy.sh phase fails with this error(sorry for italian error in ansible)

the host is a bare metal 4C 32GB Centos7 latest install

TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] ********************************************************************************************************
failed: [seed-hypervisor] (item={u'image': u'https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2', u'capacity': u'50G', u'name': u'seed-root', u'pool': u'default', u'format': u'qcow2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"capacity": "50G", "format": "qcow2", "image": "https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2", "name": "seed-root", "pool": "default"}, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.33.4 closed.\r\n", "stderr_lines": ["Shared connection to 192.168.33.4 closed."], "stdout": "Unexpected error while getting volume info\r\nerrore: impossibile ottenere il vol 'seed-root'\r\nerrore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-root'\r\n", "stdout_lines": ["Unexpected error while getting volume info", "errore: impossibile ottenere il vol 'seed-root'", "errore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-root'"]}
failed: [seed-hypervisor] (item={u'capacity': u'100G', u'name': u'seed-data', u'pool': u'default', u'format': u'qcow2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"capacity": "100G", "format": "qcow2", "name": "seed-data", "pool": "default"}, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.33.4 closed.\r\n", "stderr_lines": ["Shared connection to 192.168.33.4 closed."], "stdout": "Unexpected error while getting volume info\r\nerrore: impossibile ottenere il vol 'seed-data'\r\nerrore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-data'\r\n", "stdout_lines": ["Unexpected error while getting volume info", "errore: impossibile ottenere il vol 'seed-data'", "errore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-data'"]}
failed: [seed-hypervisor] (item={u'capacity': u'389120', u'name': u'seed-configdrive', u'format': u'raw', u'device': u'cdrom', u'image': u'/opt/kayobe/images/seed.iso', u'pool': u'default'}) => {"ansible_loop_var": "item", "changed": false, "item": {"capacity": "389120", "device": "cdrom", "format": "raw", "image": "/opt/kayobe/images/seed.iso", "name": "seed-configdrive", "pool": "default"}, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.33.4 closed.\r\n", "stderr_lines": ["Shared connection to 192.168.33.4 closed."], "stdout": "Unexpected error while getting volume info\r\nerrore: impossibile ottenere il vol 'seed-configdrive'\r\nerrore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-configdrive'\r\n", "stdout_lines": ["Unexpected error while getting volume info", "errore: impossibile ottenere il vol 'seed-configdrive'", "errore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-configdrive'"]}

markgoddard · 2020-02-06T09:56:42Z

Hi @mleone87, thanks for raising this issue. I translated the error message:

error: unable to get the 'seed-root' vol error: storage volume not found: no storage vol with path corresponding to 'seed-root'

I think this may be the cause of the problem:

output=$(virsh vol-info --pool "$POOL" --vol "$NAME" 2>&1)
result=$?
if [[ $result -eq 0 ]]; then
    echo '{"changed": false}'
    exit 0
elif ! echo "$output" | grep 'Storage volume not found' >/dev/null 2>&1; then

That code is in the stackhpc.libvirt-vm role, in the file files/virt_volume.sh. We might need to try something like this:

- output=$(virsh vol-info --pool "$POOL" --vol "$NAME" 2>&1)
+ output=$(LANG=C virsh vol-info --pool "$POOL" --vol "$NAME" 2>&1)

Is that something you could try?

markgoddard · 2020-02-06T10:44:59Z

I've tried experimenting with locales, and it's difficult to change the language of the response because the libvirt daemon process still keeps the original locale. I think we would need to avoid relying on English in stdout.

In the meantime, could you try again with your language set to English?

mleone87 · 2020-02-06T11:24:36Z

setting locale to eng fixes this task fail even if the virsh output is still in italian if you do it manually:

sudo virsh vol-list --pool default
Nome                 Percorso                                
------------------------------------------------------------------------------
seed-configdrive     /var/lib/libvirt/images/seed-configdrive
seed-data            /var/lib/libvirt/images/seed-data       
seed-root            /var/lib/libvirt/images/seed-root

sudo virsh list --all
 Id    Nome                           Stato
----------------------------------------------------
 1     seed                           in esecuzione

(now it fails there)

PLAY [Ensure known hosts are configured] ****************************************************************************************************************

TASK [ssh-known-host : Scan for SSH keys] ***************************************************************************************************************
ok: [seed -> localhost] => (item=192.168.33.5)

TASK [ssh-known-host : Ensure SSH keys are in known hosts] **********************************************************************************************
changed: [seed -> localhost] => (item=[{'stderr_lines': [u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4'], 'ansible_loop_var': u'item', u'end': u'2020-02-06 12:06:51.511766', u'stderr': u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'stdout': u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM=\n192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H\n192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+', u'changed': False, 'ansible_facts': {u'discovered_interpreter_python': u'/usr/bin/python'}, 'failed': False, u'delta': u'0:00:00.180425', u'cmd': [u'ssh-keyscan', u'192.168.33.5'], 'item': u'192.168.33.5', u'rc': 0, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'ssh-keyscan 192.168.33.5', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, u'start': u'2020-02-06 12:06:51.331341'}, u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM='])                                                             
changed: [seed -> localhost] => (item=[{'stderr_lines': [u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4'], 'ansible_loop_var': u'item', u'end': u'2020-02-06 12:06:51.511766', u'stderr': u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'stdout': u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM=\n192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H\n192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+', u'changed': False, 'ansible_facts': {u'discovered_interpreter_python': u'/usr/bin/python'}, 'failed': False, u'delta': u'0:00:00.180425', u'cmd': [u'ssh-keyscan', u'192.168.33.5'], 'item': u'192.168.33.5', u'rc': 0, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'ssh-keyscan 192.168.33.5', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, u'start': u'2020-02-06 12:06:51.331341'}, u'192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H'])                                                                                                                                                   
changed: [seed -> localhost] => (item=[{'stderr_lines': [u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4'], 'ansible_loop_var': u'item', u'end': u'2020-02-06 12:06:51.511766', u'stderr': u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'stdout': u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM=\n192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H\n192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+', u'changed': False, 'ansible_facts': {u'discovered_interpreter_python': u'/usr/bin/python'}, 'failed': False, u'delta': u'0:00:00.180425', u'cmd': [u'ssh-keyscan', u'192.168.33.5'], 'item': u'192.168.33.5', u'rc': 0, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'ssh-keyscan 192.168.33.5', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, u'start': u'2020-02-06 12:06:51.331341'}, u'192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+'])

PLAY RECAP **********************************************************************************************************************************************
seed                       : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


PLAY [Determine whether user bootstrapping is required] *************************************************************************************************

TASK [Check whether the host is accessible via SSH] *****************************************************************************************************
ok: [seed -> localhost]

TASK [Group hosts requiring kayobe user bootstrapping] **************************************************************************************************
ok: [seed]

TASK [Display a message when bootstrapping is required] *************************************************************************************************
ok: [seed] => {
    "msg": "Cannot access host via SSH using Kayobe Ansible user account - attempting bootstrap\n"
}

PLAY [Ensure the Kayobe Ansible user account exists] ****************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************
ok: [seed]

TASK [singleplatform-eng.users : Creating groups] *******************************************************************************************************

TASK [singleplatform-eng.users : Per-user group creation] ***********************************************************************************************
fatal: [seed]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}

But it could be a locale problem too, maybe it's worth to restart from scratch with a full english install?

markgoddard · 2020-02-06T11:38:54Z

I pushed a fix to the stackhpc.libvirt-vm role. It should work in Italian now. The above error looks like an ansible issue. Did you try multiple times?

mleone87 · 2020-02-06T13:52:29Z

yes, but the problem is persistent :)

markgoddard · 2020-02-06T14:43:04Z

You could at least try a reboot before reinstall.

mleone87 · 2020-02-06T16:20:20Z

I can confirm that the problem persist even with a fresh install on a full LANG=en_EN install... Maybe smth changed on kayobe scripts & ansible scripts?

markgoddard · 2020-02-06T16:59:30Z

Hmm. Can you get a simple ansible command to execute via ssh (not using local connection), also with become: true?

mleone87 · 2020-02-06T19:47:34Z

I can passwordless ssh into seed

$ ssh centos@192.168.33.5
Last login: Thu Feb  6 19:37:30 2020 from gateway

I can passwordless sudo seed

[centos@seed ~]$ sudo ls
[centos@seed ~]$

I cant find a way to run ansible, the script ./dev/seed-deploy.sh develop too deep to understand what's going on.

[mario@kayobe kayobe]$ source /home/mario/kayobe-venv/bin/activate
(kayobe-venv) [mario@kayobe kayobe]$ source /home/mario/kayobe
(kayobe-venv) [mario@kayobe kayobe]$ source /home/mario/kayobe/config/src/kayobe-config/kayobe-env
Using Kayobe config from /home/mario/kayobe/config/src/kayobe-config
(kayobe-venv) [mario@kayobe kayobe]$ ansible -i /home/mario/kayobe/config/src/kayobe-config/etc/kayobe/inventory/hosts -m ping seed
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user
configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

[WARNING]: Found both group and host with same name: seed

[WARNING]: Found both group and host with same name: seed-hypervisor

seed | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname seed: Name or service not known",
    "unreachable": true
}

markgoddard · 2020-02-07T10:30:09Z

Ah, that will not work due to the lack of hostname resolution. We have group variables when running kayobe playbooks that set ansible_host appropriately but these won't be used for an ad-hoc command.

Are you able to use IRC? I'm mgoddard on #openstack-kolla.

markgoddard · 2020-02-07T11:11:43Z

I'm running through a-universe-from-nothing on a fresh install of CentOS 7 on bare metal and have passed the above point - my seed is now configured.

There is a 'cheat script' in a-universe-from-nothing: https://github.com/stackhpc/a-universe-from-nothing/blob/stable/train/a-universe-from-nothing.sh. You won't learn as much but it might help remove any copy/paste type errors.

Alternatively, if you just want to try out an all-in-one combined controller/compute node then there is less to go wrong and it is faster: https://docs.openstack.org/kayobe/latest/development/automated.html#overcloud. I would suggest using the stable/train branch for both kayobe and kayobe-config.

mleone87 · 2020-02-07T14:25:36Z

The problem were the default centos firewalld rules, disabling firewalld and using the configure-networking script will work!

markgoddard mentioned this issue Feb 6, 2020

Volume creation script assumes English locale stackhpc/ansible-role-libvirt-vm#51

Closed

mleone87 closed this as completed Feb 7, 2020

ghost mentioned this issue May 15, 2020

Ensure the VM volumes exist #39

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] #25

Failed TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] #25

mleone87 commented Feb 5, 2020 •

edited

markgoddard commented Feb 6, 2020

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020 •

edited

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020 •

edited

markgoddard commented Feb 7, 2020

markgoddard commented Feb 7, 2020

mleone87 commented Feb 7, 2020

Failed TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] #25

Failed TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] #25

Comments

mleone87 commented Feb 5, 2020 • edited

markgoddard commented Feb 6, 2020

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020 • edited

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020

markgoddard commented Feb 6, 2020

mleone87 commented Feb 6, 2020 • edited

markgoddard commented Feb 7, 2020

markgoddard commented Feb 7, 2020

mleone87 commented Feb 7, 2020

mleone87 commented Feb 5, 2020 •

edited

mleone87 commented Feb 6, 2020 •

edited

mleone87 commented Feb 6, 2020 •

edited