New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New TripleO quickstart cheatsheet #42

Open
ccamacho opened this Issue Jan 5, 2018 · 6 comments

Comments

Projects
None yet
5 participants
@ccamacho
Member

ccamacho commented Jan 5, 2018

No description provided.

@pradhanparas

This comment has been minimized.

Show comment
Hide comment
@pradhanparas

pradhanparas Jan 11, 2018

It failed with the following error. Looks like related to the virtual bmc

TASK [setup/overcloud : Define overcloud vms] ************************************************************************************************************************************************************************************************
task path: /home/toor/.quickstart/tripleo-quickstart/roles/libvirt/setup/overcloud/tasks/main.yml:34
Thursday 11 January 2018 16:56:06 +0000 (0:00:05.872) 0:04:16.937 ******
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
failed: [127.0.0.2] (item={u'flavor': u'control', u'virtualbmc_port': 6230, u'name': u'control_0'}) => {"changed": false, "failed": true, "item": {"flavor": "control", "name": "control_0", "virtualbmc_port": 6230}, "msg": "invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm "}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
failed: [127.0.0.2] (item={u'flavor': u'compute', u'virtualbmc_port': 6231, u'name': u'compute_0'}) => {"changed": false, "failed": true, "item": {"flavor": "compute", "name": "compute_0", "virtualbmc_port": 6231}, "msg": "invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm "}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
127.0.0.2 : ok=71 changed=36 unreachable=0 failed=1
localhost : ok=11 changed=5 unreachable=0 failed=0

pradhanparas commented Jan 11, 2018

It failed with the following error. Looks like related to the virtual bmc

TASK [setup/overcloud : Define overcloud vms] ************************************************************************************************************************************************************************************************
task path: /home/toor/.quickstart/tripleo-quickstart/roles/libvirt/setup/overcloud/tasks/main.yml:34
Thursday 11 January 2018 16:56:06 +0000 (0:00:05.872) 0:04:16.937 ******
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
failed: [127.0.0.2] (item={u'flavor': u'control', u'virtualbmc_port': 6230, u'name': u'control_0'}) => {"changed": false, "failed": true, "item": {"flavor": "control", "name": "control_0", "virtualbmc_port": 6230}, "msg": "invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm "}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
failed: [127.0.0.2] (item={u'flavor': u'compute', u'virtualbmc_port': 6231, u'name': u'compute_0'}) => {"changed": false, "failed": true, "item": {"flavor": "compute", "name": "compute_0", "virtualbmc_port": 6231}, "msg": "invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm "}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
127.0.0.2 : ok=71 changed=36 unreachable=0 failed=1
localhost : ok=11 changed=5 unreachable=0 failed=0

@ccamacho

This comment has been minimized.

Show comment
Hide comment
@ccamacho

ccamacho Jan 12, 2018

Member

Seems like a issue with your dev box as the capabilities are not matching for you.

Member

ccamacho commented Jan 12, 2018

Seems like a issue with your dev box as the capabilities are not matching for you.

@dbaxps

This comment has been minimized.

Show comment
Hide comment
@dbaxps

dbaxps Jan 24, 2018

Sample as is works just fine via WKS F27 to 32 GB (4 CORE) VIRTHOST Connection to stack account .
Finally I've got 👎
UNDERCLOUD was 12 GB and 6 VCPUS
NODES each one was 8 GB and 2 VCPUS
Docker containers running Openstack Services might be seen pretty clear on both nodes
F27 VM had been created in overcloud and tested via shuttle connection to VIRTHOST from F27 WKS.
However, looks like 4 node HA deployment to 32 GB VIRTHOST fails both ways 1 and 2

  1. Undercloud set to 8 GB and VCPUS =4 causes overcloud containerized deployment to run for
    1.5 hr CPU UTILIZATION 4% - actually hangs
  2. Keeping 2 Nodes sample memory and VCPUS defaults results crash during
    "Prepare containerized overcloud deployment" phase.
    Seems like 64 GB RAM at least are required on VIRTHOST for minmal 4 Nodes HA deployment.
    If I am wrong about that, please, advise

dbaxps commented Jan 24, 2018

Sample as is works just fine via WKS F27 to 32 GB (4 CORE) VIRTHOST Connection to stack account .
Finally I've got 👎
UNDERCLOUD was 12 GB and 6 VCPUS
NODES each one was 8 GB and 2 VCPUS
Docker containers running Openstack Services might be seen pretty clear on both nodes
F27 VM had been created in overcloud and tested via shuttle connection to VIRTHOST from F27 WKS.
However, looks like 4 node HA deployment to 32 GB VIRTHOST fails both ways 1 and 2

  1. Undercloud set to 8 GB and VCPUS =4 causes overcloud containerized deployment to run for
    1.5 hr CPU UTILIZATION 4% - actually hangs
  2. Keeping 2 Nodes sample memory and VCPUS defaults results crash during
    "Prepare containerized overcloud deployment" phase.
    Seems like 64 GB RAM at least are required on VIRTHOST for minmal 4 Nodes HA deployment.
    If I am wrong about that, please, advise
@dbaxps

This comment has been minimized.

Show comment
Hide comment
@dbaxps

dbaxps Jan 27, 2018

[boris@fedora27workstation ~]$ cat deploy-config2.yaml
**overcloud_nodes:

  • name: control_0
    flavor: control
    virtualbmc_port: 6230
  • name: compute_0
    flavor: compute
    virtualbmc_port: 6231
  • name: compute_1
    flavor: compute
    virtualbmc_port: 6232
    node_count: 3
    containerized_overcloud: true
    delete_docker_cache: true
    enable_pacemaker: true
    run_tempest: false
    extra_args: >-
    --libvirt-type qemu
    --ntp-server pool.ntp.org
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml
    [boris@fedora27workstation ~]$ export CONFIG=~boris/deploy-config2.yaml
    [boris@fedora27workstation ~]$ bash ./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e @$CONFIG $VIRTHOST

[boris@fedora27workstation ~]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.0.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 14:33:07 2018 from gateway


Updating yaml template doesn't change anything


[stack@undercloud ~]$ . stackrc
(undercloud) [stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 6b59a502-b325-40a3-96b6-9b39d9daf11f | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.15 |
| db178309-e9dc-418d-94b2-1cd01a4c99a8 | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.168.24.16 |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
(undercloud) [stack@undercloud ~]$ date
Sat Jan 27 14:48:32 UTC 2018
(undercloud) [stack@undercloud ~]$ nova --version
9.1.0


Original yaml template


[boris@fedora27workstation ~]$ cat deploy-config.yaml
overcloud_nodes:

  • name: control_0
    flavor: control
    virtualbmc_port: 6230
  • name: compute_0
    flavor: compute
    virtualbmc_port: 6231
    node_count: 2
    containerized_overcloud: true
    delete_docker_cache: true
    enable_pacemaker: true
    run_tempest: false
    extra_args: >-
    --libvirt-type qemu
    --ntp-server pool.ntp.org
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml

Still requires *-scale


extra_args: >-
--libvirt-type qemu
--ntp-server pool.ntp.org
--control-scale 1
--compute-scale 2
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml

[boris@fedora27workstation ~]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.0.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 22:22:04 2018 from gateway
[stack@undercloud ~]$ . stackrc
(undercloud) [stack@undercloud ~]$ date
Sat Jan 27 22:22:47 UTC 2018
(undercloud) [stack@undercloud ~]$ nova --version
9.1.0
(undercloud) [stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 383791f4-b830-4f74-9f58-734aa8d8d9d8 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.14 |
| d44a8661-f404-44c0-aa8b-fc4c96367604 | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.168.24.11 |
| b03791d5-0962-435d-8c96-1ecc5232e5a1 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.168.24.13 |
+--------------------------------------+-------------------------+--------+------------+---------
(undercloud) [stack@undercloud ~]$ ssh heat-admin@192.168.24.14
The authenticity of host '192.168.24.14 (192.168.24.14)' can't be established.
ECDSA key fingerprint is SHA256:P06T0ieLWiOZEr/uZN7jrjLr2007SA2LG8uqd6sPAso.
ECDSA key fingerprint is MD5:be:a6:bb:2a🆎89:20:f6:75:f1:b1:40:fa:9f:65:63.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.24.14' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 22:20:42 2018 from 192.168.24.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
[root@overcloud-controller-0 ~]# vi overcloudrc
[root@overcloud-controller-0 ~]# . overcloudrc
(overcloud) [root@overcloud-controller-0 ~]# nova service-list
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 861d48d3-232d-4d85-9a8a-761606ed02b3 | nova-scheduler | overcloud-controller-0.localdomain | internal | enabled | up | 2018-01-27T22:25:39.000000 | - | False |
| a47f9e53-0131-47e8-bfdf-072c00ec3764 | nova-consoleauth | overcloud-controller-0.localdomain | internal | enabled | up | 2018-01-27T22:25:40.000000 | - | False |
| 7c474b20-653b-4b6c-98c0-860705c8523a | nova-conductor | overcloud-controller-0.localdomain | internal | enabled | up | 2018-01-27T22:25:42.000000 | - | False |
| 23590452-8420-410b-9491-cfbbfd5dab29 | nova-compute | overcloud-novacompute-0.localdomain | nova | enabled | up | 2018-01-27T22:25:39.000000 | - | False |
| a1684363-43a1-40ec-9c2d-c6b65946ddc5 | nova-compute | overcloud-novacompute-1.localdomain | nova | enabled | up | 2018-01-27T22:25:40.000000 | - | False |
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
(overcloud) [root@overcloud-controller-0 ~]# neutron agent-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
| 21784560-fb60-44c5-bc55-27ee71972722 | Metadata agent | overcloud-controller-0.localdomain | | :-) | True | neutron-metadata-agent |
| 6da4bf7b-c81b-4383-873d-07a23790bf15 | L3 agent | overcloud-controller-0.localdomain | nova | :-) | True | neutron-l3-agent |
| bfe4f4c8-5e9b-4789-a254-12909cb1a6cb | Open vSwitch agent | overcloud-controller-0.localdomain | | :-) | True | neutron-openvswitch-agent |
| c5bcf4de-7653-4c60-b596-5bb81f8045a0 | DHCP agent | overcloud-controller-0.localdomain | nova | :-) | True | neutron-dhcp-agent |
| d8e08d98-4237-43a4-9f10-771faf5693d9 | Open vSwitch agent | overcloud-novacompute-1.localdomain | | :-) | True | neutron-openvswitch-agent |
| ee4291ce-1c6e-462f-84bf-0f723aaa46c8 | Open vSwitch agent | overcloud-novacompute-0.localdomain | | :-) | True | neutron-openvswitch-agent |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
(overcloud) [root@overcloud-controller-0 ~]# docker ps | grep nova
75a84b0f9a66 192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes nova_metadata
13ed7467787f 192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) nova_api
6907bd221853 192.168.24.1:8787/master/centos-binary-nova-conductor:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_conductor
6c8beb7aba4f 192.168.24.1:8787/master/centos-binary-nova-novncproxy:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_vnc_proxy
622f26c549af 192.168.24.1:8787/master/centos-binary-nova-consoleauth:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_consoleauth
edbba6d88a10 192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes nova_api_cron
b8d46ed5f699 192.168.24.1:8787/master/centos-binary-nova-scheduler:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_scheduler
7f14c696f42b 192.168.24.1:8787/master/centos-binary-nova-placement-api:current-tripleo-rdo "kolla_start" 24 minutes ago Up 24 minutes nova_placement

(overcloud) [root@overcloud-controller-0 ~]# docker ps | grep neutron
e33fac483821 192.168.24.1:8787/master/centos-binary-neutron-openvswitch-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_ovs_agent
3323f6f23ddb 192.168.24.1:8787/master/centos-binary-neutron-l3-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_l3_agent
e617ac4974fe 192.168.24.1:8787/master/centos-binary-neutron-metadata-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_metadata_agent
4809150d5876 192.168.24.1:8787/master/centos-binary-neutron-dhcp-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_dhcp
93ff29c70447 192.168.24.1:8787/master/centos-binary-neutron-server:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes neutron_api

(undercloud) [stack@undercloud ~]$ ssh heat-admin@192.168.24.11
The authenticity of host '192.168.24.11 (192.168.24.11)' can't be established.
ECDSA key fingerprint is SHA256:nKonZHP/0ZbdZ+5OE8f0zzw1TSIjweHHPZVTXIe5WuI.
ECDSA key fingerprint is MD5:df:c1:79:99:3f:8a:44:79:09:a6:11:5c:7e:f7:41:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.24.11' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 22:19:08 2018 from gateway
[heat-admin@overcloud-novacompute-0 ~]$ sudo su -
[root@overcloud-novacompute-0 ~]# docker ps | grep nova
259b20780c35 192.168.24.1:8787/master/centos-binary-nova-compute:current-tripleo-rdo "kolla_start" 33 minutes ago Up 33 minutes nova_migration_target
5fa9eb231710 192.168.24.1:8787/master/centos-binary-nova-compute:current-tripleo-rdo "kolla_start" 33 minutes ago Up 33 minutes (healthy) nova_compute

2888264d6b46 192.168.24.1:8787/master/centos-binary-nova-libvirt:current-tripleo-rdo "kolla_start" 39 minutes ago Up 39 minutes nova_libvirt
7a6422defbe2 192.168.24.1:8787/master/centos-binary-nova-libvirt:current-tripleo-rdo "kolla_start" 39 minutes ago Up 39 minutes nova_virtlogd

dbaxps commented Jan 27, 2018

[boris@fedora27workstation ~]$ cat deploy-config2.yaml
**overcloud_nodes:

  • name: control_0
    flavor: control
    virtualbmc_port: 6230
  • name: compute_0
    flavor: compute
    virtualbmc_port: 6231
  • name: compute_1
    flavor: compute
    virtualbmc_port: 6232
    node_count: 3
    containerized_overcloud: true
    delete_docker_cache: true
    enable_pacemaker: true
    run_tempest: false
    extra_args: >-
    --libvirt-type qemu
    --ntp-server pool.ntp.org
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml
    [boris@fedora27workstation ~]$ export CONFIG=~boris/deploy-config2.yaml
    [boris@fedora27workstation ~]$ bash ./tripleo-quickstart/quickstart.sh --clean --release master --teardown all --tags all -e @$CONFIG $VIRTHOST

[boris@fedora27workstation ~]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.0.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 14:33:07 2018 from gateway


Updating yaml template doesn't change anything


[stack@undercloud ~]$ . stackrc
(undercloud) [stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 6b59a502-b325-40a3-96b6-9b39d9daf11f | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.15 |
| db178309-e9dc-418d-94b2-1cd01a4c99a8 | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.168.24.16 |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
(undercloud) [stack@undercloud ~]$ date
Sat Jan 27 14:48:32 UTC 2018
(undercloud) [stack@undercloud ~]$ nova --version
9.1.0


Original yaml template


[boris@fedora27workstation ~]$ cat deploy-config.yaml
overcloud_nodes:

  • name: control_0
    flavor: control
    virtualbmc_port: 6230
  • name: compute_0
    flavor: compute
    virtualbmc_port: 6231
    node_count: 2
    containerized_overcloud: true
    delete_docker_cache: true
    enable_pacemaker: true
    run_tempest: false
    extra_args: >-
    --libvirt-type qemu
    --ntp-server pool.ntp.org
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml

Still requires *-scale


extra_args: >-
--libvirt-type qemu
--ntp-server pool.ntp.org
--control-scale 1
--compute-scale 2
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml

[boris@fedora27workstation ~]$ ssh -F /home/boris/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added '192.168.0.74' (ECDSA) to the list of known hosts.
Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 22:22:04 2018 from gateway
[stack@undercloud ~]$ . stackrc
(undercloud) [stack@undercloud ~]$ date
Sat Jan 27 22:22:47 UTC 2018
(undercloud) [stack@undercloud ~]$ nova --version
9.1.0
(undercloud) [stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
| 383791f4-b830-4f74-9f58-734aa8d8d9d8 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.14 |
| d44a8661-f404-44c0-aa8b-fc4c96367604 | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.168.24.11 |
| b03791d5-0962-435d-8c96-1ecc5232e5a1 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.168.24.13 |
+--------------------------------------+-------------------------+--------+------------+---------
(undercloud) [stack@undercloud ~]$ ssh heat-admin@192.168.24.14
The authenticity of host '192.168.24.14 (192.168.24.14)' can't be established.
ECDSA key fingerprint is SHA256:P06T0ieLWiOZEr/uZN7jrjLr2007SA2LG8uqd6sPAso.
ECDSA key fingerprint is MD5:be:a6:bb:2a🆎89:20:f6:75:f1:b1:40:fa:9f:65:63.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.24.14' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 22:20:42 2018 from 192.168.24.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
[root@overcloud-controller-0 ~]# vi overcloudrc
[root@overcloud-controller-0 ~]# . overcloudrc
(overcloud) [root@overcloud-controller-0 ~]# nova service-list
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 861d48d3-232d-4d85-9a8a-761606ed02b3 | nova-scheduler | overcloud-controller-0.localdomain | internal | enabled | up | 2018-01-27T22:25:39.000000 | - | False |
| a47f9e53-0131-47e8-bfdf-072c00ec3764 | nova-consoleauth | overcloud-controller-0.localdomain | internal | enabled | up | 2018-01-27T22:25:40.000000 | - | False |
| 7c474b20-653b-4b6c-98c0-860705c8523a | nova-conductor | overcloud-controller-0.localdomain | internal | enabled | up | 2018-01-27T22:25:42.000000 | - | False |
| 23590452-8420-410b-9491-cfbbfd5dab29 | nova-compute | overcloud-novacompute-0.localdomain | nova | enabled | up | 2018-01-27T22:25:39.000000 | - | False |
| a1684363-43a1-40ec-9c2d-c6b65946ddc5 | nova-compute | overcloud-novacompute-1.localdomain | nova | enabled | up | 2018-01-27T22:25:40.000000 | - | False |
+--------------------------------------+------------------+-------------------------------------+----------+---------+-------+----------------------------+-----------------+-------------+
(overcloud) [root@overcloud-controller-0 ~]# neutron agent-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
| 21784560-fb60-44c5-bc55-27ee71972722 | Metadata agent | overcloud-controller-0.localdomain | | :-) | True | neutron-metadata-agent |
| 6da4bf7b-c81b-4383-873d-07a23790bf15 | L3 agent | overcloud-controller-0.localdomain | nova | :-) | True | neutron-l3-agent |
| bfe4f4c8-5e9b-4789-a254-12909cb1a6cb | Open vSwitch agent | overcloud-controller-0.localdomain | | :-) | True | neutron-openvswitch-agent |
| c5bcf4de-7653-4c60-b596-5bb81f8045a0 | DHCP agent | overcloud-controller-0.localdomain | nova | :-) | True | neutron-dhcp-agent |
| d8e08d98-4237-43a4-9f10-771faf5693d9 | Open vSwitch agent | overcloud-novacompute-1.localdomain | | :-) | True | neutron-openvswitch-agent |
| ee4291ce-1c6e-462f-84bf-0f723aaa46c8 | Open vSwitch agent | overcloud-novacompute-0.localdomain | | :-) | True | neutron-openvswitch-agent |
+--------------------------------------+--------------------+-------------------------------------+-------------------+-------+----------------+---------------------------+
(overcloud) [root@overcloud-controller-0 ~]# docker ps | grep nova
75a84b0f9a66 192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes nova_metadata
13ed7467787f 192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) nova_api
6907bd221853 192.168.24.1:8787/master/centos-binary-nova-conductor:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_conductor
6c8beb7aba4f 192.168.24.1:8787/master/centos-binary-nova-novncproxy:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_vnc_proxy
622f26c549af 192.168.24.1:8787/master/centos-binary-nova-consoleauth:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_consoleauth
edbba6d88a10 192.168.24.1:8787/master/centos-binary-nova-api:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes nova_api_cron
b8d46ed5f699 192.168.24.1:8787/master/centos-binary-nova-scheduler:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes (healthy) nova_scheduler
7f14c696f42b 192.168.24.1:8787/master/centos-binary-nova-placement-api:current-tripleo-rdo "kolla_start" 24 minutes ago Up 24 minutes nova_placement

(overcloud) [root@overcloud-controller-0 ~]# docker ps | grep neutron
e33fac483821 192.168.24.1:8787/master/centos-binary-neutron-openvswitch-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_ovs_agent
3323f6f23ddb 192.168.24.1:8787/master/centos-binary-neutron-l3-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_l3_agent
e617ac4974fe 192.168.24.1:8787/master/centos-binary-neutron-metadata-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_metadata_agent
4809150d5876 192.168.24.1:8787/master/centos-binary-neutron-dhcp-agent:current-tripleo-rdo "kolla_start" 17 minutes ago Up 17 minutes (healthy) neutron_dhcp
93ff29c70447 192.168.24.1:8787/master/centos-binary-neutron-server:current-tripleo-rdo "kolla_start" 18 minutes ago Up 18 minutes neutron_api

(undercloud) [stack@undercloud ~]$ ssh heat-admin@192.168.24.11
The authenticity of host '192.168.24.11 (192.168.24.11)' can't be established.
ECDSA key fingerprint is SHA256:nKonZHP/0ZbdZ+5OE8f0zzw1TSIjweHHPZVTXIe5WuI.
ECDSA key fingerprint is MD5:df:c1:79:99:3f:8a:44:79:09:a6:11:5c:7e:f7:41:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.24.11' (ECDSA) to the list of known hosts.
Last login: Sat Jan 27 22:19:08 2018 from gateway
[heat-admin@overcloud-novacompute-0 ~]$ sudo su -
[root@overcloud-novacompute-0 ~]# docker ps | grep nova
259b20780c35 192.168.24.1:8787/master/centos-binary-nova-compute:current-tripleo-rdo "kolla_start" 33 minutes ago Up 33 minutes nova_migration_target
5fa9eb231710 192.168.24.1:8787/master/centos-binary-nova-compute:current-tripleo-rdo "kolla_start" 33 minutes ago Up 33 minutes (healthy) nova_compute

2888264d6b46 192.168.24.1:8787/master/centos-binary-nova-libvirt:current-tripleo-rdo "kolla_start" 39 minutes ago Up 39 minutes nova_libvirt
7a6422defbe2 192.168.24.1:8787/master/centos-binary-nova-libvirt:current-tripleo-rdo "kolla_start" 39 minutes ago Up 39 minutes nova_virtlogd

@verdurin

This comment has been minimized.

Show comment
Hide comment
@verdurin

verdurin Apr 12, 2018

Just followed this and I see the same error I experienced yesterday when following the upstream documentation. There's something wrong with the undercloud VM, in that the KVM process is stuck at 100% and the script fails to pick up its IP:

TASK [setup/undercloud : Start undercloud vm] **************************************************************************************************************************************************************
task path: /home/toor/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:328
Thursday 12 April 2018  16:52:18 +0100 (0:00:00.042)       0:16:51.088 ********
changed: [127.0.0.2] => {"changed": true, "failed": false, "msg": 0}

TASK [setup/undercloud : Get undercloud vm ip address] *****************************************************************************************************************************************************
task path: /home/toor/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:341
Thursday 12 April 2018  16:52:21 +0100 (0:00:02.917)       0:16:54.005 ********
FAILED - RETRYING: Get undercloud vm ip address (20 retries left).
FAILED - RETRYING: Get undercloud vm ip address (19 retries left).
FAILED - RETRYING: Get undercloud vm ip address (18 retries left).
FAILED - RETRYING: Get undercloud vm ip address (17 retries left).
FAILED - RETRYING: Get undercloud vm ip address (16 retries left).
FAILED - RETRYING: Get undercloud vm ip address (15 retries left).
FAILED - RETRYING: Get undercloud vm ip address (14 retries left).
FAILED - RETRYING: Get undercloud vm ip address (13 retries left).
FAILED - RETRYING: Get undercloud vm ip address (12 retries left).
FAILED - RETRYING: Get undercloud vm ip address (11 retries left).
FAILED - RETRYING: Get undercloud vm ip address (10 retries left).
FAILED - RETRYING: Get undercloud vm ip address (9 retries left).
FAILED - RETRYING: Get undercloud vm ip address (8 retries left).
FAILED - RETRYING: Get undercloud vm ip address (7 retries left).
FAILED - RETRYING: Get undercloud vm ip address (6 retries left).
FAILED - RETRYING: Get undercloud vm ip address (5 retries left).
FAILED - RETRYING: Get undercloud vm ip address (4 retries left).
FAILED - RETRYING: Get undercloud vm ip address (3 retries left).
FAILED - RETRYING: Get undercloud vm ip address (2 retries left).
FAILED - RETRYING: Get undercloud vm ip address (1 retries left).
fatal: [127.0.0.2]: FAILED! => {"attempts": 20, "changed": true, "failed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Connection to 127.0.0.2 closed.\r\n", "stdout": "undercloud ip is not available\r\n", "stdout_lines": ["undercloud ip is not available"]}

PLAY RECAP *************************************************************************************************************************************************************************************************
127.0.0.2                  : ok=141  changed=71   unreachable=0    failed=1
localhost                  : ok=11   changed=5    unreachable=0    failed=0

Thursday 12 April 2018  16:56:15 +0100 (0:03:53.335)       0:20:47.341 ********

verdurin commented Apr 12, 2018

Just followed this and I see the same error I experienced yesterday when following the upstream documentation. There's something wrong with the undercloud VM, in that the KVM process is stuck at 100% and the script fails to pick up its IP:

TASK [setup/undercloud : Start undercloud vm] **************************************************************************************************************************************************************
task path: /home/toor/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:328
Thursday 12 April 2018  16:52:18 +0100 (0:00:00.042)       0:16:51.088 ********
changed: [127.0.0.2] => {"changed": true, "failed": false, "msg": 0}

TASK [setup/undercloud : Get undercloud vm ip address] *****************************************************************************************************************************************************
task path: /home/toor/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:341
Thursday 12 April 2018  16:52:21 +0100 (0:00:02.917)       0:16:54.005 ********
FAILED - RETRYING: Get undercloud vm ip address (20 retries left).
FAILED - RETRYING: Get undercloud vm ip address (19 retries left).
FAILED - RETRYING: Get undercloud vm ip address (18 retries left).
FAILED - RETRYING: Get undercloud vm ip address (17 retries left).
FAILED - RETRYING: Get undercloud vm ip address (16 retries left).
FAILED - RETRYING: Get undercloud vm ip address (15 retries left).
FAILED - RETRYING: Get undercloud vm ip address (14 retries left).
FAILED - RETRYING: Get undercloud vm ip address (13 retries left).
FAILED - RETRYING: Get undercloud vm ip address (12 retries left).
FAILED - RETRYING: Get undercloud vm ip address (11 retries left).
FAILED - RETRYING: Get undercloud vm ip address (10 retries left).
FAILED - RETRYING: Get undercloud vm ip address (9 retries left).
FAILED - RETRYING: Get undercloud vm ip address (8 retries left).
FAILED - RETRYING: Get undercloud vm ip address (7 retries left).
FAILED - RETRYING: Get undercloud vm ip address (6 retries left).
FAILED - RETRYING: Get undercloud vm ip address (5 retries left).
FAILED - RETRYING: Get undercloud vm ip address (4 retries left).
FAILED - RETRYING: Get undercloud vm ip address (3 retries left).
FAILED - RETRYING: Get undercloud vm ip address (2 retries left).
FAILED - RETRYING: Get undercloud vm ip address (1 retries left).
fatal: [127.0.0.2]: FAILED! => {"attempts": 20, "changed": true, "failed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Connection to 127.0.0.2 closed.\r\n", "stdout": "undercloud ip is not available\r\n", "stdout_lines": ["undercloud ip is not available"]}

PLAY RECAP *************************************************************************************************************************************************************************************************
127.0.0.2                  : ok=141  changed=71   unreachable=0    failed=1
localhost                  : ok=11   changed=5    unreachable=0    failed=0

Thursday 12 April 2018  16:56:15 +0100 (0:03:53.335)       0:20:47.341 ********
@rathinamtrainers

This comment has been minimized.

Show comment
Hide comment
@rathinamtrainers

rathinamtrainers Jun 7, 2018

I am also facing the same issue. Any updates?

bash quickstart.sh -R master--no-clone --tags all --nodes config/nodes/1ctlr_1comp_1ceph.yml -p quickstart.yml $VIRTHOST

Using the above command, I tried to deploy the tripleo environment on $VIRTHOST (10.160.0.2) from my kickstart machine (10.160.0.3). Everything went until quickstart.sh tried to find the ip address of the undercloud. At that point, playbook failed and errored out.

TASK [setup/undercloud : Get undercloud vm ip address] *************************
task path: /root/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:301
Thursday 07 June 2018 04:06:30 +0000 (0:00:01.577) 0:13:20.528 *********
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (20 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (19 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (18 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (17 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (16 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (15 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (14 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (13 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (12 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (11 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (10 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (9 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (8 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (7 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (6 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (5 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (4 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (3 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (2 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (1 retries left).
fatal: [10.160.0.2]: FAILED! => {"attempts": 20, "changed": true, "failed": true, "rc": 1, "stderr": "Connection to 10.160.0.2 closed.\r\n", "stdout": "undercloud ip is not available\r\n", "stdout_lines": ["undercloud ip is not available"]}

PLAY RECAP *********************************************************************
10.160.0.2 : ok=107 changed=57 unreachable=0 failed=1
localhost : ok=10 changed=5 unreachable=0 failed=0

Thursday 07 June 2018 04:10:34 +0000 (0:04:03.913) 0:17:24.441 *********

In the $VIRTHOST, I observed qemu-kvm is stuck at 100% cpu utilization.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14613 stack 20 0 12.5g 397164 9536 S 99.7 0.4 15:45.04 qemu-kvm

I tried with various releases (queens, pike, ocata). Same issue. I even tried to use old version (2.1.0) of quickstart. Still same issue. Looks like there is some issue with undercloud image.

rathinamtrainers commented Jun 7, 2018

I am also facing the same issue. Any updates?

bash quickstart.sh -R master--no-clone --tags all --nodes config/nodes/1ctlr_1comp_1ceph.yml -p quickstart.yml $VIRTHOST

Using the above command, I tried to deploy the tripleo environment on $VIRTHOST (10.160.0.2) from my kickstart machine (10.160.0.3). Everything went until quickstart.sh tried to find the ip address of the undercloud. At that point, playbook failed and errored out.

TASK [setup/undercloud : Get undercloud vm ip address] *************************
task path: /root/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:301
Thursday 07 June 2018 04:06:30 +0000 (0:00:01.577) 0:13:20.528 *********
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (20 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (19 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (18 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (17 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (16 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (15 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (14 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (13 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (12 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (11 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (10 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (9 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (8 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (7 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (6 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (5 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (4 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (3 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (2 retries left).
FAILED - RETRYING: TASK: setup/undercloud : Get undercloud vm ip address (1 retries left).
fatal: [10.160.0.2]: FAILED! => {"attempts": 20, "changed": true, "failed": true, "rc": 1, "stderr": "Connection to 10.160.0.2 closed.\r\n", "stdout": "undercloud ip is not available\r\n", "stdout_lines": ["undercloud ip is not available"]}

PLAY RECAP *********************************************************************
10.160.0.2 : ok=107 changed=57 unreachable=0 failed=1
localhost : ok=10 changed=5 unreachable=0 failed=0

Thursday 07 June 2018 04:10:34 +0000 (0:04:03.913) 0:17:24.441 *********

In the $VIRTHOST, I observed qemu-kvm is stuck at 100% cpu utilization.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14613 stack 20 0 12.5g 397164 9536 S 99.7 0.4 15:45.04 qemu-kvm

I tried with various releases (queens, pike, ocata). Same issue. I even tried to use old version (2.1.0) of quickstart. Still same issue. Looks like there is some issue with undercloud image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment