Skip to content
This repository has been archived by the owner on Apr 12, 2021. It is now read-only.

conjure-up hangs with multiple 'Missing relation' #428

Closed
ObiterDicta opened this issue Sep 22, 2016 · 3 comments
Closed

conjure-up hangs with multiple 'Missing relation' #428

ObiterDicta opened this issue Sep 22, 2016 · 3 comments
Assignees
@adam-stokes
Labels
bug
Milestone
2.0.1~beta2

Comments

@ObiterDicta
Copy link

Got around to trying conjure-up again. Got latest, fresh install.

$ juju --version
2.0-rc1-xenial-amd64

$ sudo dpkg -l conjure-up
conjure-up                       2.0.1~beta1-0~2016092

Deployment appeared to be more or less OK, except ceph failed to start, which did not seem to be a good sign. Waited over two hours to see if there would be any more progress, then interrupted the install. top showed varying but very high CPU usage by juju, looked like it was maybe in some kind of loop?

Any suggestions welcome. Thanks.

image

After interrupting the install was shown the following. conjure-up log is empty.

KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/bin/conjure-up", line 9, in <module>
    load_entry_point('conjure-up==2.0.1b1', 'console_scripts', 'conjure-up')()
  File "/usr/lib/python3/dist-packages/conjureup/app.py", line 248, in main
    EventLoop.run()
  File "/usr/lib/python3/dist-packages/ubuntui/ev.py", line 83, in run
    cls.loop.run()
  File "/usr/lib/python3/dist-packages/urwid/main_loop.py", line 278, in run
    self._run()
  File "/usr/lib/python3/dist-packages/urwid/main_loop.py", line 376, in _run
    self.event_loop.run()
  File "/usr/lib/python3/dist-packages/urwid/main_loop.py", line 1326, in run
    self._loop.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 345, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1312, in _run_once
    handle._run()
  File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run
    self._callback(*self._args)
  File "/usr/lib/python3/dist-packages/conjureup/ui/views/deploystatus.py", line 31, in _refresh_nodes_on_main_thread
    status = model_status()
  File "/usr/lib/python3/dist-packages/conjureup/juju.py", line 40, in _decorator
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/conjureup/api/models.py", line 42, in model_status
    return juju.CLIENT.Client(request="FullStatus")
  File "/usr/lib/python3/dist-packages/macumba/v2.py", line 118, in _request
    'Params': params})
  File "/usr/lib/python3/dist-packages/macumba/v2.py", line 129, in call
    return self.receive(req_id, timeout)
  File "/usr/lib/python3/dist-packages/macumba/api.py", line 121, in receive
    time.sleep(0.1)
KeyboardInterrupt

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 39, in _python_exit
    t.join()
  File "/usr/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
Exception ignored in: <function ThreadPoolExecutor._adjust_thread_count.<locals>.weakref_cb at 0x7f98edeffae8>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 121, in weakref_cb
  File "/usr/lib/python3.5/queue.py", line 145, in put
  File "/usr/lib/python3.5/threading.py", line 345, in notify
TypeError: 'NoneType' object is not callable

-- get latest
sudo apt-get update
sudo apt-get upgrade

-- clean out old config
sudo rm -r /home/<user>/.cache/conjure-up
sudo rm -r /home/<user>/.cache/conjure-up-spells/
sudo rm -r /home/<user>/.local/share/juju

-- clean install
sudo apt-get purge juju conjure-up
sudo apt-get install juju conjure-up

conjure-up openstack
-- accept all defaults

--wait 2+ hours

$ juju status
MODEL    CONTROLLER  CLOUD/REGION         VERSION
default  hippo       localhost/localhost  2.0-rc1

APP                    VERSION  STATUS   SCALE  CHARM                  STORE       REV  OS      NOTES
ceph-mon                        blocked    0/3  ceph-mon               jujucharms    3  ubuntu  
ceph-osd                        blocked    0/3  ceph-osd               jujucharms    3  ubuntu  
ceph-radosgw                    blocked    0/1  ceph-radosgw           jujucharms    5  ubuntu  
glance                          blocked    0/1  glance                 jujucharms    2  ubuntu  
keystone                        active       1  keystone               jujucharms    2  ubuntu  
lxd                             waiting      0  lxd                    jujucharms    2  ubuntu  
mysql                           active       1  percona-cluster        jujucharms    2  ubuntu  
neutron-api                     active       1  neutron-api            jujucharms    3  ubuntu  
neutron-gateway                 active       1  neutron-gateway        jujucharms    3  ubuntu  
neutron-openvswitch             waiting      0  neutron-openvswitch    jujucharms    3  ubuntu  
nova-cloud-controller           blocked    0/1  nova-cloud-controller  jujucharms    6  ubuntu  
nova-compute                    blocked    0/1  nova-compute           jujucharms    3  ubuntu  
ntp                             waiting      0  ntp                    jujucharms    0  ubuntu  
openstack-dashboard             blocked    0/1  openstack-dashboard    jujucharms    2  ubuntu  exposed
rabbitmq-server                 active       1  rabbitmq-server        jujucharms    5  ubuntu  

UNIT                     WORKLOAD  AGENT   MACHINE  PUBLIC-ADDRESS  PORTS     MESSAGE
ceph-mon/0               blocked   failed  0        10.19.57.244              Insufficient peer units to bootstrap cluster (require 3)
ceph-mon/1               blocked   failed  1        10.19.57.9                Insufficient peer units to bootstrap cluster (require 3)
ceph-mon/2               blocked   failed  2        10.19.57.77               Insufficient peer units to bootstrap cluster (require 3)
ceph-osd/0               blocked   failed  3        10.19.57.89               Missing relation: monitor
ceph-osd/1               blocked   failed  4        10.19.57.208              Missing relation: monitor
ceph-osd/2               blocked   failed  5        10.19.57.123              Missing relation: monitor
ceph-radosgw/0           blocked   failed  6        10.19.57.50               Missing relations: mon
glance/0                 blocked   failed  7        10.19.57.178              Missing relations: identity, database
keystone/0               active    idle    8        10.19.57.88     5000/tcp  Unit is ready
mysql/0                  active    idle    9        10.19.57.19               Unit is ready
neutron-api/0            active    idle    10       10.19.57.186    9696/tcp  Unit is ready
neutron-gateway/0        active    idle    11       10.19.57.216              Unit is ready
nova-cloud-controller/0  blocked   failed  12       10.19.57.97     8774/tcp  Missing relations: messaging, image, compute, identity, database
nova-compute/0           blocked   failed  13       10.19.57.139              Missing relations: messaging, image
openstack-dashboard/0    blocked   failed  14       10.19.57.159              Missing relations: identity
rabbitmq-server/0        active    idle    15       10.19.57.116    5672/tcp  Unit is ready

MACHINE  STATE    DNS           INS-ID          SERIES  AZ
0        started  10.19.57.244  juju-a207fe-0   xenial  
1        started  10.19.57.9    juju-a207fe-1   xenial  
2        started  10.19.57.77   juju-a207fe-2   xenial  
3        started  10.19.57.89   juju-a207fe-3   xenial  
4        started  10.19.57.208  juju-a207fe-4   xenial  
5        started  10.19.57.123  juju-a207fe-5   xenial  
6        started  10.19.57.50   juju-a207fe-6   xenial  
7        started  10.19.57.178  juju-a207fe-7   xenial  
8        started  10.19.57.88   juju-a207fe-8   xenial  
9        started  10.19.57.19   juju-a207fe-9   xenial  
10       started  10.19.57.186  juju-a207fe-10  xenial  
11       started  10.19.57.216  juju-a207fe-11  xenial  
12       started  10.19.57.97   juju-a207fe-12  xenial  
13       started  10.19.57.139  juju-a207fe-13  xenial  
14       started  10.19.57.159  juju-a207fe-14  xenial  
15       started  10.19.57.116  juju-a207fe-15  xenial  

RELATION                 PROVIDES               CONSUMES               TYPE
mon                      ceph-mon               ceph-mon               peer
mon                      ceph-mon               ceph-osd               regular
mon                      ceph-mon               ceph-radosgw           regular
ceph                     ceph-mon               glance                 regular
ceph                     ceph-mon               nova-compute           regular
cluster                  ceph-radosgw           ceph-radosgw           peer
identity-service         ceph-radosgw           keystone               regular
cluster                  glance                 glance                 peer
identity-service         glance                 keystone               regular
shared-db                glance                 mysql                  regular
image-service            glance                 nova-cloud-controller  regular
image-service            glance                 nova-compute           regular
amqp                     glance                 rabbitmq-server        regular
cluster                  keystone               keystone               peer
shared-db                keystone               mysql                  regular
identity-service         keystone               neutron-api            regular
identity-service         keystone               nova-cloud-controller  regular
identity-service         keystone               openstack-dashboard    regular
lxd-migration            lxd                    lxd                    peer
lxd                      lxd                    nova-compute           regular
cluster                  mysql                  mysql                  peer
shared-db                mysql                  neutron-api            regular
shared-db                mysql                  nova-cloud-controller  regular
cluster                  neutron-api            neutron-api            peer
neutron-plugin-api       neutron-api            neutron-gateway        regular
neutron-plugin-api       neutron-api            neutron-openvswitch    regular
neutron-api              neutron-api            nova-cloud-controller  regular
amqp                     neutron-api            rabbitmq-server        regular
cluster                  neutron-gateway        neutron-gateway        peer
quantum-network-service  neutron-gateway        nova-cloud-controller  regular
amqp                     neutron-gateway        rabbitmq-server        regular
neutron-plugin           neutron-openvswitch    nova-compute           regular
amqp                     neutron-openvswitch    rabbitmq-server        regular
cluster                  nova-cloud-controller  nova-cloud-controller  peer
cloud-compute            nova-cloud-controller  nova-compute           regular
amqp                     nova-cloud-controller  rabbitmq-server        regular
lxd                      nova-compute           lxd                    subordinate
neutron-plugin           nova-compute           neutron-openvswitch    subordinate
compute-peer             nova-compute           nova-compute           peer
amqp                     nova-compute           rabbitmq-server        regular
ntp-peers                ntp                    ntp                    peer
cluster                  openstack-dashboard    openstack-dashboard    peer
cluster                  rabbitmq-server        rabbitmq-server        peer

cat /home/<user>/.cache/conjure-up/openstack-novalxd.log 
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
openstack-novalxd: [DEBUG] 00_deploy-done:5 - Running deploy-done for OpenStack installation.
@adam-stokes adam-stokes self-assigned this Sep 22, 2016
@adam-stokes adam-stokes added this to the 2.0.1~beta2 milestone Sep 22, 2016
@adam-stokes
Copy link
Contributor

Thanks, I just started running through tests with Juju 2.0 rc1 ill see if I can reproduce.

@adam-stokes
Copy link
Contributor

I just ran through an install on Juju 2.0 rc1 and couldnt reproduce this problem:

.cache/conjure-up/conjure-up.log: http://paste.ubuntu.com/23214298/
juju status: http://paste.ubuntu.com/23214297/
conjure-up 2.0.1~beta1-0~201609212201~ubuntu16.04.1
juju 2.0-rc1-0ubuntu1~16.04.1~juju1

@adam-stokes
Copy link
Contributor

We've done several deploys now without issue. Could you please retry with the latest from ppa:conjure-up/daily-git and re-open if problem persists?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@adam-stokes adam-stokes

Labels
bug
Projects
None yet
Milestone
2.0.1~beta2
Development

No branches or pull requests
2 participants
@adam-stokes @ObiterDicta