nova-cloud-controller/0: hook failed

[qgriffith]

Ubuntu 16.04 using MAAS Version 2.0.0+bzr5189-0ubuntu1 (16.04.1) JUJU: 2.0-rc3-0ubuntu116.04.1juju1 conjure: 2.0.1-0201610061938ubuntu16.04.

I get the following in my log when doing a conjure up openstack and using the MAAS option

conjure-up/_unspecified_spell: [ERROR] conjure-up/_unspecified_spell: Failure in deploy done: Deployment errors:
nova-cloud-controller/0: hook failed: "cloud-compute-relation-changed" for nova-compute:cloud-compute
conjure-up/_unspecified_spell: [ERROR] conjure-up/_unspecified_spell: Showing dialog for exception:
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/conjureup/controllers/deploystatus/common.py", line 40, in wait_for_applications
    raise Exception(result['message'])
Exception: Deployment errors:
nova-cloud-controller/0: hook failed: "cloud-compute-relation-changed" for nova-compute:cloud-compute

[mikemccracken]

@qgriffith Thanks for trying this out, sorry it didn't work smoothy. This is an error from the juju charm that deploys nova on the control node. To debug this further, you'll need to look at juju debug output directly.

looking at juju status should show you the 'cloud-compute-relation-changed' error in the nova-cloud-controller application, and may show other errors that might be good clues, such as errors from any of the machines.

juju debug-log --replay -i unit-nova-cloud-controller-0 will show you all the debug log output from the erroring application. Something might be useful there. Feel free to paste the results here if it's not obvious what's going on and we can try to help.

Good luck

[qgriffith]

Thank you for your super quick reply and helpful tips. It appears to be something with DNS maybe

unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed Traceback (most recent call last):
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 1102, in <module>
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     main()
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 1096, in main
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     hooks.execute(sys.argv)
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/charmhelpers/core/hookenv.py", line 715, in execute
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     self._hooks[hook_name]()
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/cloud-compute-relation-changed", line 618, in compute_changed
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     ssh_compute_add(key, rid=rid, unit=unit)
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/nova_cc_utils.py", line 746, in ssh_compute_add
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     hn = get_hostname(private_address)
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/charmhelpers/contrib/network/ip.py", line 465, in get_hostname
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     result = ns_query(rev)
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/charmhelpers/contrib/network/ip.py", line 427, in ns_query
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     answers = dns.resolver.query(address, rtype)
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/usr/lib/python2.7/dist-packages/dns/resolver.py", line 981, in query
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     raise_on_no_answer, source_port)
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed   File "/usr/lib/python2.7/dist-packages/dns/resolver.py", line 910, in query
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed     raise NXDOMAIN
unit-nova-cloud-controller-0: 14:11:24 INFO unit.nova-cloud-controller/0.cloud-compute-relation-changed dns.resolver.NXDOMAIN
unit-nova-cloud-controller-0: 14:11:24 ERROR juju.worker.uniter.operation hook "cloud-compute-relation-changed" failed: exit status 1
unit-nova-cloud-controller-0: 14:11:24 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook

[qgriffith]

there was also this error at the start but it seems to continue past it

unit-nova-cloud-controller-0: 14:03:01 ERROR juju.worker.dependency "metric-collect" manifold worker returned unexpected error: failed to read charm from: /var/lib/juju/agents/unit-nova-cloud-controller-0/charm: stat /var/lib/juju/agents/unit-nova-cloud-controller-0/charm: no such file or directory

[mikemccracken]

This doesn't ring any bells immediately, we're looking into it.

[qgriffith]

I do notice that on the blades MAAS manages DNS is setup like this

nameserver
search maas

In the previous version of MAAS DNS was setup to point to the MAAS server and then forward do the real DNS server. I don't know if that is the issue but something I am looking into.

[battlemidget]

Not sure if this helps but this is my network setup for MAAS

[qgriffith]

I think I have fixed this by removing the DNS server from the subnet MAAS handles. This will now default all MAAS servers to use MAAS as the DNS server, which then forwards out to my real DNS server