Headless conjure up failed at juju_wait

[ryan-peachey-seon]

Report

Thank you for trying conjure-up! Before reporting a bug please make sure you've gone through this checklist:

• Is this problem already documented at https://docs.ubuntu.com/conjure-up/en/troubleshoot#common-spell-problems
• Is conjure-up running inside a virtual machine?
• Is this problem reproducible with sudo snap refresh conjure-up --edge?

Please provide the output of the following commands

which juju
/snap/bin/juju
juju version
2.3-rc2-xenial-amd64

which conjure-up
/snap/bin/conjure-up
conjure-up --version
conjure-up 2.5-alpha1

which lxc
/usr/bin/lxc
/snap/bin/lxc config show
- no such file or directory
/snap/bin/lxc version
- no such file or directory

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

Please attach tarball of ~/.cache/conjure-up:
conjure-up.tar.gz

tar cvzf conjure-up.tar.gz ~/.cache/conjure-up

Sosreport

Please attach a sosreport:

sudo apt install sosreport
sosreport


sosreport (version 3.4)

no valid plugins were enabled

The resulting output file can be attached to this issue.

What Spell was Selected?

canonical-kubernetes

What provider (aws, maas, localhost, etc)?

aws

MAAS Users

Which version of MAAS?

Commands ran

Please outline what commands were run to install and execute conjure-up:

$ aws ec2 run-instances --image-id ami-da05a4a0 --key-name redacted --security-group-ids sg-redacted --instance-type t2.small --subnet-id subnet-redacted
$ ansible-playbook -i '172.16.204.133,' bootstrapUbuntu.yml --private-key ~/.ssh/redacted
--> install python2 (required for ansible) add users + keys, install core tools such as jq, nmap and atop
$ ssh 172.16.204.133
$ sudo snap install juju --classic
$ sudo snap refresh juju --edge
$ sudo snap install conjure-up --classic
$ sudo snap refresh conjure-up --edge
$ export AWS_ACCESS_KEY_ID=<personal access key>
$ export AWS_SECRET_ACCESS_KEY=<personal secret>
$ juju bootstrap aws/us-east-1 pre-test-k8s --config vpc-id=vpc-redacted --config vpc-id-force=true --to "subnet=subnet-redacted" --constraints "root-disk=128G mem=4G" --bootstrap-series xenial
$ juju add-credential aws
--> add kubernetes IAM user credentials
$ conjure-up canonical-kubernetes aws/us-east-1 pre-test-k8s
[info] Summoning canonical-kubernetes to aws
[info] Creating Juju model.
[info] Juju model created.
[info] Running step: 00_process-providertype before-deploy.
[info] Deploying etcd...
[info] Deploying kubernetes-master...
[info] Deploying easyrsa...
[info] Deploying kubernetes-worker...
[info] Deploying flannel...
[info] Deploying kubeapi-load-balancer...
[info] easyrsa: deployed, installing.
[info] flannel: deployed, installing.
[info] kubernetes-master: deployed, installing.
[info] Exposing kubeapi-load-balancer.
[info] kubeapi-load-balancer: deployed, installing.
[info] etcd: deployed, installing.
[info] Setting relation etcd:db <-> flannel:etcd
[info] Exposing kubernetes-worker.
[info] kubernetes-worker: deployed, installing.
[info] Setting relation flannel:cni <-> kubernetes-master:cni
[info] Setting relation kubeapi-load-balancer:apiserver <-> kubernetes-master:kube-api-endpoint
[info] Setting relation easyrsa:client <-> kubernetes-master:certificates
[info] Setting relation flannel:cni <-> kubernetes-worker:cni
[info] Setting relation kubeapi-load-balancer:loadbalancer <-> kubernetes-master:loadbalancer
[info] Setting relation easyrsa:client <-> etcd:certificates
[info] Setting relation kubernetes-master:kube-control <-> kubernetes-worker:kube-control
[info] Setting relation easyrsa:client <-> kubeapi-load-balancer:certificates
[info] Setting relation kubeapi-load-balancer:website <-> kubernetes-worker:kube-api-endpoint
[info] Setting relation etcd:db <-> kubernetes-master:etcd
[info] Setting relation easyrsa:client <-> kubernetes-worker:certificates
[info] Waiting for deployment to settle.
[error] Some applications failed to start successfully.
[warning] Shutting down

Additional Information

[kwmonroe]

Piling on here to say that I've hit the same issue. In my case, I was using conjure-up 2.4.1-20171121.0211 with juju 2.3-rc2+develop-fbb8949, but the results are the same:

[error] Some applications failed to start successfully.

And the root cause in conjure-up.log from both @ryan-peachey-seon and my logs is a failure to find juju_wait. Here's the relevant bits from @ryan-peachey-seon's log:

2017-11-23 21:19:07,301 [INFO] conjure-up/canonical-kubernetes - common.py:56 - Waiting for deployment to settle.
2017-11-23 21:19:07,465 [ERROR] conjure-up/canonical-kubernetes - juju.py:894 - Traceback (most recent call last):
  File "/snap/conjure-up/842/bin/juju-wait", line 7, in <module>
    from juju_wait import wait_cmd
ModuleNotFoundError: No module named 'juju_wait'
2017-11-23 21:19:07,467 [DEBUG] conjure-up/canonical-kubernetes - events.py:53 - Setting Error at conjureup/events.py:148
2017-11-23 21:19:07,467 [ERROR] conjure-up/canonical-kubernetes - events.py:160 - Unhandled exception in <Task finished coro=<DeployController._wait_for_applications() done, defined at /snap/conjure-up/842/lib/python3.6/site-packages/conjureup/controllers/deploy/tui.py:12> exception=DeploymentFailure('Some applications failed to start successfully.',)>
Traceback (most recent call last):
  File "/snap/conjure-up/842/lib/python3.6/site-packages/conjureup/controllers/deploy/tui.py", line 13, in _wait_for_applications
    await common.wait_for_applications(utils.info)
  File "/snap/conjure-up/842/lib/python3.6/site-packages/conjureup/controllers/deploy/common.py", line 59, in wait_for_applications
    await juju.wait_for_deployment()
  File "/snap/conjure-up/842/lib/python3.6/site-packages/conjureup/juju.py", line 896, in wait_for_deployment
    "Some applications failed to start successfully.")
conjureup.errors.DeploymentFailure: Some applications failed to start successfully.

And here's mine:

2017-11-27 13:26:53,270 [DEBUG] conjure-up/hadoop-processing - events.py:53 - Setting DeploymentComplete at conjureup/controllers/deploy/common.py:46
2017-11-27 13:26:53,270 [DEBUG] conjure-up/hadoop-processing - events.py:53 - Received DeploymentComplete at conjureup/controllers/deploy/common.py:50 in task _wait_for_applications at conjureup/controllers/deploy/gui.py:90
2017-11-27 13:26:53,270 [INFO] conjure-up/hadoop-processing - common.py:52 - Waiting for deployment to settle.
2017-11-27 13:26:53,537 [ERROR] conjure-up/hadoop-processing - juju.py:894 - Traceback (most recent call last):
  File "/snap/conjure-up/837/bin/juju-wait", line 7, in <module>
    from juju_wait import wait_cmd
ModuleNotFoundError: No module named 'juju_wait'

[raema]

Also seeing same 'ModuleNotFoundError: No module named 'juju_wait'' error when deploying nova-lxd on localhost.

snap list
Name Version Rev Developer Notes
conjure-up 2.4.1-20171121.0211 837 canonical classic
core 16-2.29.3 3440 canonical core
juju 2.2.6 2739 canonical classic
lxd 2.20 5061 canonical -

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

Tried the conjure up edge channel with the same result.

2017-11-28 00:59:49,276 [DEBUG] conjure-up/openstack-novalxd - events.py:53 - Received DeploymentComplete at conjureup/controllers/deploy/common.py:50 in task _wait_for_applications at conjureup/controllers/deploy/gui.py:90
2017-11-28 00:59:49,276 [INFO] conjure-up/openstack-novalxd - common.py:52 - Waiting for deployment to settle.
2017-11-28 00:59:49,563 [ERROR] conjure-up/openstack-novalxd - juju.py:894 - Traceback (most recent call last):
  File "/snap/conjure-up/837/bin/juju-wait", line 7, in <module>
    from juju_wait import wait_cmd
ModuleNotFoundError: No module named 'juju_wait'
2017-11-28 00:59:49,565 [DEBUG] conjure-up/openstack-novalxd - events.py:53 - Setting Error at conjureup/events.py:148
2017-11-28 00:59:49,566 [ERROR] conjure-up/openstack-novalxd - events.py:160 - Unhandled exception in <Task finished coro=<DeployController._wait_for_applications() done, defined at /snap/conjure-up/837/lib/python3.6/site-packages/conjureup/controllers/deploy/gui.py:89> exception=DeploymentFailure('Some applications failed to start successfully.',)>
Traceback (most recent call last):
  File "/snap/conjure-up/837/lib/python3.6/site-packages/conjureup/controllers/deploy/gui.py", line 90, in _wait_for_applications
    await common.wait_for_applications(app.ui.set_footer)
  File "/snap/conjure-up/837/lib/python3.6/site-packages/conjureup/controllers/deploy/common.py", line 55, in wait_for_applications
    await juju.wait_for_deployment()
  File "/snap/conjure-up/837/lib/python3.6/site-packages/conjureup/juju.py", line 896, in wait_for_deployment
    "Some applications failed to start successfully.")
conjureup.errors.DeploymentFailure: Some applications failed to start successfully.
2017-11-28 00:59:49,578 [DEBUG] conjure-up/openstack-novalxd - __init__.py:21 - Showing dialog for exception: Some applications failed to start successfully.
2017-11-28 01:00:24,802 [DEBUG] conjure-up/openstack-novalxd - events.py:53 - Setting Shutdown at conjureup/events.py:144
2017-11-28 01:00:24,803 [DEBUG] conjure-up/openstack-novalxd - events.py:53 - Received Shutdown at conjureup/events.py:175
2017-11-28 01:00:24,803 [INFO] conjure-up/openstack-novalxd - events.py:179 - Shutting down
2017-11-28 01:00:24,803 [INFO] conjure-up/openstack-novalxd - app_config.py:190 - Storing conjure-up state
2017-11-28 01:00:24,851 [INFO] conjure-up/openstack-novalxd - app_config.py:194 - State saved in model config
2017-11-28 01:00:24,853 [INFO] conjure-up/openstack-novalxd - events.py:189 - Disconnecting model
2017-11-28 01:00:24,859 [INFO] conjure-up/openstack-novalxd - events.py:191 - Disconnected
2017-11-28 01:00:24,860 [DEBUG] conjure-up/openstack-novalxd - events.py:201 - Cancelling pending task: <Task finished coro=<WebSocketCommonProtocol.close_connection() done, defined at /snap/conjure-up/837/lib/python3.6/site-packages/websockets/protocol.py:695> result=None>
2017-11-28 01:00:24,861 [DEBUG] conjure-up/openstack-novalxd - events.py:201 - Cancelling pending task: <Task finished coro=<WebSocketCommonProtocol.transfer_data() done, defined at /snap/conjure-up/837/lib/python3.6/site-packages/websockets/protocol.py:486> result=None>```

[battlemidget]

@raema what if you sudo snap remove juju? conjure-up bundles juju so I'm curious if there is some confusion with snaps having the juju snap installed in addition to our bundled juju binary

[raema]

@battlemidget Thanks for the suggestion! I removed the juju snap and I'm no longer seeing the error

[battlemidget]

/cc @johnsca apparently there is some kind of conflict having our provided juju and the snap juju installed alongside it