/bundletester

handle reset timeouts better #45

Open
tvansteenburgh opened this Issue May 10, 2016 · 6 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@tvansteenburgh @Yrrsinn
@tvansteenburgh
Member

tvansteenburgh commented May 10, 2016 

ubuntu@xenial:/charms/trusty/quobyte-client$ /home/ubuntu/.local/bin/bundletester -l DEBUG -F -v
DEBUG:bundletester.utils:Updating JUJU_ENV: "" -> "amazon"
DEBUG:root:Bootstrap environment: amazon
DEBUG:deployer.env:Connecting to environment...
DEBUG:deployer.env:Connected to environment
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/usr/bin/charm-proof'] (cwd: /tmp/bundletester-9fmFGc/quobyte-client)
DEBUG:runner:W: Includes template README.ex file
DEBUG:runner:W: README.ex includes boilerplate: Describe the intended usage of this charm and anything unique about how this charm relates to others here.
DEBUG:runner:W: README.ex includes boilerplate: This README will be displayed in the Charm Store, it should be either Markdown or RST. Ideal READMEs include instructions on how to use the charm, expected usage, and charm features that your audience might be interested in. For an example of a well written README check out Hadoop: http://jujucharms.com/charms/precise/hadoop
DEBUG:runner:W: README.ex includes boilerplate: Use this as a Markdown reference if you need help with the formatting of this README: http://askubuntu.com/editing-help
DEBUG:runner:W: README.ex includes boilerplate: This charm provides [service](http://example.com). Add a description here of what the service itself actually does.
DEBUG:runner:W: README.ex includes boilerplate: Also remember to check the [icon guidelines](https://jujucharms.com/docs/stable/authors-charm-icon) so that your charm looks good in the Juju GUI.
DEBUG:runner:W: README.ex includes boilerplate: Step by step instructions on using the charm:
DEBUG:runner:W: README.ex includes boilerplate: and so on. If you're providing a web service or something that the end user needs to go to, tell them here, especially if you're deploying a service that might listen to a non-default port.
DEBUG:runner:W: README.ex includes boilerplate: You can then browse to http://ip-address to configure the service.
DEBUG:runner:W: README.ex includes boilerplate: If the charm has any recommendations for running at scale, outline them in examples here. For example if you have a memcached relation that improves performance, mention it here.
DEBUG:runner:W: README.ex includes boilerplate: This not only helps users but gives people a place to start if they want to help you add features to your charm.
DEBUG:runner:W: README.ex includes boilerplate: The configuration options will be listed on the charm store, however If you're making assumptions or opinionated decisions in the charm (like setting a default administrator password), you should detail that here so the user knows how to change it immediately, etc.
DEBUG:runner:W: README.ex includes boilerplate: Though this will be listed in the charm store itself don't assume a user will know that, so include that information here:
DEBUG:runner:W: README.ex includes boilerplate: - Upstream mailing list or contact information
DEBUG:runner:W: README.ex includes boilerplate: - Feel free to add things if it's useful for users
DEBUG:runner:I: all charms should provide at least one thing
DEBUG:runner:Exit Code: 100
quobyte-client
    charm-proof                                                            FAIL
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/usr/bin/make', '-s', 'lint'] (cwd: /tmp/bundletester-9fmFGc/quobyte-client)
DEBUG:runner:/bin/sh: 1: flake8: not found
DEBUG:runner:Makefile:7: recipe for target 'lint' failed
DEBUG:runner:make: *** [lint] Error 127
DEBUG:runner:Exit Code: 2
    make lint                                                              FAIL
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/tmp/bundletester-9fmFGc/quobyte-client/tests/00-setup'] (cwd: /tmp/bundletester-9fmFGc/quobyte-client)
DEBUG:runner:gpg: keyring `/tmp/tmpze38tcsw/secring.gpg' created
DEBUG:runner:gpg: keyring `/tmp/tmpze38tcsw/pubring.gpg' created
DEBUG:runner:gpg: requesting key C8068B11 from hkp server keyserver.ubuntu.com
DEBUG:runner:gpg: /tmp/tmpze38tcsw/trustdb.gpg: trustdb created
DEBUG:runner:gpg: key C8068B11: public key "Launchpad Ensemble PPA" imported
DEBUG:runner:gpg: Total number processed: 1
DEBUG:runner:gpg:               imported: 1  (RSA: 1)
DEBUG:runner:OK
DEBUG:runner:Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
DEBUG:runner:Get:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
DEBUG:runner:Hit:3 http://security.ubuntu.com/ubuntu xenial-security InRelease
DEBUG:runner:Hit:4 http://ppa.launchpad.net/juju/devel/ubuntu xenial InRelease
DEBUG:runner:Hit:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
DEBUG:runner:Get:6 http://ppa.launchpad.net/juju/stable/ubuntu xenial InRelease [17.5 kB]
DEBUG:runner:Get:7 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main amd64 Packages [6,240 B]
DEBUG:runner:Get:8 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main Translation-en [2,728 B]
DEBUG:runner:Fetched 121 kB in 1s (116 kB/s)
DEBUG:runner:Reading package lists...
DEBUG:runner:Reading package lists...
DEBUG:runner:Building dependency tree...
DEBUG:runner:Reading state information...
DEBUG:runner:python-requests is already the newest version (2.9.1-3).
DEBUG:runner:python-requests set to manually installed.
DEBUG:runner:The following additional packages will be installed:
DEBUG:runner:  libpython3-all-dev python-amulet python3-all python3-all-dev python3-amulet
DEBUG:runner:  python3-libcharmstore python3-path python3-theblues
DEBUG:runner:The following NEW packages will be installed:
DEBUG:runner:  amulet libpython3-all-dev python-amulet python3-all python3-all-dev
DEBUG:runner:  python3-amulet python3-libcharmstore python3-path python3-theblues
DEBUG:runner:0 upgraded, 9 newly installed, 0 to remove and 7 not upgraded.
DEBUG:runner:Need to get 76.2 kB of archives.
DEBUG:runner:After this operation, 410 kB of additional disk space will be used.
DEBUG:runner:Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libpython3-all-dev amd64 3.5.1-3 [996 B]
DEBUG:runner:Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-all amd64 3.5.1-3 [970 B]
DEBUG:runner:Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-all-dev amd64 3.5.1-3 [990 B]
DEBUG:runner:Get:4 http://archive.ubuntu.com/ubuntu xenial/universe amd64 python3-theblues all 0.2.0-0ubuntu1 [7,398 B]
DEBUG:runner:Get:5 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main amd64 python3-libcharmstore all 0.0.3-1ubuntu4~ubuntu16.04.1~ppa1 [4,276 B]
DEBUG:runner:Get:6 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main amd64 python3-path all 8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1 [24.8 kB]
DEBUG:runner:Get:7 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main amd64 python3-amulet all 1.14.4-0ubuntu4~ubuntu16.04.1~ppa2 [17.9 kB]
DEBUG:runner:Get:8 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main amd64 python-amulet all 1.14.4-0ubuntu4~ubuntu16.04.1~ppa2 [17.8 kB]
DEBUG:runner:Get:9 http://ppa.launchpad.net/juju/stable/ubuntu xenial/main amd64 amulet all 1.14.4-0ubuntu4~ubuntu16.04.1~ppa2 [1,010 B]
DEBUG:runner:Fetched 76.2 kB in 1s (68.0 kB/s)
                                              DEBUG:runner:Selecting previously unselected package libpython3-all-dev:amd64.
                                                                                                                            DEBUG:runner:(Reading (Reading database ... 92296 files and directories currently installed.)
                                                                       DEBUG:runner:Preparing to unpack .../libpython3-all-dev_3.5.1-3_amd64.deb ...
  DEBUG:runner:Unpacking libpython3-all-dev:amd64 (3.5.1-3) ...
                                                               DEBUG:runner:Selecting previously unselected package python3-all.
                                                                                                                                DEBUG:runner:Preparing to unpack .../python3-all_3.5.1-3_amd64.deb ...
                                                    DEBUG:runner:Unpacking python3-all (3.5.1-3) ...
                                                                                                    DEBUG:runner:Selecting previously unselected package python3-all-dev.
                       DEBUG:runner:Preparing to unpack .../python3-all-dev_3.5.1-3_amd64.deb ...
                                                                                                 DEBUG:runner:Unpacking python3-all-dev (3.5.1-3) ...
   DEBUG:runner:Selecting previously unselected package python3-theblues.
                                                                         DEBUG:runner:Preparing to unpack .../python3-theblues_0.2.0-0ubuntu1_all.deb ...
       DEBUG:runner:Unpacking python3-theblues (0.2.0-0ubuntu1) ...
                                                                   DEBUG:runner:Selecting previously unselected package python3-libcharmstore.
                                                                                                                                              DEBUG:runner:Preparing to unpack .../python3-libcharmstore_0.0.3-1ubuntu4~ubuntu16.04.1~ppa1_all.deb ...
                                                                                                    DEBUG:runner:Unpacking python3-libcharmstore (0.0.3-1ubuntu4~ubuntu16.04.1~ppa1) ...
                                      DEBUG:runner:Selecting previously unselected package python3-path.
                                                                                                        DEBUG:runner:Preparing to unpack .../python3-path_8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1_all.deb ...
                                                          DEBUG:runner:Unpacking python3-path (8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1) ...
                                                                                                                                          DEBUG:runner:Selecting previously unselected package python3-amulet.
                                                            DEBUG:runner:Preparing to unpack .../python3-amulet_1.14.4-0ubuntu4~ubuntu16.04.1~ppa2_all.deb ...
            DEBUG:runner:Unpacking python3-amulet (1.14.4-0ubuntu4~ubuntu16.04.1~ppa2) ...
                                                                                          DEBUG:runner:Selecting previously unselected package python-amulet.
           DEBUG:runner:Preparing to unpack .../python-amulet_1.14.4-0ubuntu4~ubuntu16.04.1~ppa2_all.deb ...
                                                                                                            DEBUG:runner:Unpacking python-amulet (1.14.4-0ubuntu4~ubuntu16.04.1~ppa2) ...
                                       DEBUG:runner:Selecting previously unselected package amulet.
                                                                                                   DEBUG:runner:Preparing to unpack .../amulet_1.14.4-0ubuntu4~ubuntu16.04.1~ppa2_all.deb ...
                                           DEBUG:runner:Unpacking amulet (1.14.4-0ubuntu4~ubuntu16.04.1~ppa2) ...
                                                                                                                 DEBUG:runner:Setting up libpython3-all-dev:amd64 (3.5.1-3) ...
                             DEBUG:runner:Setting up python3-all (3.5.1-3) ...
                                                                              DEBUG:runner:Setting up python3-all-dev (3.5.1-3) ...
                                                                                                                                   DEBUG:runner:Setting up python3-theblues (0.2.0-0ubuntu1) ...
                                              DEBUG:runner:Setting up python3-libcharmstore (0.0.3-1ubuntu4~ubuntu16.04.1~ppa1) ...
                                                                                                                                   DEBUG:runner:Setting up python3-path (8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1) ...
                                                                  DEBUG:runner:Setting up python3-amulet (1.14.4-0ubuntu4~ubuntu16.04.1~ppa2) ...
                                                                                                                                                 DEBUG:runner:Setting up python-amulet (1.14.4-0ubuntu4~ubuntu16.04.1~ppa2) ...
                                                                             DEBUG:runner:Setting up amulet (1.14.4-0ubuntu4~ubuntu16.04.1~ppa2) ...
  DEBUG:runner:Exit Code: 0
    00-setup                                                               PASS
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/tmp/bundletester-9fmFGc/quobyte-client/tests/10-deploy'] (cwd: /tmp/bundletester-9fmFGc/quobyte-client)
DEBUG:runner:2016-05-10 18:33:47 Starting deployment of amazon
DEBUG:runner:2016-05-10 18:33:48 Deploying services...
DEBUG:runner:2016-05-10 18:33:48  Deploying service quobyte-client using /tmp/charm0pylglbq/trusty/quobyte-client
DEBUG:runner:2016-05-10 18:39:21 Adding relations...
DEBUG:runner:2016-05-10 18:39:22 Deployment complete in 334.74 seconds
DEBUG:runner:Exit Code: 0
    10-deploy                                                              PASS
DEBUG:deployer.env: Destroying service quobyte-client
DEBUG:deployer.env:  No unit errors found.
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:deployer.env:  Terminating machine 3
INFO:deployer.env:  Waiting for machine termination
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/tmp/bundletester-9fmFGc/quobyte-client/tests/20-relation-with-registry'] (cwd: /tmp/bundletester-9fmFGc/quobyte-client)
DEBUG:runner:2016-05-10 18:43:39 Starting deployment of amazon
DEBUG:runner:2016-05-10 18:43:40 Deploying services...
DEBUG:runner:2016-05-10 18:43:40  Deploying service quobyte-client using /tmp/charmpjchivjm/trusty/quobyte-client
DEBUG:runner:2016-05-10 18:43:55  Deploying service quobyte-registry using cs:~3-bruno/trusty/quobyte-registry-4
DEBUG:runner:Timeout occurred, printing juju status...environment: amazon
DEBUG:runner:machines:
DEBUG:runner:  "0":
DEBUG:runner:    agent-state: started
DEBUG:runner:    agent-version: 1.25.5
DEBUG:runner:    dns-name: 54.85.89.39
DEBUG:runner:    instance-id: i-a9e7c234
DEBUG:runner:    instance-state: running
DEBUG:runner:    series: xenial
DEBUG:runner:    hardware: arch=amd64 cpu-cores=1 cpu-power=300 mem=3840M root-disk=8192M availability-zone=us-east-1b
DEBUG:runner:    state-server-member-status: has-vote
DEBUG:runner:  "4":
DEBUG:runner:    agent-state: started
DEBUG:runner:    agent-version: 1.25.5
DEBUG:runner:    dns-name: 52.87.253.99
DEBUG:runner:    instance-id: i-d80a5d5f
DEBUG:runner:    instance-state: running
DEBUG:runner:    series: trusty
DEBUG:runner:    hardware: arch=amd64 cpu-cores=1 cpu-power=300 mem=3840M root-disk=8192M availability-zone=us-east-1c
DEBUG:runner:  "5":
DEBUG:runner:    agent-state: error
DEBUG:runner:    agent-state-info: 'cannot run instances: Your quota allows for 0 more running
DEBUG:runner:      instance(s). You requested at least 1 (InstanceLimitExceeded)'
DEBUG:runner:    instance-id: pending
DEBUG:runner:    series: trusty
DEBUG:runner:  "6":
DEBUG:runner:    agent-state: error
DEBUG:runner:    agent-state-info: 'cannot run instances: Your quota allows for 0 more running
DEBUG:runner:      instance(s). You requested at least 1 (InstanceLimitExceeded)'
DEBUG:runner:    instance-id: pending
DEBUG:runner:    series: trusty
DEBUG:runner:  "7":
DEBUG:runner:    agent-state: error
DEBUG:runner:    agent-state-info: 'cannot run instances: Your quota allows for 0 more running
DEBUG:runner:      instance(s). You requested at least 1 (InstanceLimitExceeded)'
DEBUG:runner:    instance-id: pending
DEBUG:runner:    series: trusty
DEBUG:runner:services:
DEBUG:runner:  quobyte-client:
DEBUG:runner:    charm: local:trusty/quobyte-client-4
DEBUG:runner:    exposed: false
DEBUG:runner:    service-status:
DEBUG:runner:      current: waiting
DEBUG:runner:      message: no quobyte-registry-relation given, waiting for registry
DEBUG:runner:      since: 10 May 2016 18:50:48Z
DEBUG:runner:    units:
DEBUG:runner:      quobyte-client/3:
DEBUG:runner:        workload-status:
DEBUG:runner:          current: waiting
DEBUG:runner:          message: no quobyte-registry-relation given, waiting for registry
DEBUG:runner:          since: 10 May 2016 18:50:48Z
DEBUG:runner:        agent-status:
DEBUG:runner:          current: idle
DEBUG:runner:          since: 10 May 2016 18:55:51Z
DEBUG:runner:          version: 1.25.5
DEBUG:runner:        agent-state: started
DEBUG:runner:        agent-version: 1.25.5
DEBUG:runner:        machine: "4"
DEBUG:runner:        public-address: 52.87.253.99
DEBUG:runner:  quobyte-registry:
DEBUG:runner:    charm: cs:~3-bruno/trusty/quobyte-registry-4
DEBUG:runner:    exposed: false
DEBUG:runner:    service-status:
DEBUG:runner:      current: unknown
DEBUG:runner:      message: Waiting for agent initialization to finish
DEBUG:runner:      since: 10 May 2016 18:43:59Z
DEBUG:runner:    relations:
DEBUG:runner:      quobyte-registry-replica:
DEBUG:runner:      - quobyte-registry
DEBUG:runner:    units:
DEBUG:runner:      quobyte-registry/0:
DEBUG:runner:        workload-status:
DEBUG:runner:          current: unknown
DEBUG:runner:          message: Waiting for agent initialization to finish
DEBUG:runner:          since: 10 May 2016 18:43:59Z
DEBUG:runner:        agent-status:
DEBUG:runner:          current: allocating
DEBUG:runner:          since: 10 May 2016 18:43:59Z
DEBUG:runner:        agent-state: pending
DEBUG:runner:        machine: "5"
DEBUG:runner:      quobyte-registry/1:
DEBUG:runner:        workload-status:
DEBUG:runner:          current: unknown
DEBUG:runner:          message: Waiting for agent initialization to finish
DEBUG:runner:          since: 10 May 2016 18:44:10Z
DEBUG:runner:        agent-status:
DEBUG:runner:          current: allocating
DEBUG:runner:          since: 10 May 2016 18:44:10Z
DEBUG:runner:        agent-state: pending
DEBUG:runner:        machine: "6"
DEBUG:runner:      quobyte-registry/2:
DEBUG:runner:        workload-status:
DEBUG:runner:          current: unknown
DEBUG:runner:          message: Waiting for agent initialization to finish
DEBUG:runner:          since: 10 May 2016 18:44:20Z
DEBUG:runner:        agent-status:
DEBUG:runner:          current: allocating
DEBUG:runner:          since: 10 May 2016 18:44:20Z
DEBUG:runner:        agent-state: pending
DEBUG:runner:        machine: "7"
DEBUG:runner:Environment wasn't stood up in time
DEBUG:runner:Exit Code: 1
    20-relation-with-registry                                              ERROR
DEBUG:deployer.env: Destroying service quobyte-client
DEBUG:deployer.env: Destroying service quobyte-registry
DEBUG:deployer.env:  No unit errors found.
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:deployer.env:  Terminating machine 4
DEBUG:deployer.env:  Terminating machine 5
DEBUG:deployer.env:  Terminating machine 6
DEBUG:deployer.env:  Terminating machine 7
INFO:deployer.env:  Waiting for machine termination
DEBUG:root:Waiting for services to be removed...
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
ERROR:runner:Timeout exceeded. Failed to destroy all services  in 60 seconds.
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python2.7/site-packages/bundletester/runner.py", line 173, in _run_test
    self.builder.reset()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/bundletester/builder.py", line 136, in reset
    ' in %s seconds.' % timeout)
RuntimeError: Timeout exceeded. Failed to destroy all services  in 60 seconds.
DEBUG:bundletester.utils:Updating JUJU_ENV: "amazon" -> ""
Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/bundletester", line 11, in <module>
    sys.exit(entrypoint())
  File "/home/ubuntu/.local/lib/python2.7/site-packages/bundletester/tester.py", line 149, in entrypoint
    status = main()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/bundletester/tester.py", line 141, in main
    [report.emit(result) for result in run()]
  File "/home/ubuntu/.local/lib/python2.7/site-packages/bundletester/reporter.py", line 145, in emit
    cmd = message.test
  File "/home/ubuntu/.local/lib/python2.7/site-packages/bundletester/reporter.py", line 14, in __getattr__
    return self[k]
KeyError: 'test'
ubuntu@xenial:/charms/trusty/quobyte-client$
@Yrrsinn

Yrrsinn commented Jun 10, 2016

This uncaught KeyError exception is still around:

ubuntu@bruno:~/trusty$ cd quobyte-api/
ubuntu@bruno:~/trusty/quobyte-api$ bundletester -l DEBUG -F -v
/usr/local/lib/python2.7/dist-packages/jujuclient.py:53: RuntimeWarning: This version of Python does not support TLSv1.2. Please use Python 2.7.9+ or 3.4+ instead. Attempting to use TLSv1 - may not work with all versions of Juju.
  'all versions of Juju.', RuntimeWarning)
DEBUG:bundletester.utils:Updating JUJU_ENV: "" -> "amazon"
DEBUG:root:Bootstrap environment: amazon
INFO:root:Bootstrapping Juju Environment...
INFO:deployer.env:bootstrapping, this might take a while...
INFO:deployer.env: Bootstrap complete
DEBUG:deployer.env:Connecting to environment...
DEBUG:deployer.env:Connected to environment
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/usr/local/bin/charm-proof'] (cwd: /tmp/bundletester-5FdkdC/quobyte-api)
DEBUG:runner:I: all charms should provide at least one thing
DEBUG:runner:Exit Code: 0
quobyte-api
    charm-proof                                                            PASS
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/usr/bin/make', '-s', 'lint'] (cwd: /tmp/bundletester-5FdkdC/quobyte-api)
DEBUG:runner:flake8 is OK
DEBUG:runner:I: all charms should provide at least one thing
DEBUG:runner:charm-proof is OK
DEBUG:runner:Exit Code: 0
    make lint                                                              PASS
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/tmp/bundletester-5FdkdC/quobyte-api/tests/00-setup.sh'] (cwd: /tmp/bundletester-5FdkdC/quobyte-api)
DEBUG:runner:gpg: keyring `/tmp/tmphc5ht27w/secring.gpg' created
DEBUG:runner:gpg: keyring `/tmp/tmphc5ht27w/pubring.gpg' created
DEBUG:runner:gpg: requesting key C8068B11 from hkp server keyserver.ubuntu.com
DEBUG:runner:gpgkeys: key A2EB2DEC0BD7519B7B38BE38376A290EC8068B11 can't be retrieved
DEBUG:runner:gpg: no valid OpenPGP data found.
DEBUG:runner:gpg: Total number processed: 0
DEBUG:runner:Hit http://ppa.launchpad.net trusty InRelease
DEBUG:runner:Ign http://archive.ubuntu.com trusty InRelease
DEBUG:runner:Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
DEBUG:runner:Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
DEBUG:runner:Hit http://archive.ubuntu.com trusty Release.gpg
DEBUG:runner:Hit http://archive.ubuntu.com trusty Release
DEBUG:runner:Hit http://ppa.launchpad.net trusty/main amd64 Packages
DEBUG:runner:Get:3 http://archive.ubuntu.com trusty-updates/main Sources [348 kB]
DEBUG:runner:Get:4 http://archive.ubuntu.com trusty-updates/restricted Sources [5217 B]
DEBUG:runner:Get:5 http://archive.ubuntu.com trusty-updates/universe Sources [196 kB]
DEBUG:runner:Get:6 http://archive.ubuntu.com trusty-updates/main amd64 Packages [979 kB]
DEBUG:runner:Get:7 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [23.5 kB]
DEBUG:runner:Get:8 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [467 kB]
DEBUG:runner:Get:9 http://archive.ubuntu.com trusty-security/main Sources [148 kB]
DEBUG:runner:Get:10 http://archive.ubuntu.com trusty-security/restricted Sources [3920 B]
DEBUG:runner:Get:11 http://archive.ubuntu.com trusty-security/universe Sources [43.2 kB]
DEBUG:runner:Get:12 http://archive.ubuntu.com trusty-security/main amd64 Packages [618 kB]
DEBUG:runner:Get:13 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [20.2 kB]
DEBUG:runner:Get:14 http://archive.ubuntu.com trusty-security/universe amd64 Packages [169 kB]
DEBUG:runner:Hit http://archive.ubuntu.com trusty/main Sources
DEBUG:runner:Hit http://archive.ubuntu.com trusty/restricted Sources
DEBUG:runner:Hit http://archive.ubuntu.com trusty/universe Sources
DEBUG:runner:Hit http://archive.ubuntu.com trusty/main amd64 Packages
DEBUG:runner:Hit http://archive.ubuntu.com trusty/restricted amd64 Packages
DEBUG:runner:Hit http://archive.ubuntu.com trusty/universe amd64 Packages
DEBUG:runner:Fetched 3153 kB in 2s (1053 kB/s)
DEBUG:runner:Reading package lists...
DEBUG:runner:Reading package lists...
DEBUG:runner:Building dependency tree...
DEBUG:runner:Reading state information...
DEBUG:runner:python-requests is already the newest version.
DEBUG:runner:python-requests set to manually installed.
DEBUG:runner:The following extra packages will be installed:
DEBUG:runner:  libpython-all-dev libpython3-all-dev libpython3-dev libpython3.4
DEBUG:runner:  libpython3.4-dev python-all python-all-dev python-amulet python3-all
DEBUG:runner:  python3-all-dev python3-amulet python3-chardet python3-dev
DEBUG:runner:  python3-jujubundlelib python3-libcharmstore python3-path python3-requests
DEBUG:runner:  python3-setuptools python3-six python3-theblues python3-urllib3 python3-yaml
DEBUG:runner:  python3.4-dev
DEBUG:runner:Suggested packages:
DEBUG:runner:  python-setuptools-doc
DEBUG:runner:The following NEW packages will be installed:
DEBUG:runner:  amulet libpython-all-dev libpython3-all-dev libpython3-dev libpython3.4
DEBUG:runner:  libpython3.4-dev python-all python-all-dev python-amulet python3-all
DEBUG:runner:  python3-all-dev python3-amulet python3-chardet python3-dev
DEBUG:runner:  python3-jujubundlelib python3-libcharmstore python3-path python3-requests
DEBUG:runner:  python3-setuptools python3-six python3-theblues python3-urllib3 python3-yaml
DEBUG:runner:  python3.4-dev
DEBUG:runner:0 upgraded, 24 newly installed, 0 to remove and 13 not upgraded.
DEBUG:runner:Need to get 20.0 MB of archives.
DEBUG:runner:After this operation, 37.9 MB of additional disk space will be used.
DEBUG:runner:Get:1 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpython3.4 amd64 3.4.3-1ubuntu1~14.04.3 [1308 kB]
DEBUG:runner:Get:2 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-yaml amd64 3.11-2build1~ubuntu14.04.1~ppa1 [93.0 kB]
DEBUG:runner:Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpython3.4-dev amd64 3.4.3-1ubuntu1~14.04.3 [17.8 MB]
DEBUG:runner:Get:4 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-setuptools all 20.1.1-1~ubuntu14.04.1~ppa1 [89.4 kB]
DEBUG:runner:Get:5 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-jujubundlelib all 0.4.1-1~ubuntu14.04.1~ppa1 [25.2 kB]
DEBUG:runner:Get:6 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-theblues all 0.2.0-0ubuntu1~ubuntu14.04.1~ppa1 [7412 B]
DEBUG:runner:Get:7 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-libcharmstore all 0.0.3-1ubuntu4~ubuntu14.04.1~ppa1 [4276 B]
DEBUG:runner:Get:8 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-path all 8.1.2+dfsg-1ubuntu1~ubuntu14.04.1~ppa1 [24.9 kB]
DEBUG:runner:Get:9 http://archive.ubuntu.com/ubuntu/ trusty/main libpython3-dev amd64 3.4.0-0ubuntu2 [7002 B]
DEBUG:runner:Get:10 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python3-amulet all 1.15.0-0ubuntu1~ubuntu14.04.1~ppa2 [18.8 kB]
DEBUG:runner:Get:11 http://archive.ubuntu.com/ubuntu/ trusty/main libpython3-all-dev amd64 3.4.0-0ubuntu2 [996 B]
DEBUG:runner:Get:12 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main python-amulet all 1.15.0-0ubuntu1~ubuntu14.04.1~ppa2 [18.7 kB]
DEBUG:runner:Get:13 http://archive.ubuntu.com/ubuntu/ trusty/main python3-all amd64 3.4.0-0ubuntu2 [978 B]
DEBUG:runner:Get:14 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amulet all 1.15.0-0ubuntu1~ubuntu14.04.1~ppa2 [1014 B]
DEBUG:runner:Get:15 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python3.4-dev amd64 3.4.3-1ubuntu1~14.04.3 [419 kB]
DEBUG:runner:Get:16 http://archive.ubuntu.com/ubuntu/ trusty/main python3-dev amd64 3.4.0-0ubuntu2 [1192 B]
DEBUG:runner:Get:17 http://archive.ubuntu.com/ubuntu/ trusty/main python3-all-dev amd64 3.4.0-0ubuntu2 [1000 B]
DEBUG:runner:Get:18 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python3-chardet all 2.2.1-2~ubuntu1 [96.5 kB]
DEBUG:runner:Get:19 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python3-six all 1.5.2-1ubuntu1 [8310 B]
DEBUG:runner:Get:20 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python3-urllib3 all 1.7.1-1ubuntu4 [39.6 kB]
DEBUG:runner:Get:21 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python3-requests all 2.2.1-1ubuntu0.3 [43.0 kB]
DEBUG:runner:Get:22 http://archive.ubuntu.com/ubuntu/ trusty/main python-all amd64 2.7.5-5ubuntu3 [984 B]
DEBUG:runner:Get:23 http://archive.ubuntu.com/ubuntu/ trusty/main libpython-all-dev amd64 2.7.5-5ubuntu3 [996 B]
DEBUG:runner:Get:24 http://archive.ubuntu.com/ubuntu/ trusty/main python-all-dev amd64 2.7.5-5ubuntu3 [1008 B]
DEBUG:runner:Fetched 20.0 MB in 2s (8270 kB/s)
DEBUG:runner:Selecting previously unselected package libpython3.4:amd64.
DEBUG:runner:(Reading database ... 24609 files and directories currently installed.)
DEBUG:runner:Preparing to unpack .../libpython3.4_3.4.3-1ubuntu1~14.04.3_amd64.deb ...
DEBUG:runner:Unpacking libpython3.4:amd64 (3.4.3-1ubuntu1~14.04.3) ...
DEBUG:runner:Selecting previously unselected package libpython3.4-dev:amd64.
DEBUG:runner:Preparing to unpack .../libpython3.4-dev_3.4.3-1ubuntu1~14.04.3_amd64.deb ...
DEBUG:runner:Unpacking libpython3.4-dev:amd64 (3.4.3-1ubuntu1~14.04.3) ...
DEBUG:runner:Selecting previously unselected package libpython3-dev:amd64.
DEBUG:runner:Preparing to unpack .../libpython3-dev_3.4.0-0ubuntu2_amd64.deb ...
DEBUG:runner:Unpacking libpython3-dev:amd64 (3.4.0-0ubuntu2) ...
DEBUG:runner:Selecting previously unselected package libpython3-all-dev:amd64.
DEBUG:runner:Preparing to unpack .../libpython3-all-dev_3.4.0-0ubuntu2_amd64.deb ...
DEBUG:runner:Unpacking libpython3-all-dev:amd64 (3.4.0-0ubuntu2) ...
DEBUG:runner:Selecting previously unselected package python3-all.
DEBUG:runner:Preparing to unpack .../python3-all_3.4.0-0ubuntu2_amd64.deb ...
DEBUG:runner:Unpacking python3-all (3.4.0-0ubuntu2) ...
DEBUG:runner:Selecting previously unselected package python3.4-dev.
DEBUG:runner:Preparing to unpack .../python3.4-dev_3.4.3-1ubuntu1~14.04.3_amd64.deb ...
DEBUG:runner:Unpacking python3.4-dev (3.4.3-1ubuntu1~14.04.3) ...
DEBUG:runner:Selecting previously unselected package python3-dev.
DEBUG:runner:Preparing to unpack .../python3-dev_3.4.0-0ubuntu2_amd64.deb ...
DEBUG:runner:Unpacking python3-dev (3.4.0-0ubuntu2) ...
DEBUG:runner:Selecting previously unselected package python3-all-dev.
DEBUG:runner:Preparing to unpack .../python3-all-dev_3.4.0-0ubuntu2_amd64.deb ...
DEBUG:runner:Unpacking python3-all-dev (3.4.0-0ubuntu2) ...
DEBUG:runner:Selecting previously unselected package python3-yaml.
DEBUG:runner:Preparing to unpack .../python3-yaml_3.11-2build1~ubuntu14.04.1~ppa1_amd64.deb ...
DEBUG:runner:Unpacking python3-yaml (3.11-2build1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Selecting previously unselected package python3-chardet.
DEBUG:runner:Preparing to unpack .../python3-chardet_2.2.1-2~ubuntu1_all.deb ...
DEBUG:runner:Unpacking python3-chardet (2.2.1-2~ubuntu1) ...
DEBUG:runner:Selecting previously unselected package python3-six.
DEBUG:runner:Preparing to unpack .../python3-six_1.5.2-1ubuntu1_all.deb ...
DEBUG:runner:Unpacking python3-six (1.5.2-1ubuntu1) ...
DEBUG:runner:Selecting previously unselected package python3-urllib3.
DEBUG:runner:Preparing to unpack .../python3-urllib3_1.7.1-1ubuntu4_all.deb ...
DEBUG:runner:Unpacking python3-urllib3 (1.7.1-1ubuntu4) ...
DEBUG:runner:Selecting previously unselected package python3-requests.
DEBUG:runner:Preparing to unpack .../python3-requests_2.2.1-1ubuntu0.3_all.deb ...
DEBUG:runner:Unpacking python3-requests (2.2.1-1ubuntu0.3) ...
DEBUG:runner:Selecting previously unselected package python3-setuptools.
DEBUG:runner:Preparing to unpack .../python3-setuptools_20.1.1-1~ubuntu14.04.1~ppa1_all.deb ...
DEBUG:runner:Unpacking python3-setuptools (20.1.1-1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Selecting previously unselected package python3-jujubundlelib.
DEBUG:runner:Preparing to unpack .../python3-jujubundlelib_0.4.1-1~ubuntu14.04.1~ppa1_all.deb ...
DEBUG:runner:Unpacking python3-jujubundlelib (0.4.1-1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Selecting previously unselected package python3-theblues.
DEBUG:runner:Preparing to unpack .../python3-theblues_0.2.0-0ubuntu1~ubuntu14.04.1~ppa1_all.deb ...
DEBUG:runner:Unpacking python3-theblues (0.2.0-0ubuntu1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Selecting previously unselected package python3-libcharmstore.
DEBUG:runner:Preparing to unpack .../python3-libcharmstore_0.0.3-1ubuntu4~ubuntu14.04.1~ppa1_all.deb ...
DEBUG:runner:Unpacking python3-libcharmstore (0.0.3-1ubuntu4~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Selecting previously unselected package python3-path.
DEBUG:runner:Preparing to unpack .../python3-path_8.1.2+dfsg-1ubuntu1~ubuntu14.04.1~ppa1_all.deb ...
DEBUG:runner:Unpacking python3-path (8.1.2+dfsg-1ubuntu1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Selecting previously unselected package python3-amulet.
DEBUG:runner:Preparing to unpack .../python3-amulet_1.15.0-0ubuntu1~ubuntu14.04.1~ppa2_all.deb ...
DEBUG:runner:Unpacking python3-amulet (1.15.0-0ubuntu1~ubuntu14.04.1~ppa2) ...
DEBUG:runner:Selecting previously unselected package python-all.
DEBUG:runner:Preparing to unpack .../python-all_2.7.5-5ubuntu3_amd64.deb ...
DEBUG:runner:Unpacking python-all (2.7.5-5ubuntu3) ...
DEBUG:runner:Selecting previously unselected package libpython-all-dev:amd64.
DEBUG:runner:Preparing to unpack .../libpython-all-dev_2.7.5-5ubuntu3_amd64.deb ...
DEBUG:runner:Unpacking libpython-all-dev:amd64 (2.7.5-5ubuntu3) ...
DEBUG:runner:Selecting previously unselected package python-all-dev.
DEBUG:runner:Preparing to unpack .../python-all-dev_2.7.5-5ubuntu3_amd64.deb ...
DEBUG:runner:Unpacking python-all-dev (2.7.5-5ubuntu3) ...
DEBUG:runner:Selecting previously unselected package python-amulet.
DEBUG:runner:Preparing to unpack .../python-amulet_1.15.0-0ubuntu1~ubuntu14.04.1~ppa2_all.deb ...
DEBUG:runner:Unpacking python-amulet (1.15.0-0ubuntu1~ubuntu14.04.1~ppa2) ...
DEBUG:runner:Selecting previously unselected package amulet.
DEBUG:runner:Preparing to unpack .../amulet_1.15.0-0ubuntu1~ubuntu14.04.1~ppa2_all.deb ...
DEBUG:runner:Unpacking amulet (1.15.0-0ubuntu1~ubuntu14.04.1~ppa2) ...
DEBUG:runner:Setting up libpython3.4:amd64 (3.4.3-1ubuntu1~14.04.3) ...
DEBUG:runner:Setting up libpython3.4-dev:amd64 (3.4.3-1ubuntu1~14.04.3) ...
DEBUG:runner:Setting up libpython3-dev:amd64 (3.4.0-0ubuntu2) ...
DEBUG:runner:Setting up libpython3-all-dev:amd64 (3.4.0-0ubuntu2) ...
DEBUG:runner:Setting up python3-all (3.4.0-0ubuntu2) ...
DEBUG:runner:Setting up python3.4-dev (3.4.3-1ubuntu1~14.04.3) ...
DEBUG:runner:Setting up python3-dev (3.4.0-0ubuntu2) ...
DEBUG:runner:Setting up python3-all-dev (3.4.0-0ubuntu2) ...
DEBUG:runner:Setting up python3-yaml (3.11-2build1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Setting up python3-chardet (2.2.1-2~ubuntu1) ...
DEBUG:runner:Setting up python3-six (1.5.2-1ubuntu1) ...
DEBUG:runner:Setting up python3-urllib3 (1.7.1-1ubuntu4) ...
DEBUG:runner:Setting up python3-requests (2.2.1-1ubuntu0.3) ...
DEBUG:runner:Setting up python3-setuptools (20.1.1-1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Setting up python3-jujubundlelib (0.4.1-1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Setting up python3-theblues (0.2.0-0ubuntu1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Setting up python3-libcharmstore (0.0.3-1ubuntu4~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Setting up python3-path (8.1.2+dfsg-1ubuntu1~ubuntu14.04.1~ppa1) ...
DEBUG:runner:Setting up python3-amulet (1.15.0-0ubuntu1~ubuntu14.04.1~ppa2) ...
DEBUG:runner:Setting up python-all (2.7.5-5ubuntu3) ...
DEBUG:runner:Setting up libpython-all-dev:amd64 (2.7.5-5ubuntu3) ...
DEBUG:runner:Setting up python-all-dev (2.7.5-5ubuntu3) ...
DEBUG:runner:Setting up python-amulet (1.15.0-0ubuntu1~ubuntu14.04.1~ppa2) ...
DEBUG:runner:Setting up amulet (1.15.0-0ubuntu1~ubuntu14.04.1~ppa2) ...
DEBUG:runner:Processing triggers for libc-bin (2.19-0ubuntu6.7) ...
DEBUG:runner:Exit Code: 0
    00-setup.sh                                                            PASS
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/tmp/bundletester-5FdkdC/quobyte-api/tests/10-deploy'] (cwd: /tmp/bundletester-5FdkdC/quobyte-api)
DEBUG:runner:/usr/local/lib/python2.7/dist-packages/jujuclient.py:53: RuntimeWarning: This version of Python does not support TLSv1.2. Please use Python 2.7.9+ or 3.4+ instead. Attempting to use TLSv1 - may not work with all versions of Juju.
DEBUG:runner:  'all versions of Juju.', RuntimeWarning)
DEBUG:runner:2016-06-10 13:44:10 Starting deployment of amazon
DEBUG:runner:2016-06-10 13:44:11 Deploying services...
DEBUG:runner:2016-06-10 13:44:11  Deploying service quobyte-api using /tmp/charmnbhwoclc/trusty/quobyte-api
DEBUG:runner:2016-06-10 13:47:14 Adding relations...
DEBUG:runner:2016-06-10 13:47:14 Deployment complete in 184.41 seconds
DEBUG:runner:Exit Code: 0
    10-deploy                                                              PASS
DEBUG:deployer.env: Destroying service quobyte-api
DEBUG:deployer.env:  No unit errors found.
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:deployer.env:  Terminating machine 1
INFO:deployer.env:  Waiting for machine termination
DEBUG:root:Waiting for services to be removed...
DEBUG:runner:call ['/tmp/bundletester-5FdkdC/quobyte-api/tests/20-relation-with-registry'] (cwd: /tmp/bundletester-5FdkdC/quobyte-api)
DEBUG:runner:/usr/local/lib/python2.7/dist-packages/jujuclient.py:53: RuntimeWarning: This version of Python does not support TLSv1.2. Please use Python 2.7.9+ or 3.4+ instead. Attempting to use TLSv1 - may not work with all versions of Juju.
DEBUG:runner:  'all versions of Juju.', RuntimeWarning)
DEBUG:runner:2016-06-10 13:49:45 Starting deployment of amazon
DEBUG:runner:2016-06-10 13:49:45 Deploying services...
DEBUG:runner:2016-06-10 13:49:46  Deploying service quobyte-api using /tmp/charmp1vqqjhv/trusty/quobyte-api
DEBUG:runner:2016-06-10 13:49:56  Deploying service quobyte-registry using cs:~3-bruno/trusty/quobyte-registry-4
DEBUG:runner:2016-06-10 13:53:20 Adding relations...
DEBUG:runner:2016-06-10 13:53:21  Adding relation quobyte-registry:quobyte-registry <-> quobyte-api:quobyte-registry
DEBUG:runner:2016-06-10 13:54:21 Deployment complete in 276.79 seconds
DEBUG:runner:Exit Code: 0
    20-relation-with-registry                                              PASS
DEBUG:deployer.env: Destroying service quobyte-api
DEBUG:deployer.env: Destroying service quobyte-registry
DEBUG:deployer.env:  No unit errors found.
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:deployer.env:  Terminating machine 2
DEBUG:deployer.env:  Terminating machine 3
DEBUG:deployer.env:  Terminating machine 4
DEBUG:deployer.env:  Terminating machine 5
INFO:deployer.env:  Waiting for machine termination
DEBUG:root:Waiting for services to be removed...
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
DEBUG:root: Remaining services: [u'quobyte-registry']
ERROR:runner:Timeout exceeded. Failed to destroy all services  in 60 seconds.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/bundletester/runner.py", line 174, in _run_test
    self.builder.reset()
  File "/usr/local/lib/python2.7/dist-packages/bundletester/builder.py", line 133, in reset
    ' in %s seconds.' % timeout)
RuntimeError: Timeout exceeded. Failed to destroy all services  in 60 seconds.
DEBUG:bundletester.utils:Updating JUJU_ENV: "amazon" -> ""
Traceback (most recent call last):
  File "/usr/local/bin/bundletester", line 9, in <module>
    load_entry_point('bundletester==0.7.0', 'console_scripts', 'bundletester')()
  File "/usr/local/lib/python2.7/dist-packages/bundletester/tester.py", line 149, in entrypoint
    status = main()
  File "/usr/local/lib/python2.7/dist-packages/bundletester/tester.py", line 141, in main
    [report.emit(result) for result in run()]
  File "/usr/local/lib/python2.7/dist-packages/bundletester/reporter.py", line 145, in emit
    cmd = message.test
  File "/usr/local/lib/python2.7/dist-packages/bundletester/reporter.py", line 14, in __getattr__
    return self[k]
KeyError: 'test'
ubuntu@bruno:~/trusty/quobyte-api$
  • What is causing this reset timeouts?
  • Why is bundletester running the Makefile? (afaik the definition of a charm does not say anything about makefiles)
@tvansteenburgh
Member

tvansteenburgh commented Jun 10, 2016 •
Edited 1 time

Bundletester resets the env between each test. If it can't complete the reset in 60s it gives up. I'm not sure why it's taking so long to remove the quobyte-registry service. I'm also not sure what the best way to handle it is. One idea is to allow the timeout value to be configured in tests.yaml, so you can increase it. Also need to handle the timeout more gracefully, but that's what this bug is already about. :)

Bundletester tests charms and bundles. Part of the testing is detecting and executing Makefile targets. You can turn this off by passing -s, or by setting makefile: [] in tests.yaml (see https://github.com/juju-solutions/bundletester#testsyaml)
@tvansteenburgh
Member

tvansteenburgh commented Jun 10, 2016

Should've also mentioned: if you don't want to reset the env between each test you can add reset: false to tests.yaml.
@Yrrsinn

Yrrsinn commented Jun 10, 2016 •
Edited 1 time

Ah, thanks for pointing out this details of bundletester. Resetting between tests is fine, I will set-up a better integration for my makefile.

Actually the tests for this charms does not always fails, sometimes all tests passes. I guess the error is less related to a time-out, but to some problem with Juju Storage.

From the output of juju status and juju debug-log --replay I learned that 2 out of 3 machines with a quobyte-registry failed to die properly in the test-run filed above.

From the debug-log:

[...]
unit-quobyte-registry-1[8247]: 2016-06-10 13:57:17 DEBUG unit.quobyte-registry/1.juju-log server.go:268 quobyte-registry-replica:0: quobyte-registry-replica-relation-broken called
machine-3[8191]: 2016-06-10 13:57:17 ERROR juju.worker.storageprovisioner common.go:116 failed to set status: cannot set invalid status "detached"
machine-4[8192]: 2016-06-10 13:57:17 ERROR juju.worker.storageprovisioner common.go:116 failed to set status: cannot set invalid status "detached"
[...]
machine-3[8191]: 2016-06-10 13:57:22 ERROR juju.worker runner.go:223 exited "storageprovisioner-machine": getting life of volume-3-0 attached to machine-3: permission denied
machine-4[8192]: 2016-06-10 13:57:22 ERROR juju.worker runner.go:223 exited "storageprovisioner-machine": getting life of volume-4-1 attached to machine-4: permission denied
machine-5[8192]: 2016-06-10 13:57:22 ERROR juju.worker.storageprovisioner common.go:116 failed to set status: cannot set invalid status "detached"
[...]
machine-0: 2016-06-10 13:57:18 WARNING juju.state cleanup.go:106 cleanup failed: unit has storage attachments

machine-3, 4 and 5 are the hosts quobyte-registry has been deployed to. machine-3 and 4 failed to die. machine-5 is gone.

(Edit: additional note:

  • the storage-detaching hook is only called for machine-5, according to juju debug-log --replay )

juju-status.txt
juju-debug-log.txt
@tvansteenburgh
Member

tvansteenburgh commented Jun 10, 2016

@Yrrsinn You may want to submit this info as a bug to juju-core: https://bugs.launchpad.net/juju-core
@Yrrsinn

Yrrsinn commented Jun 14, 2016

https://bugs.launchpad.net/juju-core/+bug/1592366
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment