Bundletester "passes" and exits cleanly when passed a test filename that does not exist

[ryan-beisner]

Bundletester "passes" and exits cleanly when passed a test filename that does not exist. It essentially bootstrapped then destroyed and exited 0.

I would expect it to fail early in this case.

func27-smoke runtests: commands[0] | bundletester -vl DEBUG -r json -o func-results.json gate-basic-xenial-mitaka --no-destroy
DEBUG:bundletester.utils:Updating JUJU_ENV: "" -> "osci-sv14"
DEBUG:root:Bootstrap environment: osci-sv14
INFO:root:Bootstrapping Juju Environment...
INFO:deployer.env:bootstrapping, this might take a while...
INFO:deployer.env: Bootstrap complete
DEBUG:deployer.env:Connecting to environment...
/var/lib/jenkins/workspace/test_charm_amulet_smoke/asset/barbican/.tox/py27/local/lib/python2.7/site-packages/jujuclient/connector.py:18: RuntimeWarning: This version of Python does not support TLSv1.2. Please use Python 2.7.9+ or 3.4+ instead. Attempting to use TLSv1 - may not work with all versions of Juju.
  'all versions of Juju.', RuntimeWarning)
DEBUG:deployer.env:Connected to environment
DEBUG:deployer.env: Terminating machines forcefully
DEBUG:root:Waiting for applications to be removed...
DEBUG:runner:call ['/var/lib/jenkins/workspace/test_charm_amulet_smoke/asset/barbican/.tox/py27/bin/charm-proof'] (cwd: /tmp/bundletester-DM9iEv/barbican)
DEBUG:runner:I: No icon.svg file.
DEBUG:runner:I: all charms should provide at least one thing
DEBUG:runner:I: config.yaml: option ssl_key has no default value
DEBUG:runner:I: config.yaml: option os-public-hostname has no default value
DEBUG:runner:I: config.yaml: option vip has no default value
DEBUG:runner:I: config.yaml: option ssl_cert has no default value
DEBUG:runner:I: config.yaml: option os-internal-network has no default value
DEBUG:runner:I: config.yaml: option os-admin-network has no default value
DEBUG:runner:I: config.yaml: option ssl_ca has no default value
DEBUG:runner:I: config.yaml: option os-public-network has no default value
DEBUG:runner:Exit Code: 0
DEBUG:bundletester.utils:Updating JUJU_ENV: "osci-sv14" -> ""

[ryan-beisner]

The impact to test automation is that an author can declare named test to run at gate, not include a test file at all, and have a test pass.

[chuckbutler]

Right but theres 3 things currently that bother me about this.

1. Excludes dont appear to work, and we have some localized tests that are getting included when building charms. This means we have to sit through these tests even though they should only be run in specific contexts. Slow tests are better than no tests i guess?

2. Our default base layer template includes 00-setup, which isn't encouraging the right behavior either.

3. I should have the option to say NO TESTS IN THIS CHARM, LOOK AT THE BUNDLE and bundletester should respect that instead of failing me.

[ryan-beisner]

1. Excludes aren't a bundletester bug. That should be addressed where the bug lives.

2. 00-setup should go away altogether imho. Tests should not assume the test user can sudo anything.

3. I believe you do have the option to say no tests in bundletester.