feature request: expose test actions to inherited layers

[adam-stokes]

Was thinking maybe having the ability to run simple tox tests on layers that inherit the base class without having to go through the whole amulet song and dance for simple isolated charm tests.

Something like

juju do service/0 tox or juju do service/0 tox -e checkservices, etc.

[johnsca]

I'm not sure I fully understand this. Are you suggesting adding a "smoke test" action to the base layer that other layers could easily register tests for? Can you give some examples of the types of tests you are envisioning? Also, how does tox fit into the picture for a deployed service?

[adam-stokes]

I'm not fully up to speed on how bundletester or amulet plays into all of this yet. I was thinking more smoke tests like I would manually deploy the service and run tox (or a test runner) against the services after they've come up.

[johnsca]

Smoke tests to verify a deployment do seem useful, but I'm not sure that tox is the right approach, as it is very oriented to doing unit tests of Python code.

In the big data charms, we have a few smoke test Juju actions and I could see creating a framework for having a single smoke-test action that various layers can register test cases to instead of each defining their own separate action might well be useful to prevent conflicts and generally encourage the writing of smoke tests.

But then I start to wonder exactly what smoke tests could usefully be added by layers lower than the top-level charm layer? For example, how much could the apache-php layer usefully assert about the vanilla charm without specific knowledge of how the charm should respond and under what conditions? I think I just need more examples about what each layer could test.

[adam-stokes]

Agreed, we can close this for now, I need to actually do more testing of my charms to get a feel for where I'm getting hungup and how to improve it.

[johnsca]

Closing per last comment; re-open as necessary.