Testing layers

[TheBB]

There's an issue at #110 but I open a new one to generate some discussion.

Currently only core has tests, but the vast majority of spacemacs code exists in layers. Even if you don't explicitly enable any layers, you still get the (huge) spacemacs layer, which is also untested.

I just tried to figure out how a test framework for layers might look like, but it's not as easy as I might have liked.

In order to test layers you have to download packages. This is fine on Travis, but one might like to run tests locally as well. Since you want to be able to test any layer in Spacemacs—even the ones you don't have enabled personally—tests will necessarily need to download and compile packages, possibly even remove ones you have but which the test won't use. This means we have to prevent testing from messing up the developer's .emacs.d.

Some options:

1. We make gentlemen's agreement that layer tests should generally be run on Travis and not locally. We'll still find bugs, but not as smoothly as before.
2. We write a script that runs layer tests in a separate directory pointed to by HOME everywhere, on Travis as well as locally.
3. We write naive layer tests and make it the developer's responsibility to have the layer enabled and installed if they want to run them locally. We write a script to run only on Travis that will run them all in sequence with generated dotfiles, cleaning up elpa each time.

I think I'm leaning towards the third option. That means local testing will be useful and quick but not 100% complete.

Other thoughts:

• The environment variable SPACEMACSDIR can be used to point to different testing dotfiles without having to copy them to ~/.spacemacs, and so we don't have to worry about overwriting that file locally.
• Similar environment variables for the other things we don't want to mess up (elpa and .cache) don't exist, but we could implement those, no? That would make this a bit easier.

[Kethku]

If there could be some guidelines or examples of what a good tested layer would look like I think it could go a long way. My small contributions are definitely not tested with unit tests or automation tests, and I just did manual testing because I did not know what the recommended testing frameworks were if any in emacs. Do you have suggestions for resources to look at?

[robbyoconnor]

👍

[TheBB]

@Devagamster I was hoping to put together a testing framework and some sample tests for a layer, so that other people could see how to do it. I'm certainly not about to test everything by myself. :-)

I just wanted some input on how to put together such a framework, how to make it work both on a developer's computer an on Travis.

[TheBB]

I am procrastinating at work, and the result is #2454.