[RFC] BATS testing setup

[ecdye]

I personally believe that we should rethink the way that we categorize BATS tests (ie. 'destructive' and 'installation') as right now the way they are setup is very blurred when it comes to categorization.

I believe that this happened because of the many changes in the build system and the way that we test.

I think that we should move towards not categorizing the BATS tests but rather treat them as a single unit to test the correctness of all of the functions that can be tested with BATS. I can start implementing this with #937

Thoughts?

[mstormi]

Well I introduced dev- fairly recently only.
In terms of WHAT they test they're mostly the same as the others. They're just meant to be used when you're developing on a feature so they are tested first and allowed to fail.
Then again things changed. Regular BATS tests moved to 2nd stage so we could as well unite those. Read: give up on dev- as a separate Travis stage. We can still allow for naming tests like that. Current ones should be removed, you can add and remove them as needed if they help with development. But we should keep spinning up a separate Docker VM for those. No dev- -> no Travis action.

Having "installation" and "destructive" I believe still makes sense as the latter test are allowed to "destroy" a VM setup so subsequent tests need a new VM while the former are meant to imitate the install process, ideally calling the same routine that's used during install. If you have dependencies (say Java needs to be installed before OH), you need to execute multiple installation- tests one after the other on the same VM.
Note Travis already spins up those 2 VMs in parallel.

When we change this, let's not forget to rework CONTRIBUTING.md

[ecdye]

My thoughts were more that there is only one test that currently uses the moniker "installation" making it a little worthless. Additionally, where do we draw the line? IMO if we follow the currently defined standard, we should be calling the Java tests "installation" as they are part of a standard installation.

I hope you get the point I am trying to make.

[mstormi]

Just a little bit, and mind implementation. It must ensure not to run any other tests after one that was (or could have been) destructive in its impact. Running one destructive test after another destructive one could make the 2nd fail in theory, but as tests should not destroy more than necessary, it usually does not harm because their impact is on separate parts of the system so does not interfere.

I think it makes sense to keep test naming. We can change the implementation to parallelize into VMs for dev- (allowed to fail), unit-, installation- and destructive- (all not allowed to fail).
BTW I forgot about unit- which should be there for every function but as we just started recently it's only for new features from now on

[ecdye]

Also certain functions are unable to be used in a unattended mode (ie. FIND) therefore will never receive bats tests.