proposal: switch to a tests framework

[jarondl]

Hi,
The current saltfs tests are a collection of Python scripts with custom made Failure and Success classes. While these definitely work, I believe that transitioning to a framework should make the tests more usable and inviting. It is also easier to switch early on.
So I propose to switch to pytest, which is nice and easy, and also included with travis. As an alternative, we can also use the built-in unitesting framework of Python.

Rewriting the tests should be very easy, and I can take on this responsibility.
Any thoughts? Should I start working on that?

[aneeshusa]

I originally wrote the custom micro testing lib for a few reasons:

• Lack of familiarity with existing Python testing frameworks
• Desire to avoid all external dependencies, having to install them, etc.

Additionally, our tests are mostly integration style tests, not unit tests, and the default output style of most test runners seems to be optimized for unit tests. One of my main motivators was the ability to more precisely control the output format:

• PASS/FAIL at start of line, not at end for quick skimming
• Human readable summaries of each test, not method names all prefixed with test_
• Customized, human readable, immediately actionable test output for failures,
  indented to make it easy to read (i.e. what exactly failed?)
• Color support
• Automatic test discovery based on folders

It was easiest for me to quickly write what we have now than try to learn and wrangle an existing framework.

The test suite now does have some dependencies, so I wouldn't mind switching to an existing test framework. However, having usable testing output is really important to me, especially since it can take some time to run the Salt highstate just so you can start running tests, along with some other things I mentioned. You're welcome to open a PR, just keep in mind that I value usable tests more than which framework is being used, and I may ask for the output format to be customized (I have no idea how possible this is with existing frameworks).

Happy to discuss more as well.

[edunham]

The repo has grown extensively since this conversation, and IMO we would benefit greatly from using someone else's testing framework at this point. @jarondl, if you're still interested in tackling this issue, please re-open.