Build reporting capabilities

[kkozmic]

As discussed with @JakeGinnivan and @mwhelan in comments for #8

Using BDDfy for that would be nice, but we need to find a way to keep it as an option, not mandatory. Possibly #6 could offer a solution to that

[kkozmic]

ok, so let's start talking about this. For now (v2) let's concentrate on keeping it simple and ConventionTests specific. Later, once we've proven it works well, we'll look at perhaps extracting it out to meet the needs of BDDfy or other parts of TestStack.

So where are we now?

Currently we're hardcoded to two kinds of tests:

• Simple ensure convention is met by every item.
• Ensure convention is met by every item, but also allow "approved" exceptions. An extreme form of that approach is to ignore the part with exceptions and just use the convention to generate the list of all items (every item is the "approved" exception).

The idea about reporting came up when building support for readable output of 2nd convention type. At the moment it's all text fed into a StringBuilder. It's the responsibility of the Convention class to build that string. Some chunks might come from the data type as well.

What scenarios do we want to support?

Here's a bunch of scenarios I think we might like to support

• report of exceptions (CSV (or xls(x))): on my current gig the product owner has a 'spec' Excel file that the testers then use to manually verify certain aspects of the app's behaviour/structure.
  Certain parts of that could be verified by a convention however and remove the manual error prone steps.
  Convention could be built outputting a CSV file describing the relevant aspects of the system, that could then be approved and/or compared against the original spec XLS much more easily without needing to run the application.
• living documentation (html (or markdown) document(s)): the grand idea behind convention tests (yes, it does have one) is to make it a tool for building living documentation. Scenario would be for all convention tests to output a nice human readable (meta)format (like html or markdown) and then for that to be harvested and put in an easily accessible place for people to view (most likely as part of CI build), but not necessarily.

What does that imply?

The first scenario (csv) is admittedly much simpler (and therefore we might decide to stick to that for v2, and expand further to support the second scenario in v3).
It requires reporting only for approved tests. Also it doesn't require the reporting to be dynamic (that is possibly changing from environment to environment).

The second scenario would possibly require dynamic reporting capabilities (generate markdown on CI server, but normal text on dev machines). Also the second scenario would require reporting for every kind of convention (including absolute ones, without exceptions).

[kkozmic]

@JakeGinnivan what do you think? Does that make sense? Do you agree that's roughly where we might want to go?

[kkozmic]

This is a csv file generated with my first attempt at building this. PR coming in a sec

[JakeGinnivan]

Will you be rebasing the CSV onto the other reporting stuff. It looks cool

[kkozmic]

I kept my local branch. I'll cherry pick whatever makes sense and build it into what we've got

[JakeGinnivan]

This is done really. We can raise new issues for specific things