Add outgoing metadata to TestContext

[bradwilson]

Now that v3 has a TestContext object, we should provide a way for test authors (and extensibility authors who are writing test extensions) a way to record additional information that can be reported (which I'll call "outgoing metadata").

One example of such a request is UI-based testing systems which would like to attach screenshots to failed test records.

Let's assume a model that is like traits (aka "incoming metadata"): introduce name/value pairs of string => string, where the data is then added to the "results" object model, which then in turn gets placed into XML on the way out for reporting (this XML can be written directly to disk, and is also the basis of our transformation system to produce other report files, like competing/compatible XML report files and an HTML-based report).

I believe the use of strings here rather than arbitrary types (i.e., string => object) places a reasonable limit on the outgoing data to be able to ensure that it is serializable; using arbitrary objects then places an additional serialization burden on the end user, and at the end of the day, still needs to get translated into a string to be placed into XML. The usefulness of a Base64 encoding of a .NET binary serialized object is effectively zero; forcing the user to think about how to make a string value useful, therefore, ensures that this design consideration is not overlooked. It makes more sense, then, to just use string as the value type in the key/value pair, rather than object + yet another arbitrary serialization system.

What remains, then, is the decision about how much we should expect from the end user about the string values, and whether we should attempt to suggest or enforce some restrictions. For example, should they be limited in size? Should we suggest or mandate that they be human readable? If we don't mandate human readability and/or limit the size, does that mean we shouldn't render their values into the HTML report, which is intended to be relatively compact and human readable?

In the aforementioned example of screenshots, I feel a bit torn by this requirement and how best to support it (indirectly).

There are two obvious (and perhaps other non-obvious) implementations here.

1. Write the screenshot to disk and use the outgoing string value to point to the screenshot file.
2. Use a compacted version of the screenshot, and then Base64 encode that into the outgoing string value.

Option 1 is more readable in report form, but has an inherent external dependency (the file on disk) that may not have been preserved in some circumstances, like a continuous integration environment. Option 2 is not reasonable readable in a generic report, and would require some additional transformation during display that would not necessarily be inherent in the built-in HTML reporting.

As such, I'm looking for feedback on the design of this outgoing metadata system. I like the self-contained nature of the XML, and am concerned that introducing the extra concept of files necessarily complicates that model. If we directly support files, what's the best way to represent this? Perhaps the outgoing metadata is a trio of information ("key", "value", and "value-data-type") which can be used to help reports determine what to do with that information? Do we also need to provide an arbitrary external storage mechanism for values, or should we assume that specific value types are actually just Base64-encoded binary values?

[HoraceBury]

@bradwilson Any news on this issue? So desperately want to report out screenshots! Thanks for the hard work.

[bradwilson]

@HoraceBury It's on the v3 roadmap. It will be part of v3, but has not been implemented yet. No other information is available at this time.

[naeemakram]

Hey! What's up with this one? I would love to add some screenshots to my test reports.

[JamesTerwilliger]

OK, I'm digging into this item in my brain just a little bit just out of curiosity. Please excuse my ignorance. Some questions:

1. It seems as if options 1 and 2 above are kind of the same option if one leaves it open to the user or the extensibility point to decide how to use it. I would imagine that whoever is calling the reporting methods in the tests is at least talking to the person writing the UI for the test output, so whether the caller writes a URI or a base64 encoded string is up to the caller, and the UI writer will just need to know what to do with it, and from the xUnit perspective, a string is a string is a string. Or am I missing something?
2. Poking around at the TestContext object, there already appear to be some very handy SendDiagnosticMessage methods. These obviously don't do quite the right thing, but maybe have the right interface? As in, "I have a context object, I have some of my own context, let's write it down!" So perhaps what's needed are (1) methods that look like ReportState(string key, string message) with possibly some of the same overloads as SendDiagnosticMessage for templating, but also (2) save that new state alongside the key metadata at that point of the execution so that it can be collated later, and finally (3) a sink to which to write all of that delicious, delicious new state when it's all done. Is that right?
3. And if that's right, does that mostly mean that we need an interface to specify that sink and possibly what serialization format to use (XML versus JSON) alongside 1 and 2 above?

[bradwilson]

whether the caller writes a URI or a base64 encoded string is up to the caller

I don't think that's sufficient, though, because if someone just writes a file to a disk, how do you translate that into a durable URI?

Poking around at the TestContext object, there already appear to be some very handy SendDiagnosticMessage methods. These obviously don't do quite the right thing, but maybe have the right interface?

I don't think so, because this is more like traits than diagnostic messages. If the user wants to report something which is a formatted string, they can just use string.Format or interpolation on their own.

does that mostly mean that we need an interface to specify that sink and possibly what serialization format to use (XML versus JSON) alongside 1 and 2 above?

The discussion here for XML is the reporting format that already exists. What would show up to the runners would likely just be a Dictionary<string, string> like traits today.