Autotesting

It's important to be able to autotest as much of renderdoc's functionality as possible. Using the external API (the one used by C#) and some basic test setups it should be possible to verify a lot of the functionality. This will also work as proof-of-concept of the API being general enough to be used outside of the C# UI.

Testing the UI itself will be more difficult to automate and I'm hoping that as long as I can do an in-depth test of the functionality, a relatively lightweight test of the UI will suffice. The idea of this is that I can kick this off overnight after an overnight build, and then do some more basic manual testing on top of that, and be pretty confident that the build is good.

NOTE: this isn't intended to replace proper thorough testing for a full versioned release, which would have a week or so of stabilisation and testing as much as possible. It's more for being able to put out a beta version each week as an intermediate point between official builds every few months and master HEAD changing every few days.

The tests will fall along two lines:

1. Capture side - these will involve making targeted tiny demos that, apart from necessary boilerplate, only do one single thing. These should ideally cover as much of the possible use of the API as I can. It can also be an evolving list. "Success" for these tests would pretty much just entail capturing and replaying correctly.
2. Replay side - these will be a test of the replay and analysis functionality. Either a bespoke-written or a pre-existing public demo will be used, probably a few different demos to expose different functionality. For some tests we'll use the tiny apps from above to test e.g. a particular part of shader debugging. This part will have more 'test' code written, to verify that the different analysis functions work as expected.

Requirements:

• A framework for auto testing.
  1. Take a process/working directory/command line and run the program using the API, get the capture and save it off somewhere temporary.
  2. Run a test in a separate process, monitor it to make sure it exits successfully (ie. no crashes encountered running the test, it doesn't time out for some reasonable timeout) and that the test returns success for whatever it was testing (could be many outputs).
  3. Utility functions for checking results that are 'hard', like image diffs.
  4. Either embed a tiny intepreter for some scripting language if there's one simple enough, or just write a little DSL. That way the framework can be data-driven just to determine what to do on the lines of "run X and capture to file", "load capture and check that texture 123 looks like this: reference", "load capture and check that pixel debugging pixel X,Y at event N returns this result", etc. TinyJS looks like it would be perfect for this.
• A framework for building tests. This will be per-API, just providing utility functions to make writing a tiny app really simple to test anything within that API. Also a few higher-level functions that do things like open and render a complex mesh for targeted tests that need more complex data than just a couple of quads.