-
Notifications
You must be signed in to change notification settings - Fork 0
Test Plan
This page describes the testing process I go through for a new release. Currently there are no automated tests although I do try and keep a bank of example logfiles. I use these for regressing against during development but at release time I tend to re-capture all of the logs to test the capturing process.
Some of these are inspired by bugs that have happened in the past that can be generalised a bit, so feel free to add to this list if you find a good general case that should be checked.
- Capture a 32bit program with a 32bit renderdoc. Test capture options to make sure they work and test callstack gathering + resolving. Load the logfiles in 64bit renderdoc and verify that everything works the same ('everything' is vague, but try to test a good range of features; shader debugging, texture viewing, mesh viewing, pipeline state, shader editing, etc).
- Capture a 32bit program from a 64bit renderdoc. Do the above tests and load the logfile in 32bit renderdoc respectively.
- Capture a 64bit program from a 64bit renderdoc. Again do the above tests and load the logfile in 32bit renderdoc.
- The rest of the tests detailed can be done on whichever UI.
Semi-complete list of features to test. These are replay-side features that can be done on (most) logs. This really needs replacing with automated testing for at least some of this.
- Markers expand/collapse correctly and resize as they become visible
- Reads, Writes and Clears to targets are displayed as they're selected
- Current drawcall (if visible by marker) is highlighted
- Jumping between drawcalls works and is responsive (no performance/usability regressions)
- Expanding markers and sections
- Selecting drawcalls within command lists (if API-applicable)
- Use find to find some substring that matches multiple times in the frame, all matches should be highlighted and previous/next will let you jump between them.
- Use goto to jump to an EID. In all these cases it should work and select the right event:
- Try jumping to the EID of a drawcall exactly.
- Try jumping to the EIDs before or after a drawcall.
- Try jumping to the EID of a marker push.
- Time the drawcalls and ensure durations are fetched and are self consistent (ie. sum of child durations are shown on the durations of parents).
- Rearrange columns, add a new one and remove one displayed.
- Capture a log with no markers and ensure good auto-markers are generated.
- Callstacks are covered partially in the 32bit/64bit case to make sure that logs work back and forth, but make sure they resolve and display correctly for each API call.
- If there aren't any callstacks, that should be indicated and the tools -> resolve symbols menu item should be disabled.
- Also make sure all events that should be displayed are (ie. none are missing from the list and the EIDs are consistent).
Obviously, check that all information is correct as best you can. Also throughout make sure that the information is trimmed by default to only shader-referenced resources, and that "show disabled" and "show empty" show the relevant extra resources where available.
Also, make sure that for each drawcall that the correct pipeline stages are highlighted or not highlighted depending on whether they're active.
Input Layout
- Check that input layouts highlight the corresponding vertex buffers, and that the index buffer is displayed for indexed draws and not otherwise.
- Launch the mesh view from the input layouts and from the button.
- Show the input layout bytecode shader and check that its signature matches the input layout, or if not that a correct error is displayed.
Shader Stages
- If shader debug information is available, entry point is listed on the shader, otherwise a default name. Launch shader editor and viewer for bound stages and make sure it does nothing for unbound stages.
- Launch a texture viewer, buffer viewer resources bound as relevant. Make sure for buffer viewers that the definition is propogated, if reflection data is available.
- Launch constant buffer previewer for constant buffers.
- Check that UAVs are properly listed next to render targets/depth targets.
- Fairly simple test - make sure variables are displayed and parsed with the correct values, including matrices, arrays, (nested) structures, arrays of structures, arrays of matrices, etc.
- When moving between events it should update to whatever constant buffer is in the slot open, or become empty if no constant buffer is referenced there.
- Open a buffer, either from vertex buffer or from a buffer resource
- Check that buffer formats both structured (complex) and simple (basic types) display correctly.
- Change the format, ensure the data is reinterpreted correctly. Test various combinations of matrices, vectors, types, widths, etc
- Check that slices of a 3D texture/2D array, faces of cubemap, and mips all display correctly.
- Check range control - zoom into a section, move black/white, try to make the range empty, use auto-fit and view histogram.
- Check pixel picking, make sure the right value is fetched for the texture format, and that the pixel context updates. For MSAA textures ensure that a reasonable value comes out (either one sample, or a simple sample-average resolved value).
- Open locked tabs of new textures and make sure that they stay on the same texture as you move between draws
- The usage of the current texture should be displayed on the timeline bar.
- Check each of the overlays to ensure they're working.
- Toggle channels on/off, and use the RGBM and custom shader visualisation modes.
- Launch debug a pixel and check that it works when you have the drawcall selected and doesn't work if you click on a non-covered pixel.
- Save a texture out to dds
- Open the texture list, open some textures from this list, filter it by name/RT/Tex and unfilter.
- Check that VS in is always available, that VS out is always available, and when relevant that GS/DS output is available.
- Columns should be named correctly for the inputs/outputs of each shader
- Sync views between all three tabs
- Launch shader debugger for a vertex
- Export raw data to both CSV and raw. Check that this behaves sensibly if there is no real data.
- Use the row offset to look at a given row as you move between drawcalls
- Select a different instance on an instanced drawcall
- When a position component can be identified, check that the mesh renders in the preview for the relevant tab.
- Make sure that the highlight triangles line up correctly when you look at different vertices
- Use the different camera modes (arcball/WASD) and use the auto-fit to zoom in on some inputs.
- On vertex input, use the different shading modes to visualise the mesh.
- Use the "since last clear" mode to view a whole scene's worth of mesh data.
- Tune the vs output projection matrix parameters - FOV for perspective, near/far/aspect for ortho (e.g. shadow renders)
- Make sure syntax highlighting works, and that assembly is properly inlined with source when available
- Check that registers are renamed for constants, samplers, etc.
- When debugging make sure you can step back, forward, set breakpoints, run to cursor, and that this all behaves correctly even with loops and things.
- When editing you can change a shader and have it be reflected in the capture
- Edit errors and warnings appear in the message box
- Closing the shader editor reverts and edits that have been applied.
- Launch a program on one PC and capture it on another. Test this with all four combinations of 32bit/64bit.
- Check you can kick off someone who's attached to an injected program.
- Thumbnails should appear when captures are made remotely, you should be able to trigger and queue captures, then copy & load them locally.
- Every element in the UI should do nothing, or at the very least not crash or give some sensible behaviour if you interact with it while no log is loaded.
- Check both before any log has been opened and after a log has been opened then closed to make sure there isn't anything that lingers on.
- Both with and without a log loaded, close down all windows and open them all back up again to recreate the layout. Save and restore different layouts back and forth
This is a list of per-API functionality that should be tested, i.e. you should find a set of programs that utilise these and capture them to ensure that the results are as expected and correct. This list comes from just a brain-dump of different cases, most from cases that have broken at one point or another.
This should translate into a set of simple programs (or one program that can be run N ways) that utilises each of these in isolation for easier testing and automated regression.
- Fill in this list...
- For now just try and get coverage over as many games/apps/demos as possible
It's good to go through the changes since the last release and prepare a changelog, then use this changelog to test specific areas that have changed. This is useful both to make sure the changes are good and fix the problem, as well as to find anything they might have broken in the neighbourhood.