Command Line Switches?

[michaellarabel]

Hi Tyler, just checking to see if you plan on implementing command-line switches for controlling all the different toggles from the command line so the testing could be run in a fully-automated manner from start to finish and then dumping the performance output to a log file or just the command line? Thanks.

[Tylemagne]

Command line arguments are easier to implement in Windows Unity builds due to them being a part of the .NET framework.

If I can't get command line arguments to work on Linux and Mac, I'd rather approach a tri-platform solution like a separate config and output file.

What would you prefer between JSON and XML for a config file? As for the log file I could output a simple "result.txt" with the number inside if I can't get a command line return to work. Do you have a preference on score output if command line returns aren't possible?

[michaellarabel]

Really indifferent between JSON and XML, but more people these days generally seem to like JSON over XML.

Yeah a txt output for results work of the average FPS and ideally min/max FPS if it's exposed too or any extras like per-frame time, etc.

[Tylemagne]

I like the idea of outputting more than just a single number for a score. That was actually a plan of mine.

With that, I might just output the final score information as a JSON object as well. Off the top of my head, I am absolutely certain I can output the score, the score's difficulty level, the average framerate, the time of the longest frame, and the time of the shortest frame (in ms). Are there any other pieces of result data you'd find useful for the suite?

[michaellarabel]

About the only other thing was if possible to like dump a comma delimited list of all the frame times, that way could toss it into a line graph for seeing the data to a greater detail.

JSON works for result output too or even like CSV format might work for some if trying to import it into a spreadsheet or just a simple key: value list, from the test suite side the PTS parser should be able to handle any/all of them.

[Tylemagne]

That's an even better idea. I was toying with the idea, but I was stumped because I couldn't build a UI graphing system that I liked, but a comma separated list is a perfect alternative.

I think the most elegant way of including a comma-separated block of frame times would be to simply include it in the JSON object as a nested array of floats.

Maybe, for the sake of ease of use, I can have it push this array into both the final JSON object -and- a CSV file (if building a reliable CSV file is easy across all 3 platforms). Or maybe just one. Whatever makes it the most useful (excessive file accumulation can be annoying).

That reminds me, what sort of naming convention would work with the suite best? Should I have one single file name that it overwrites, or would it be best to unix timestamp the name? (ex: 1518567808.json) Or, maybe the output name could even be specified in the config.json file (outfilename: "PTS.json").

[michaellarabel]

With regard to the filename, entirely indifferent, basically whatever would be easier for you.