Get the number of passed/failed tests in the code

[erelsgl]

I would like to use doctest for automatic grading of homework assignments. For this, I need to get the number of passed tests and the number of failed tests in the code, so that I can write, at the end of the main testing program, something like: "grade = passed*100/(passed + failed)". I do not need to print a report - I only need to insert this grade into a database. Is it possible?

[onqtam]

Currently you can implement a reporter: https://github.com/onqtam/doctest/blob/master/doc/markdown/reporters.md

and in that reporter in test_run_end() you can use the stats coming in the TestRunStats struct.

And then either choose your reporter from the command line with --reporters=my_grading_reporter or implement your own main() like discussed in the docs and set the reporter like this: context.addFilter("reporters", "my_grading_reporter");.

Keep in mind that version 2.3 is coming out really soon and the reporter interface might change slightly (nothing complicated). Also 2.3 will come with an xml reporter which you could use to parse more easily the test results.

Let me know if this helps!

[erelsgl]

Thanks! Where can I find the documentation of the TestRunStats class?

[onqtam]

it's a simple struct with a few integers - no methods:

• numTestCases
• numTestCasesPassingFilters
• numTestSuitesPassingFilters
• numTestCasesFailed
• numAsserts
• numAssertsFailed

so the executed tests are numTestCasesPassingFilters (which is the same as numTestCases if there are no filter options passed) and the failed ones are numTestCasesFailed. You can also inspect the asserts.

look it up in the header

But once I release version 2.3 the registering will be made a little bit different - you'll be passing not an address to an instance but just the type name, and the framework will be instantiating the reporter. Also a constructor might be required. But I guess if you setup your grading servers with version 2.2.3 you won't have to change anything for the entire semester if you don't update the testing framework.

[erelsgl]

OK, thanks! This solves the problem