Generate a HTML report for SageNB tests

[qed777]

It would be useful to have a HTML report summarizing the results of Sage Notebook tests, including the output and traceback, if any, of each test.

We can begin with the notebook's functional test suites (cf. #7343), but we could add doctests, eventually. And with a backend server, we might select, run, and monitor tests remotely.

CC: @williamstein @TimDumol

Component: notebook

Author: Mitesh Patel

Reviewer: Tim Dumol

Issue created by migration from https://trac.sagemath.org/ticket/7390

[qed777]

Attachment: trac_7390-sagenb_test_report_A.patch.gz

HTML test report. Part A. Apply to sagenb repo.

[qed777]

Attachment: trac_7390-sagenb_test_report_B.patch.gz

HTML test report. Part B. Apply to sagenb repo after part A.

[qed777]

comment:1

Note: The patches depend on #7343. Please apply parts A and B, in order. To test the test report generator, try

sage: from sagenb.testing.run_tests import run_and_report; run_and_report()

This should run the notebook's Selenium test suite, generate and write a self-contained report.html, and open the report in SAGE_BROWSER.

Remarks:

• Part A adds HTMLTestRunner.
• Part B makes significant changes to HTMLTestRunner.py and sets up the test runner for sagenb.
• The runner captures all output, e.g., we can display information from passing tests, too.

To do, but not necessarily in this ticket:

• Stabilize the results table's column widths, especially when toggling tracebacks.
• Support multiple results tables.
• Use the runner to display doctest results.
• Use a backend server to select, run, and monitor live tests.

[qed777]

comment:2

Mistake: run_and_report's docstring should say open_viewer defaults to True.

[qed777]

comment:3

Doctesting may be straightforward. Given foo.py, sage-doctest preparses its triple-quoted blocks and writes them as docstrings of example_* functions in .doctest_foo.py. This file calls on itself a subclass of doctest.DocTestRunner. Since Python's doctest module can also generate unittests, we can "replace" the end of .doctest_foo.py with, e.g.,

if __name__ ==  '__main__':
    from sagenb.testing.run_tests import run_and_report
    import doctest
    run_and_report(doctest.DocTestSuite())
    sys.exit(0)

Although the test names example_* are not informative, sage -python .doctest_foo_mod.py runs the tests and makes a report!

[qed777]

HTML test report. Part B Version 2. Apply to sagenb repo after part A.

[qed777]

comment:4

Attachment: trac_7390-sagenb_test_report_B_v2.patch.gz

Version 2 (of part B):

• Solves the table width problem for Cr3, FF3.5, IE8, O10, S4 on Windows XP and Cr4, FF3.5, O10 on Linux.
• Works around a jQuery / IE8 toggle bug.
• Fixes run_and_report's docstring.

[qed777]

comment:6

Feel free to suggest improvements to the form and function of the report!

[qed777]

comment:7

There's sample report at

• http://sage.math.washington.edu/home/mpatel/trac/7390/report.html

[TimDumol]

comment:8

Replying to @qed777:

There's sample report at

• http://sage.math.washington.edu/home/mpatel/trac/7390/report.html

If you don't mind, I can try restyling the tests. The colors are a bit jarring, in my opinion.

[qed777]

comment:9

Feel free.

[TimDumol]

Attachment: trac_7390-sagenb_test_report_referee.patch.gz

Removes unicode characters from the documentation. Restyles color scheme to a lighter layout.

[TimDumol]

Reviewer: Tim Dumol

[TimDumol]

comment:10

This patch removes unicode characters from the documentation that would otherwise require magic comments (feel free to do so instead). The color scheme has been inverted to a lighter layout, with colors taken from the color palette on Wikipedia.

Things look great. The only thing I might want added are timings for each individual test, but that can go in another patch. Positive review. All that's needed now is for someone to review the referee patch.

[qed777]

Attachment: trac_7390-sagenb_test_report_referee_v2.patch.gz

Update TODO list. Version 2 of referee patch.

[qed777]

comment:11

Looks good to me. I'll change this status to "positive review."

[williamstein]

comment:12

merged into sagenb for sage-4.3