Richer Test Statuses

[fpottbaecker]

I have recently stumbled across the TAP, and this is an excellent approach to supplying a more interoperable standard for reporting test run statuses. I just had some ideas I would consider good. While I have read the guidelines and understand that changes will not be accepted unless adopted by multiple implementations, this is apparently also the place to get some guidance.

So, TAP provides a decent overview of the rough progress of a test suite, there is a few things that could be considered missing from some perspective, especially when looking at more graphical representations of test suites:

1. Marking the start of tests and reporting intermediate progress (which could be shown as an activity indicator in front of the test name).
   From reading some other issue discussions, I understand that some producers use comments in front of the test points, but this does not enabled association with a specific test until the test point has been emitted.
2. Parallelism with respect to Subtests. Once a sub test has been started, it must be concluded with a valid test point. This forbids all reporting of the potential status of other tests in the test suite. It also kind of goes in hand with the prior point, "starting" a subtest and reporting at least the plan would enable a more accurate estimation of the number of tests to represent the status of the full test run.
3. Test discovery without running the tests, at least in a more formalized way.
   A test harness could currently output just skipped tests when asked to list the full test structure, but this might collide with the semantics of actually skipped tests during discovery.

To maybe give a better understanding of this idea, a possible document supporting this could maybe look like this:

TAP version NEXT
1..6

test 1 - Check simple addition
test 2 - Check simple subtraction
# naming is secondary, maybe also: (register|waiting|pending|...)

test 3 - StringParser # subtest
# same here, could be "(register|waiting|pending|...) sub", subtest, subt, ...
# the idea is to immediately communicate this is a subtest without requiring the
# harness to switch this test to a subtest once a valid indented line is found.

    # Opening the subtest for the first time instructs the harness to start
    # keeping track of a new subtest identified by its numerical path (here: 3)

    plan 1..3
    # A plain plan would make this a valid line and start this document/subtest,
    # to report this later we want to keep this invalid until completion

    test 1 - handles valid strings

    test 2 - rejects invalid strings # subtest
        # start of subtest document 3.2

        test 1 - too long
        test 2 - too short
        test 3 - illegal characters

        plan 1..3
        # Deviating from TAP 14, this plan can also appear before test points,
        # but could be restricted to forbid registering new tests afterwards.

    # A valid line of lower indentation pauses all lower level documents and
    # resumes the document at this level (here 3).

    # test 2
    # For the purposes of human readability, the producer could/should emit the
    # ID of the id of the closed subtest, maybe as a comment.

    test 3 - parses data
# test 3

test 4 - Another Thing # subtest
    plan 1..1
    test 1 - works too
# test 4

start 1 - Check simple addition
# This marks a test as started, again, maybe also "running", ...
# Also it is an open question whether changing the description of a test is
# allowed. Restating the same description should be allowed (and possibly
# encouraged if human readability is desired).

start 2
  ~~~
  Database starting...
  Service starting...
  ~~~
# This is where things might get tricky, clearly marking plain log output to
# displayed to the user directly (more friendly than yaml), maybe just an a half
# indent without ---, terminated by other lines, including comments of lower
# level. Or maybe this should have another surrounding marker, like ~~~.

# A test can only be started once, but can be targeted for further events.
test 1
  ---
  event: cleanup
  ---

start 5 - Check simple multiplication
# The first event of a test is its registration. The `test ID` is just an event
# without a further meaning.

ok 6 - very quick test
# The producer could debounce test events if human readability is desired.
# This was only the third line of valid TAP 14.

not ok 5 - Check simple multiplication
# If registered with a description, it must repeated for the (not) ok line to
# provide compatibility with TAP 14

start 3
# subtest directive could be omitted if the test was already registered.

    start 1
# test 3

start 4 # subtest
    start 1
# test 4

test 3
    start 2
        pass 1
        fail 2
        pass 3
    fail 2
    # Comment closing form should be emitted if an event occured. Maybe closing
    # should just be done using a plain `test ID` line anyway.
# test 3
# No ok has been emitted for subtest 3.2 yet, since subtest 3 cannot be closed,
# as tests 3.1 and 3.3 are still running and pending.

# 4.1 succeeded now, this marks the completion of every child of root subtest 4
# hence it can be reported fully.
# Subtest: Another Thing
    ok 1 - works too
ok 4 - Another Thing

test 3
    pass 1

# 3.3 succeeded now, report root subtest 3.
# emitting pass 3 in the block earlier is optional, but ok 3 should be first.

# Subtest: StringParser
# Subtest comments starting any child of the current active path should be ignored.
    ok 3 - parses data
    ok 1 - handles valid strings
    # Subtest: rejects invalid strings 
        ok 1 - too long
        not ok 2 - too short
        ok 3 - illegal characters
    not ok 2 - rejects invalid strings
ok 3 - StringParser

ok 1 - Check simple addition
ok 2 - Check simple subtraction
        

With these (or similar) features, tests and subtests can be marked as running and intermittently update (1 & 2) and a test discovery run could just register existing tests (3).
Of course these features could be further configured by producers for potentially desired easier human readability (example setup: disable TAP 14 compat, do not register tests, only emit a start for a test or subtest if longer than x seconds, only leave the current subtest prematurely if not update in less than x seconds, ...).

Having written this now, I realize it is a lot. I am happy to hear feedback on whether this is feasible, or further pointers where to look as well (specific implementations, ...).