Failing tests counted but displayed as TODO with $TODO defined but false

[dboehmer]

Test::More seems to behave inconsistently when $TODO has a defined but false value. A failing test is counted for the test summary but is displayed as TODO test. That is really confusing.

TODO, or not TODO: that is the question
William Shakespeare

The following test file illustrates the issue:

use strict;
use warnings;

use Test::More;

my @values = ( 
    undef,            # as expected
    "",               # false but defined -> inconsistent
    0,                # false but defined -> inconsistent
    0.0,              # false but defined -> inconsistent
    "0.0",            # true -> TODO
    "this is why",    # as expected
);

for my $value (@values) { 
  TODO: {
        local $TODO = $value;
        fail "foo";
    }
}

pass "bar";

done_testing;

Only the first and the two last failing tests are displayed consistently. The other 3 are both TODO and not-TODO:

$ perl todo.t 
not ok 1 - foo
#   Failed test 'foo'
#   at todo.t line 18.
not ok 2 - foo
#   Failed (TODO) test 'foo'
#   at todo.t line 18.
not ok 3 - foo
#   Failed (TODO) test 'foo'
#   at todo.t line 18.
not ok 4 - foo
#   Failed (TODO) test 'foo'
#   at todo.t line 18.
not ok 5 - foo # TODO 0.0
#   Failed (TODO) test 'foo'
#   at todo.t line 18.
not ok 6 - foo # TODO this is why
#   Failed (TODO) test 'foo'                                                                                        
#   at todo.t line 18.                                                                                              
ok 7 - bar                                                                                                          
1..7                                                                                                                
# Looks like you failed 4 tests of 7.

I am not sure whether $TODO defined but false should make a test TODO or not TODO. Most of the false values seem not to be a useful TODO message but having "" for no message at all might be desirable.

Unfortunately I also don't know enough about the guts of Test::More to provide a fix. I'd gladly do with some hints.

[exodist]

a defined but falsy $TODO making TODO effective is a legacy Test::More behavior, too late to debate the behavior without breaking things (Though if it were not too late I would be on the side of a false value disabling todo). However it is absolutely a bug that it displays the TODO but counts a failure, it should not count a failure.

[dboehmer]

Thanks for picking this up!

Can you give me a hint where this is to be fixed or are you going to fix it yourself?

[exodist]

it is probably somewhere in Test2::API::Hub, if not there then maybe in Test2::Event::*. I will fix it when I have a moment, but probably not today.