Test suite started to fail (with Test2::Suite 0.000121?)

[eserte]

The tests started to fail. Beginning of test log:

    # Failed test at t/01_basic.t line 47.
    # +---------------+-----+---------------+----+---------------+--------+
    # | PATH          | LNs | GOT           | OP | CHECK         | LNs    |
    # +---------------+-----+---------------+----+---------------+--------+
    # |               |     | ARRAY(0x426a2 |    | <ARRAY>       | 33, 39 |
    # |               |     | d38)          |    |               |        |
    # |               |     |               |    |               |        |
    # | [0]           | 28  | Test2::Event: |    | <EVENT: Ok>   | 34     |
    # |               |     | :Fail=HASH(0x |    |               |        |
    # |               |     | 43b6e120)     |    |               |        |
    # |               |     |               |    |               |        |
    # | [0] <blessed> |     | Test2::Event: | eq | Test2::Event: |        |
    # |               |     | :Fail         |    | :Ok           |        |
    # |               |     |               |    |               |        |
    # | [0]{pass}     |     | <DOES NOT EXI |    | 0             |        |
    # |               |     | ST>           |    |               |        |
    # |               |     |               |    |               |        |
    # | [1]           |     | <DOES NOT EXI |    | <EVENT: Diag> | 35     |
    # |               |     | ST>           |    |               |        |
    # |               |     |               |    |               |        |
    # | [2]           |     | <DOES NOT EXI |    | <EVENT: Diag> | 38     |
    # |               |     | ST>           |    |               |        |
    # +---------------+-----+---------------+----+---------------+--------+

# Failed test 'JSON cmp failure (expect raw hash)'
# at t/01_basic.t line 48.
... (etc) ...

Statistical analysis suggests that the failure started with Test2::Suite 0.000121 (@exodist: FYI):

****************************************************************
Regression 'mod:Test2::Suite'
****************************************************************
Name           	       Theta	      StdErr	 T-stat
[0='const']    	      1.0000	      0.0000	16611328747973736.00
[1='eq_0.000097']	     -0.0000	      0.0000	  -0.92
[2='eq_0.000100']	      0.0000	      0.0000	   0.80
[3='eq_0.000111']	     -0.0000	      0.0000	  -0.84
[4='eq_0.000114']	     -0.0000	      0.0000	  -1.74
[5='eq_0.000115']	     -0.0000	      0.0000	  -1.30
[6='eq_0.000116']	      0.0000	      0.0000	   0.00
[7='eq_0.000119']	      0.0000	      0.0000	   0.00
[8='eq_0.000120']	     -0.0000	      0.0000	  -2.61
[9='eq_0.000121']	     -1.0000	      0.0000	-16115356002295272.00

R^2= 1.000, N= 111, K= 10
****************************************************************

[exodist]

I introduced a new 'Fail' event that combines the failed assertion and the diagnostics into a single event, some Test2::Suite tools were switched to use it. These tests will need to be updated to support the new events.

I am thinking about how to avoid these kinds of issues in the future, I am thinking a test-testing tool that does not pay attention to the event types, but rather just pays attention to their effect. But you should not wait for that before updating your tests.

Sorry for the inconvenience.

[eserte]

With 0.04 it looks like this:

****************************************************************
Regression 'mod:Test2::Suite'
****************************************************************
Name           	       Theta	      StdErr	 T-stat
[0='const']    	      0.0000	      0.0000	   0.00
[1='eq_0.000114']	      0.0000	      0.0000	   6.08
[2='eq_0.000115']	      0.0000	      0.0000	   7.02
[3='eq_0.000116']	      0.0000	      0.0000	  14.05
[4='eq_0.000117']	      0.0000	      0.0000	  12.17
[5='eq_0.000118']	      0.0000	      0.0000	   0.00
[6='eq_0.000119']	      0.0000	      0.0000	  12.17
[7='eq_0.000121']	      1.0000	      0.0000	54788654129668552.00
[8='eq_0.000122']	      1.0000	      0.0000	76428620070309264.00

R^2= 1.000, N= 46, K= 9
****************************************************************

That is, it passes if Test2::Suite 0.000121 or 0.000122 is installed, and fails with older versions.

[akiym]

@eserte Thank you. I've released these fixes with 0.05.