Test failures caused by whitespace differences

[DrHyde]

Today when installing the most recent version from the CPAN I got ...

#   Failed test at t/Legacy/harness_active.t line 72.
#          got: '#   Failed test 'this fails'
# #   at t/Legacy/harness_active.t line 71.
# '
#     expected: '
# #   Failed test 'this fails'
# #   at t/Legacy/harness_active.t line 71.
# '
#   Failed test at t/Legacy/harness_active.t line 85.
#          got: '#   Failed test at t/Legacy/harness_active.t line 84.
# #          got: '1'
# #     expected: '0'
# '
#     expected: '
# #   Failed test at t/Legacy/harness_active.t line 84.
# #          got: '1'
# #     expected: '0'
# '
t/Legacy/harness_active.t ...................... 
Failed 2/4 subtests 

The only difference appears to be the absence of a leading \n in the output, possibly caused by some subtle difference in one of the dependencies.

Here's the diagnostics:

# DIAGNOSTICS INFO IN CASE OF FAILURE:

# Perl: 5.020003

# CAPABILITIES:
# CAN_FORK         Yes
# CAN_REALLY_FORK  Yes
# CAN_THREAD       No

# DEPENDENCIES:
# Carp          1.3301
# File::Spec    3.48_01
# File::Temp    0.2304
# PerlIO        1.09
# Scalar::Util  1.42
# Storable      2.49_01
# Test2         1.302078
# overload      1.22
# threads       N/A
# utf8          1.13_01

[exodist]

any chance you can pull down the git repo and run a bisect for me? the past 2 weeks I have been seeing odd failures without goo reports and no way to reproduce locally.

[DrHyde]

I'll see what I can do. It looks like it's going to be "challenging", as after having force-installed it I can no longer replicate the problem.

And while it doesn't appear to be helpful in this case, have you seen Andreas König's test analysis website? http://analysis.cpantesters.org/solved?distv=Test-Simple-1.302078

[gregoa]

I just tried to package 1.302085 for Debian, and I'm seeing the same test failure, both with 5.24.1 and 5.26.0:

#   Failed test at t/Legacy/harness_active.t line 72.
#          got: '#   Failed test 'this fails'
# #   at t/Legacy/harness_active.t line 71.
# '
#     expected: '
# #   Failed test 'this fails'
# #   at t/Legacy/harness_active.t line 71.
# '
#   Failed test at t/Legacy/harness_active.t line 85.
#          got: '#   Failed test at t/Legacy/harness_active.t line 84.
# #          got: '1'
# #     expected: '0'
# '
#     expected: '
# #   Failed test at t/Legacy/harness_active.t line 84.
# #          got: '1'
# #     expected: '0'
# '
t/Legacy/harness_active.t ......................
1..4
ok 1
ok 2
not ok 3
not ok 4
Failed 2/4 subtests

[..]

Test Summary Report
-------------------
t/Legacy/harness_active.t                    (Wstat: 0 Tests: 4 Failed: 2)
  Failed tests:  3-4
Files=192, Tests=2272, 19 wallclock secs ( 0.62 usr  0.18 sys +  9.99 cusr  0.96 csys = 11.75 CPU)
Result: FAIL

Diagnostics for 5.24.1:

# DIAGNOSTICS INFO IN CASE OF FAILURE:

# Perl: 5.024001

# CAPABILITIES:
# CAN_FORK         Yes
# CAN_REALLY_FORK  Yes
# CAN_THREAD       Yes
   
# DEPENDENCIES:
# Carp          1.40
# File::Spec    3.6301
# File::Temp    0.2304
# PerlIO        1.09
# Scalar::Util  1.4202
# Storable      2.56_01
# Test2         1.302085
# overload      1.26
# threads       2.07
# utf8          1.19
t/00-report.t ..................................
ok 1
1..1

and for 5.26.0

# DIAGNOSTICS INFO IN CASE OF FAILURE:

# Perl: 5.026000

# CAPABILITIES:
# CAN_FORK         Yes
# CAN_REALLY_FORK  Yes
# CAN_THREAD       Yes
   
# DEPENDENCIES:
# Carp          1.42
# File::Spec    3.67
# File::Temp    0.2304
# PerlIO        1.10
# Scalar::Util  1.4602
# Storable      2.62
# Test2         1.302085
# overload      1.28
# threads       2.15
# utf8          1.19
t/00-report.t ..................................
ok 1
1..1

[gregoa]

I tried to bisect this between HEAD and v1.302075 (last working version in debian) and ran a test script with prove --blib --verbose t/Legacy/harness_active.t. Result:

b9c8b098fcb6be510d418b9356c3437b3660d67f is the first bad commit
commit b9c8b098fcb6be510d418b9356c3437b3660d67f
Author: Chad Granum <exodist7@gmail.com>
Date:   Sun Feb 19 13:18:18 2017 -0800

    Move diag newline to TAP formatter
    
    Fixes #751

:040000 040000 2b34d5487f164b666418b0a1b904e9dee478d5ab 3b54ea551d66ea4b5cf81cf4d2c8970dc26c6bad M	lib
bisect run success

Looking at the commit, I notice that there's HARNESS_IS_VERBOSE, and indeed, this works in HEAD:

# prove --blib t/Legacy/harness_active.t          
t/Legacy/harness_active.t .. ok   
All tests successful.
Files=1, Tests=4,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.03 cusr  0.00 csys =  0.05 CPU)
Result: PASS

(i.e. leaving out --verbose)

HTH, gregor

[exodist]

thanks, I will look into this when I have a free moment.

[exodist]

Found the source of the issue, will have a fix out in the next day or 2

[gregoa]

Thanks, I can confirm the fix, and I've uploaded 1.302098 to Debian/unstable.

[DrHyde]

Thanks