One of the bigger issues blocking release is that about 100 of the 1000 Test:: modules on CPAN are broken with Test::Builder. Most of these are due to minor formatting changes in the TAP output in modules that don't use Test::Builder::Tester or Test::Tester (and some that do).
The new TAP v13 formatter has a few minor changes compared to the existing formatter. The one which has the largest effect is a TAP version is output before the plan. This confuses a lot of Test::* tests which expect the plan to be the first thing.
@chromatic suggested a fix is to make the default formatter match what Test::Builder currently outputs and worry about introducing formatting changes later. Fortunately we already have a formatter which simulates the quirks of the existing Test::Builder.
Here's the choices I'd like to hear people's opinions on.
To be clear none of the formatter changes break Test:: modules at runtime_. That is, if you have Test::Foo 1.23 installed right now it will still work with the new formatter no matter what we do right now. This is only about Test::_'s own tests. There will be no need for a user to update their install of Test::Foo for Test::Builder 1.5 because of the formatter change.
The work involved in the first is fixing about 50-100 Test module tests. This can either be a straight hack or converting them to Test::Builder::Tester or Test::Tester. Most of this is fairly straight forward for somebody who understands Perl and tests, it's just a lot of work and pushing back to authors and getting releases made. I can be done at the Perl QA Hackathon.
The work involved in the second is fixing Test::Builder 1.5's own tests. Unfortunately, I wasn't very good at developing and using a formatted agnostic self-test suite and suffer from exactly the same problem as the Test::* modules, just in the other direction. I'd guestimate the patching will be more difficult than patching the Test::* modules, but that's only a guess.
PS This effects the upcoming Perl QA Hackathon and I need to make a decision on this before Friday.
Which Test::* modules are broken? And next, how many other modules in the CPAN use these tests? (Without presuming you have the answers now -- in which case the question becomes "how can we find this out?")
I have the means to find that out and I'm running it now. I'll post it once it's done.
There's a program in dist/test_dists which runs the tests of every /^Test-/ dist against both your installed Test::More and against one installed in /tmp/tb2/lib/perl5 for comparison. If you'd like to run it yourself...
git checkout Test-Builder1.5
perl Makefile.PL INSTALL_BASE=/tmp/tb2
It takes some time, there are 722 Test- dists. I have an old list kicking around that tested 388 of them. These are just distributions which fail, I don't know why.
I've run nearly a full scan and made the list available as #352. There are at least 104 failing distributions out of 722.
I'm running the test comparison with the old formatter to see how much better that works. We have the handy TB2_FORMATTER_CLASS environment variable to control the default formatter.
TB2_FORMATTER_CLASS=TB2::Formatter::TAP::TB1 perl dist/test_dists --mirror file:///var/local/CPAN_mirror --lib=/tmp/TB2/lib/perl5
I ran 215 tests, shuffled, before the test script blew out. 44 failed which gives a 20% failure rate. This is worse than the 13-14% failure rate with the new formatter
Turns out this is a bug in TB2::CanLoad. See #356
After analysing a sample of those failures as part of the work for issue 352 it doesn't appear that there are many distributions that are broken solely by TAP format issues. I'd suggest we don't gain much by rolling back.
Yeah, the event storage change seems to be a much bigger problem. Maybe we should talk about changing that default.
I'm going to close this up.