API changes breaking CPANs modules with ld errors are not caught by BBC

[ericherman]

Description
0351a62 hides symbols, some of which were used by CPAN modules.

We noticed that Algorithm-Heapify-XS depends upon Perl_do_ncmp.

And it seems FreeBSD dropped support for other distributions related to this commit:

• Data-Swap
• Devel-BeginLift
• Text-ClearSilver
• Apache-DB

(Note that some had other issues before.)

Steps to Reproduce

If we look at cpantesters, we see the ld failures are "UNKNOWN", and not "FAIL".

To see an example of the kind of failure we'd expect caught, clone demerphq/Algorithm-Heapify-XS and type perl Makefile.PL && make

Expected behavior

Changes made to the API don't break CPAN.
It would be expected that BBC would have caught these.

[jkeenan]

May I respond to your comments in somewhat a different order from that you made them in?

Expected behavior

Changes made to the API don't break CPAN. It would be expected that BBC would have caught these.

"BBC" is not some automated process. It's a set of programs run, and actions taken, by a number of different humans in support of our Perl 5 core distribution maintenance efforts. These efforts are not part of the Perl 5 Porters' responsibilities or authorities.

First, individuals have to set up machines for the purpose of running tests with CPAN::Reporter, App::cpanminus::Reporter, etc., and have to send those reports to cpantesters.org. cpantesters.org (I think) generates emails to distribution maintainers informing them of failures. (I don't know what emails are generated when the failure is at the 'Makefile.PL/Build.PL' stage or at the 'make' stage; I've only had failures at the 'make test' stage.) The grades for those reports are collated in the CPANtesters matrix, to which there are various interfaces; currently the most performant is http://fast2-matrix.cpantesters.org. So far everything is automated.

At this point a certain number of people running CPANtesters rigs, myself included, have evolved their own individual way of reporting failures to P5P via the GitHub issues queues. The presence (or absence) of a human here is crucial. That's because, given the number of distributions on CPAN, the number of the subset of distros which failed years ago and have never been fixed, and the time it takes to investigate and confirm a test failure, we generally focus on new failures. Your investigation indicates that Algorithm-Heapify-XS began to fail in June 2022. I can't say whether the distro's author was properly notified of the failure or what action he might have taken in response. However, @eserte (one of our heavy-duty CPANtesters of long-standing) reported the problem in that distribution's GH issue queue in October 2024. I had never heard of this distro until your report, so I've never included it in the list of 500 to 600 distros which I test after every monthly dev release.

So a new CPANtesters failure is not automatically reported to P5P's GH Issues queue. If it is reported to that queue, I will generally slap a "BBC" label on it, but that's a manual process. And even if a CPANtesters failure is classified as BBC in our Issues queue, a bug report is not automatically opened in that distribution's bug tracker (whether rt.cpan.org, github or elsewhere). A lot depends on our analysis of the problem once it's reported to our GH Issue tracker. If a CPAN distribution depends upon code in the Perl 5 core distro that we have deemed sub-standard, and if we repair the code in the core and the CPAN code breaks as a result, then the onus is on the CPAN maintainer to repair the code to meet the core's new quality expectations.

Description 0351a62 hides symbols, some of which were used by CPAN modules.

We noticed that Algorithm-Heapify-XS depends upon Perl_do_ncmp.

And it seems FreeBSD dropped support for other distributions related to this commit:

* Data-Swap

Last CPAN release: Sept 2008. Issue reported upstream: Sept 2018.

* Devel-BeginLift

Last CPAN release: May 2010. Failing on CPANtesters since 2016.

* Text-ClearSilver

Last CPAN release: April 2010. [Failures on systems with bsdmake](http://fast2-matrix.cpantesters.org/?dist=Data-Swap( reported in Nov 2015.

* Apache-DB

(I'll let someone else investigate that. Ever since I had problems with Apache and modperl doing CPAN River testing six years ago, I've stayed away from them when doing automated testing.)

I don't know whether the FreeBSD ports people have any procedure for notifying CPAN distribution authors that the ports built on top of the latters' distros are being classified as BROKEN. However, I think I can fairly state that while individual FreeBSD developers have reported problems to P5P, we've never (well, at least not since 2016, when I began smoke-testing on FreeBSD) had any systematic communication of problems from the ports people with respect to breakage due to changes in the Perl 5 core.

(Note that some had other issues before.)

Steps to Reproduce

If we look at cpantesters, we see the ld failures are "UNKNOWN", and not "FAIL".

UNKNOWN is how CPAN::Reporter grades distributions which fail during the make stage. (Whether that's a helpful wording of the grade is something which you'd have to take up with the Perl-Toolchain-Gang. It's not in P5P's scope.)

To see an example of the kind of failure we'd expect caught, clone demerphq/Algorithm-Heapify-XS and type perl Makefile.PL && make

Just now I ran Algorithm-Heapify-XS through my CPANtesters rig for perl-5.43.4. See: https://www.cpantesters.org/cpan/report/f8a4ec42-bfef-11f0-9a8f-ab229f11bece. I also cloned that GH repo and ran the same commands you did. The failures appear to be identical.

[book]

I think there are several discussions items wrapped in a single issue:

1. The naming of test results (PASS, FAIL, NA, UNKNOWN, INVALID).

   That ship has sailed: the only thing that can be done is document them well enough that everyone understands what they mean. UNKNOWN in particular should more readily be treated as a failure.

2. Hidden symbols

   0351a62 (between 5.37.0 and 5.37.1) hid several symbols, some of which were used in the wild by CPAN distributions, instantly breaking them (Algorithm-Heapify-XS is one example).
   Our assumption from the situation was that this wasn't reported loudly enough at time, possibly early enough to "unhide" some of those symbols, and "unbreak" the affected distributions.

   More than 3 stable versions after the fact, it's unclear if we could re-enable the symbols that make sense to expose to userland for future versions of Perl.

3. BBC itself

   BBC was a bit mysterious to us. We knew it as a tag and subject line on issues, and that it's connected to CPAN releases being tested on development versions of Perl.

   Besides the obvious fact that it's run by volunteers on their own time and resources (for which the Perl community should be very grateful), we had no idea of the level of automation (or not) incurred by the whole setup.

This ticket was trying to expose an issue (some changes to Perl broke CPAN distributions, and that didn't seem to have been detected) and possibly figure out ways to improve the whole system. This sounds like a discussion for the next Perl Toolchain Summit.

PS: I'm still interested in getting demerphq/Algorithm-Heapify-XS to work with Perl after 5.36, but I don't have the skills to debug the issue and offer a patch. Can someone please help?

[Leont]

To be honest, making do_ncmp public sounds fine to me, it's just unfortunate that we didn't caught up on it before 5.38 came out so AHXS will need to adopt that code anyway.