BBC: Blead Breaks CryptX

[cjg-cguevara]

This is a bug report for perl from "Carlos Guevara" carlos@carlosguevara.com,
generated with the help of perlbug 1.43 running under perl 5.39.4.

---

BBC: Blead Breaks CryptX

Please see http://fast-matrix.cpantesters.org/?dist=CryptX%200.078

---

Flags

• category=core
• severity=low

---

Perl configuration

Site configuration information for perl 5.39.4:

Configured by cpan at Mon Sep 25 13:36:02 EDT 2023.

Summary of my perl5 (revision 5 version 39 subversion 4) configuration:
  Commit id: f1a9bff9425f5b43d0d63d1c4be758d25dd17925
  Platform:
    osname=openbsd
    osvers=7.3
    archname=OpenBSD.amd64-openbsd
    uname='openbsd cjg-openbsd7 7.3 generic#3 amd64 '
    config_args='-des -Dprefix=/home/cpan/bin/perl -Dscriptdir=/home/cpan/bin/perl/bin -Dusedevel -Duse64bitall'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    optimize='-O2'
    cppflags='-fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='OpenBSD Clang 13.0.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-Wl,-E  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/lib/clang/13.0.0/lib /usr/lib /usr/local/lib
    libs=-lpthread -lm -lutil -lc
    perllibs=-lpthread -lm -lutil -lc
    libc=/usr/lib/libc.so.97.0
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags='-DPIC -fPIC '
    lddlflags='-shared -fPIC  -L/usr/local/lib -fstack-protector-strong'


---
@INC for perl 5.39.4:
    /home/cpan/bin/perl/lib/site_perl/5.39.4/OpenBSD.amd64-openbsd
    /home/cpan/bin/perl/lib/site_perl/5.39.4
    /home/cpan/bin/perl/lib/5.39.4/OpenBSD.amd64-openbsd
    /home/cpan/bin/perl/lib/5.39.4

---
Environment for perl 5.39.4:
    HOME=/home/cpan
    LANG (unset)
    LANGUAGE (unset)
    LC_ALL=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/cpan/bin/perl/bin:/home/cpan/bin:/home/cpan/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/bash

[hvds]

Sample fail report http://www.cpantesters.org/cpan/report/fdc0c49a-5a47-11ee-9ece-aaae1ab0edc7 suggests this is falling foul of the planned changes to Math::BigInt handling of >>. @pjacklam can you take a look, and perhaps talk with @karel-m about adapting to the changes?

[mauke]

Yeah, a bisect points to commit 6a23753 ("cpan/Math-BigInt - Update to version 1.999841"), so this is really a Math::BigInt issue, isn't it?

[pjacklam]

I'll take a look.

[jkeenan]

Given the large number of CPAN distros that depend on Math-BigInt, there is likely to be a lot of breakage. I'm doing a CPANtesters run of the 189 first-order revdeps so that we get some idea of the scope of the problem. (This will take some hours to run.)

[sisyphus]

... and perhaps talk with @karel-m about adapting to the changes?

Yes - these recent changes actually bring Math::BigInt's shift operations into line with the way perl's shift operators deal with negative arguments.

OTOH, if you wanted to advocate that the old behaviour be retained you'd probably point out that such operations are "undefined behaviour" in C99.

Cheers,
Rob

[pjacklam]

Yes - these recent changes actually bring Math::BigInt's shift operations into line with the way perl's shift operators deal with negative arguments.

Yes, more specifically when the second operand, the number of places to shift, is negative. The changes also bring Math::BigFloat's shift operations into line with the way perl’s shift operators deal with non-integer arguments.

While many distributions use Math::BigInt, I am not sure how common it is to use negative shifts, e.g., $x << -3, especially since this used to return a NaN anyway. And using << and >> on non-integer values was considered a “corner case” in a recent discussion. Making the behaviour of << and >> in line with core Perl will also make bignum’s claim “transparent big number support for Perl” less misleading. Also note that the changes only affect the shift operators << and >>. The methods blsft() and brsft() work as before.

The CryptX distribution seems to be using test files directly copied from the Math-BigInt distribution, so the simplest solution to the failing tests is to copy the latest test files from the Math-BigInt distribution.

As for how much breakage this causes, I’ll wait for the results from @jkeenan. If the damage is too great to be justified, I will revert the change.

[jkeenan]

The CryptX distribution seems to be using test files directly copied from the Math-BigInt distribution, so the simplest solution to the failing tests is to copy the latest test files from the Math-BigInt distribution.

As for how much breakage this causes, I’ll wait for the results from @jkeenan. If the damage is too great to be justified, I will revert the change.

I think we may have dodged a bullet here. AFAICT, the only CPAN distribution which is directly dependent on Math-BigInt and which has suffered a failure in CPANtesters as a result of the recent upgrade to M-BI version 1.999841 (and its subsequent synching into the Perl core distribution in 6a23753 on Sep 24) is the one originally reported in this ticket: CryptX.

Here is what I did to reach this inference.

I screen-scraped http://deps.cpantesters.org/depended-on-by.pl?dist=Math-BigInt, then wrote a program (available upon request) that processed that data into a list of CPAN modules suitable for passing as arguments to cpan.

$ perl $P5P_DIR/math-bigint-revdeps.pl | wc -l
189

$ perl $P5P_DIR/math-bigint-revdeps.pl 
Acme::Affinity
Acme::KeyboardMarathon
Affix
Algorithm::Bayesian
Algorithm::BitVector
Algorithm::RandomMatrixGeneration
Algorithm::Voting
Alt::Crypt::RSA::BigInt
Alt::Data::Frame::ButMore
Astro::Units
Authen::Bitcard
Authen::HOTP
Authen::OATH
Authen::OATH::OCRA
Authen::TypeKey
Avro
BSON
Bencher::Scenario::BigFloat
Bencher::Scenario::BigInt
Bencher::Scenarios::MathBigFloat
BioPerl
Bitcoin::Crypto
Bitmask::Data
Blockchain::Contract::Solidity::ABI
Blockchain::Ethereum::ABI
Bundle::Zavitan
Business::IBAN
Business::RO::TaxDeduction
CAD::Calc
CLDR::Number
CORBA::IDL
CPU::x86_64::InstructionWriter
CXC::Number
Chart::Math::Axis
Chemistry::Canonicalize
Chemistry::Harmonia
Chicken::Ipsum
Convert::ASN1
Crixa
Crypt::AES::CTR
Crypt::AppleTwoFish
Crypt::DH
Crypt::DRBG
Crypt::DSA
Crypt::DSA::GMP
Crypt::HSXKPasswd
Crypt::MagicSignatures::Key
Crypt::OpenSSL::Base::Func
Crypt::PBC
Crypt::RFC8188
Crypt::SRP
CryptX
DBD::Firebird
DBD::Oracle
DBIx::Class::InflateColumn::BigFloat
DBIx::Class::Tree::Mobius
DBIx::CouchLike
DNS::Bananafonana
DOCSIS::ConfigFile
Data::Dumper::Interp
Data::Frame
Data::LazyACL
Data::MEID
Data::MessagePack
Data::Money
Data::ParseBinary
Data::UUID::Concise
Data::UUID::NCName
Data::Validate
DateTime::Format::Epoch
DateTime::Format::WindowsFileTime
DateTime::Util::Astro
DateTime::Util::Calc
Dev::Bollocks
Devel::Cover
ELF::Writer
Encode::Base58::BigInt
Encode::Positive::Digits
Encode::Positive::Pairs
Enterprise::Licence
Ethereum::RPC::Client
FFI::Raw
Finance::Tax::Aruba
Form::Sensible::Reflector::MySQL
Format::Util
FunctionalPerl
Geo::Calc
Geo::Code
Geo::Code::XYZ
Geo::Code::XYZEN
GnuPG::Interface
Google::ProtocolBuffers
Hashids
Hessian::Tiny
IP::Geolocation::MMDB
IPDR
InterMine::TypeLibrary
JSON::Schema::Modern
JSON::Schema::Tiny
Ledger::Parser
Library::CallNumber::LC
Logwatch::RecordTree
MRS::Client
Mac::Alias::Parse
Mac::PropertyList
Mail::Graph
MarpaX::ESLIF
MarpaX::Java::ClassFile
MarpaX::Languages::ECMAScript::AST
MarpaX::Languages::IDL::AST
MarpaX::Languages::SQL2003::AST
Math::Algebra::Symbols
Math::Assistant
Math::Base36
Math::Base85
Math::BaseCnv
Math::BaseConvert
Math::Big
Math::BigInt::BitVect
Math::BigInt::Constant
Math::BigInt::FastCalc
Math::BigInt::GMP
Math::BigInt::GMPz
Math::BigInt::Lite
Math::BigInt::Named
Math::BigInt::Pari
Math::BigInt::Parts
Math::BigInt::Random
Math::BigInt::Random::OO
Math::BigRat
Math::Counting
Math::Currency
Math::DyckWords
Math::FractionManip
Math::Int128
Math::Logic::Ternary
Math::ModInt
Math::Prime::Util::GMP
Math::ProvablePrime
Math::RandomOrg
Math::Revhash
Math::Roman
Math::String
MaxMind::DB::Common
MooseX::Types::XMLSchema
Music::RecRhythm
Net::Amazon::HadoopEC2::S3fs
Net::Async::Blockchain
Net::BitTorrent
Net::Ethereum
Net::IP::Identifier
Net::IP::XS
Net::IPAddress::Util
Net::IPv6Addr
Net::Netmask
Net::OpenID::Common
Net::SNMP
Number::Format::BigFloat
Number::WithError
OPTIMADE::Filter
Object::Import
PGObject::Type::BigFloat
Parse::StackTrace
Pass::OTP
Path::Hilbert
Protocol::CassandraCQL
RDF::Helper
RDF::Trine
REFECO::Blockchain::Contract::Solidity::ABI
REFECO::Blockchain::Contract::Solidity::ABI::Encoder
Rose::DB::Object
SNMP::Info
SQL::Translator::Parser::OpenAPI
Sphinx::Search
Statistics::Test::RandomWalk
Statistics::Test::Sequence
Store::Digest
TOML::Tiny
Template::Mustache
Text::Wrap::Smart
Time::Duration::Object
Time::Moment::Epoch
Tk::ObjScanner
Types::Numbers
Types::XMLSchema
WARC
Win32::UrlCache
Win32API::File
perfSONAR_PS::Services::MA::perfSONARBUOY

On FreeBSD-13, I built a threaded perl in my customary manner at f1a9bff (which was HEAD yesterday afternoon).

$ ./bin/perl -Ilib -v |head -2 | tail -1
This is perl 5, version 39, subversion 4 (v5.39.4 (v5.39.3-49-gf1a9bff942)) built for amd64-freebsd-thread-multi

I then installed Task::CPAN::Reporter against that perl and emptied out my data base CPAN-Reporter reports sent. In a screen session, I then called:

$ perl $P5P_DIR/math-bigint-revdeps.pl | xargs ./bin/cpan

I checked the results today:

$ wc -l reports-sent.db
     846 reports-sent.db

$ grep -c FAIL reports-sent.db
21

$ grep FAIL ~/.cpanreporter/reports-sent.db
test FAIL CryptX-0.078 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Astro-Units-1.10 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Data-Clone-0.004 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Data-Sah-Coerce-0.053 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Devel-Size-0.83 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL BSD-Sysctl-0.11 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Bitmask-Data-2.05 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL CXC-Number-0.12 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Net-AMQP-RabbitMQ-2.40010 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Crypt-HSXKPasswd-v3.6 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Crypt-OpenSSL-Base-Func-0.037 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL DBIx-CouchLike-0.16 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL DateTime-Util-Astro-0.12000 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Dev-Bollocks-0.06 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Ethereum-RPC-Client-0.05 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Form-Sensible-0.20023 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL FunctionalPerl-0.72.76 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL GnuPG-Interface-1.03 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Unicode-Map8-0.13 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Math-RandomOrg-0.04 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1
test FAIL Net-BitTorrent-0.052 (perl-5.39.4) amd64-freebsd-thread-multi 13.2-release-p1

Of the 21 distributions that were graded FAIL, some (e.g., Devel-Size) were those I recognized from earlier CPAN-River-3000 testing as failing to build on FreeBSD for years. Many of the others have not built on any platform for years. Only CryptX's failure was directly attributable to Math::BigInt.

However, I caution that my inspection of CPANtesters reports for the 21 FAILs was strictly visual, so my eyes might have missed something.

I further caution that no Windows-specific distributions (or prerequisites therefor) were tested.

Nonetheless, we don't yet have any evidence that the recent upgrade to Math-BigInt will cause widespread breakage once it goes into wide release with the release of perl-5.40 next May.

[sisyphus]

The methods blsft() and brsft() work as before.

I'm seeing different behaviour regarding blsft() and brsft() withM-BI-1.999839 and M-BI-1.999841

Using 1.999839:

D:\>perl -MMath::BigInt -le "print $Math::BigInt::VERSION; print Math::BigInt->new(12345)->blsft(-1);"
1.999839
NaN

D:\>perl -MMath::BigInt -le "print $Math::BigInt::VERSION; print Math::BigInt->new(12345)->brsft(-1);"
1.999839
NaN

But I get something different when using 1.999841:

D:\>perl -MMath::BigInt -le "print $Math::BigInt::VERSION; print Math::BigInt->new(12345)->blsft(-1);"
1.999841
6172

D:\>perl -MMath::BigInt -le "print $Math::BigInt::VERSION; print Math::BigInt->new(12345)->brsft(-1);"
1.999841
24690

Sorry if I've misunderstood what you mean by "work as before" ?

Cheers,
Rob

[pjacklam]

@sisyphus, you are of course correct, and I am suitably humbled. I have to keep better track of the changes I make.

I did indeed make some changes to blsft() and brsft() partly based on wishes from users and partly based on my desire for consistency and fixing bugs. The methods blsft() and brsft() should support shifting an arbitrary value $x an arbitrary number of places $n in an arbitrary base $b.

• As you show above, I added support for negative values of $n.
• I improved the handling of NaNs. For example, the following now gives the correct result NaN, but with version 1.999839 you get

$ perl -MMath::BigInt -wle '$x = Math::BigInt -> new(2); print $x -> blsft(3, "NaN")'
0

• I added support for arbitrarily large bases $b. In previous versions, a very large base would give the error “Base is too large”.

I will update the appropriate entry in the CHANGES file and also check/update the documentation and tests for the two methods.

Thank you very much for pointing this out, and sorry for being sloppy.

[andk]

From the 21 FAIL reports above one would want to point out that CXC-Number-0.12 has a solid history of PASS reports on Linux in the matrix: http://matrix.cpantesters.org/?dist=CXC-Number

But then from http://www.cpantesters.org/cpan/report/9c3c52ca-5ba5-11ee-8668-ee14a3603691

Deep recursion on subroutine "Math::BigInt::babs" at /home/sand/src/perl/repoperls/installed-perls/host/k93jammy/v5.39.3/322f/lib/site_perl/5.39.3/Math/BigInt.pm line 1603.

[pjacklam]

That looks pretty bad. I'll dig into it and see what is causing this and what I need to change/revert.

[pjacklam]

From the 21 FAIL reports above one would want to point out that CXC-Number-0.12 has a solid history of PASS reports on Linux in the matrix: […]

I have gone through all the 21 modules that failed. Most of them failed because of issues not related to the Math::Big* modules, but there were some exceptions.

• CryptX fails for known reasons (see above). Hopefully it is sufficient to update test files.
• Astro::Units started failing with Math-BigInt 1.999829 released 2021-12-29 when div_scale() started requiring a positive argument. Astro::Units sets div_scale to zero, which doesn't make much sense, since it is the fall-back value for the accuracy when accuracy or precision hasn't been set explicitly. Astro::Units seems like a useful module, so I can change the behaviour from throwing an error to a giving a warning and setting the div_scale to 1.
• Dev::Bollocks has failed for a long time. I'll have a look at why it fails, but it seems like a module made just for fun, so this has lower priority.
• CXC::Number fails because of a recent attempt at improving the handling of subclasses. Unfortunately, when Math::BigInt upgrades to Math::BigFloat, and Math::BigFloat inherits a method from Math::BigInt, this can result in an endless loop. The following illustrates the problem:

$ perl -MMath::BigFloat -le 'Math::BigInt -> upgrade("Math::BigFloat"); $x = Math::BigFloat -> bone() -> babs()'
Deep recursion on subroutine "Math::BigInt::babs" at /usr/local/lib/perl5/Math/BigInt.pm line 1603.

I will release a new version of the Math-BigInt distribution as soon as I can.