calculates incorrect MD5 with large scalars [rt.cpan.org #123185]

[toddr]

Migrated from rt.cpan.org#123185 (status was 'new')

From @jmdh on 2017-10-03 13:54:31
:
Forwarding from https://bugs.debian.org/877369:

Digest::MD5 calculates the wrong MD5 for large scalars:

$ perl -e 'print "\x00" x 1024 x 1024 x 1024 x 9'|md5sum -
97606009c3309d3a0b4b40ae9fadc720  -

$ perl -MDigest::MD5 -E 'say Digest::MD5::md5_hex "\x00" x 1024 x 1024 x 1024 x 9'
e02694e929aec3068ff272bebe0ae0a4

This is likely due to buggy length calculation in MD5.xs, which fails for
large values of "len" (and hasn't changed for 5.26):

    ctx->bytes_low += len;
    if (ctx->bytes_low < len) /* wrap around */
        ctx->bytes_high++;

Note that I couldn't reproduce this myself as I got an out of memory error (on a machine with 16GB of RAM). The problem does not occur when adding the data 1GB at a time using the OO interface.

Digest::MD5 is at 2.54, unmodified by Debian.

Thanks for your work on maintaining Digest::MD5!

[toddr]

Reproduced:

$>perl -e 'my $c = "\x00" x 1024; $c = $c x 1024; $c = $c x (1024* 9); print $c'|md5sum -
97606009c3309d3a0b4b40ae9fadc720  -

$>perl -Mblib -MDigest::MD5 -e 'my $c = "\x00" x 1024; $c = $c x 1024; $c = $c x (1024* 9); print Digest::MD5::md5_hex($c); print "\n"'
e02694e929aec3068ff272bebe0ae0a4

[Leont]

Given that it doesn't happen with 1GiB but does with 9, I'm guessing this is a 32 bit integer overflowing.

What does 3GiB do?

[toddr]

$>perl -e 'my $c = "\x00" x 1024; $c = $c x 1024; $c = $c x (1024* 3); print $c'|md5sum -
c698c87fb53058d493492b61f4c74189  -

$>perl -Mblib -MDigest::MD5 -e 'my $c = "\x00" x 1024; $c = $c x 1024; $c = $c x (1024* 3); print Digest::MD5::md5_hex($c); print "\n"'
c698c87fb53058d493492b61f4c74189

[toddr]

$>perl -e 'my $c = "\x00" x 1024; $c = $c x 1024; $c = $c x (1024* 4); print $c'|md5sum -                         
c9a5a6878d97b48cc965c1e41859f034  -

$>perl -Mblib -MDigest::MD5 -e 'my $c = "\x00" x 1024; $c = $c x 1024; $c = $c x (1024* 4); print Digest::MD5::md5_hex($c) . "\n"'     
c9a5a6878d97b48cc965c1e41859f034

[toddr]

It appears you have to be 8GiB to break it.