panic: sv_len_utf8 cache 0 real 1

[dur-randir]

This is a bug report for perl from afl@dorothy,
generated with the help of perlbug 1.41 running under perl 5.31.5.

[Please describe your issue here]

While fuzzing perl v5.29.10-23-g7c0d7520a3 built with afl and run
under libdislocator, I found the following program

printf "%n0%n\x{100}", $_, 0

to cause a panic: sv_len_utf8 cache 0 real 1 for 0. This is broken from at least 5.16.3.

[Please do not change anything below this line]
Flags:
category=core
severity=medium
Site configuration information for perl 5.31.5:

Configured by root at Fri Oct 18 05:50:54 MSK 2019.

Summary of my perl5 (revision 5 version 31 subversion 5) configuration:
Derived from: 859b78b
Platform:
osname=linux
osvers=4.19.0-6-amd64
archname=x86_64-linux
uname='linux dorothy 4.19.0-6-amd64 #1 smp debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 gnulinux '
config_args='-des -Dusedevel -Dcc=gcc -DDEBUGGING -Doptimize=-O0 -g -ggdb3'
hint=recommended
useposix=true
d_sigaction=define
useithreads=undef
usemultiplicity=undef
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='gcc'
ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
optimize='-O0 -g -ggdb3'
cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion=''
gccversion='8.3.0'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='gcc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.28.so
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.28'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O0 -g -ggdb3 -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl):
Compile-time options:
DEBUGGING
HAS_TIMES
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
PERL_USE_DEVEL
USE_64_BIT_ALL
USE_64_BIT_INT
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
Locally applied patches:
uncommitted-changes
Built under linux
Compiled at Oct 18 2019 06:02:50
%ENV:
PERLBREW_BASHRC_VERSION="0.78"
PERLBREW_HOME="/home/afl/.perlbrew"
PERLBREW_MANPATH="/home/afl/perlbrew/perls/perl-5.20.2/man"
PERLBREW_PATH="/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.20.2/bin"
PERLBREW_PERL="perl-5.20.2"
PERLBREW_ROOT="/home/afl/perlbrew"
PERLBREW_VERSION="0.78"
@inc:
lib
/usr/local/lib/perl5/site_perl/5.31.5/x86_64-linux
/usr/local/lib/perl5/site_perl/5.31.5
/usr/local/lib/perl5/5.31.5/x86_64-linux
/usr/local/lib/perl5/5.31.5

[tonycoz]

printf "%n0%n\x{100}", $_, 0

The problem is the utf8 length magic isn't being updated. Normally if you modify a SV you'd call SvSETMAGIC() which will remove the cached length magic and allow it to be regenerated the next time the length is needed.

But sv_catpvfn and friends don't do set magic as the SV is concatenated to (or at all), so the length magic isn't removed and ends up inconsistent with the actual length in utf8 characters, causing an assertion failure in debugging builds, and the wrong length in non-debugging builds.

Another issue here is the code uses the entire length of the string for utf8 strings rather than the change in length (unlike the non-utf8 case), though I don't think this is visible from Perl code (sprintf() always uses sv_vsetpvfn() and %n isn't used much.)

The fix for the first is probably to bypass the cache by calling Perl_utf8_length() and to explicitly remove this magic once we're done (otherwise a caller might get a nasty surprise.)

[tonycoz]

Fixed by PR 17239