Warning: Malformed UTF-8 character in substitution operation

[p5pRT]

Migrated from rt.perl.org#70998 (status was 'resolved')

Searchable as RT70998$

[p5pRT]

From slavenr@iconmobile.com

Created by slavenr@devpc01-debian.iconmobile.de

The following code worked fine with 5.8.8, but outputs a number of warnings
with 5.10.0 and 5.10.1​:

#!/usr/bin/perl

use strict;
use warnings;

my %conv = (
  "\xab" => "<",
  "\xa9" => "(c)",
  );
my $conv_rx = '(' . join('|', map { quotemeta } keys %conv) . ')';
$conv_rx = qr{$conv_rx};

my $x = qq{\x{3042}\x{304b}\x{3055}\x{305f}\x{306a}\x{306f}\x{307e}\x{3084}\x{3089}\x{308f}\x{3093}\x{3042}\x{304b}\x{3055}\x{305f}\x{306a}\x{306f}\x{307e}\x{3084}\x{3089}\x{308f}\x{3093}\x{30a2}\x{30ab}\x{30b5}\x{30bf}\x{30ca}\x{30cf}\x{30de}\x{30e4}\x{30e9}\x{30ef}\x{30f3}\x{30a2}\x{30ab}\x{30b5}\x{30bf}\x{30ca}\x{30cf}\x{30de}\x{30e4}\x{30e9}\x{30ef}\x{30f3}\x{30a2}\x{30ab}\x{30b5}\x{30bf}\x{30ca}\x{30cf}\x{30de}\x{30e4}\x{30e9}\x{30ef}\x{30f3}};

$x =~ s{$conv_rx}{$conv{$1}}eg;

__END__

The warnings are​:

Malformed UTF-8 character (unexpected continuation byte 0xab, with no preceding start byte) in substitution (s///) at /tmp/bla.pl line 15.
Malformed UTF-8 character (unexpected continuation byte 0xa9, with no preceding start byte) in substitution (s///) at /tmp/bla.pl line 15.
(etc)

Regards,
  Slaven

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.10.1:

Configured by slavenr at Wed Sep  2 17:15:52 CEST 2009.

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:
   
  Platform:
    osname=linux, osvers=2.6.18-4-486, archname=i686-linux
    uname='linux devpc01-debian 2.6.18-4-486 #1 wed feb 21 15:25:16 utc 2007 i686 gnulinux '
    config_args='-ds -e -Dprefix=/usr/perl5.10.1'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.1.2 20061115 (prerelease) (Debian 4.1.1-21)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /usr/lib64
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.9.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.9'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

Locally applied patches:
    


@INC for perl 5.10.1:
    /usr/perl5.10.1/lib/5.10.1/i686-linux
    /usr/perl5.10.1/lib/5.10.1
    /usr/perl5.10.1/lib/site_perl/5.10.1/i686-linux
    /usr/perl5.10.1/lib/site_perl/5.10.1
    .


Environment for perl 5.10.1:
    HOME=/home/slavenr
    LANG=hr_HR.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/mysql-max-5.0.15-linux-i686/bin:/usr/local/netpbm/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/slavenr/bin/linux-gnu:/home/slavenr/bin/sh:/home/slavenr/bin:/usr/X11R6/bin:/usr/X11/bin:/usr/games:/home/slavenr/devel:/services/mobile/framework/bin/:/epoc/nokia60/epoc32/tools:/usr/local/er6/bin/:/services/mobile/clientframework/perl/bin/:/usr/lib/jvm/java-1.5.0-sun/bin:/usr/java/ant/bin
    PERLDOC=-MPod::Perldoc::ToTextOverstrike
    PERL_BADLANG (unset)
    PERL_HTML_DISPLAY_CLASS=HTML::Display::Mozilla
    SHELL=/usr/bin/zsh

[p5pRT]

From @cpansprout

This bug (which was caused by change 28373/07be1b8) can be reduced to​:

qq{\x{30ab}} =~ /\xab|\xa9/;

Malformed UTF-8 character (unexpected continuation byte 0xab, with no
preceding start byte) in pattern match (m//) at /Users/sprout/Perl/
5.8.7-regressions/70998.d/70998 copy line 2.

30ab is e3 82 ab in UTF-8.

The trie optimisation in S_find_byclass (added by change
28373/07be1b8) searches in e3 82 ab for a char matching [\xab\xa9],
and sets 2 (\xab) as its starting position. The code it passes control
to then stumbles across this ‘lone’ \xab.

The attached patch fixes this.

[p5pRT]

From @cpansprout

Inline Patch

diff -Nurp blead/regexec.c blead-70998/regexec.c
--- blead/regexec.c	2009-12-12 00:57:11.000000000 -0800
+++ blead-70998/regexec.c	2009-12-13 23:26:41.000000000 -0800
@@ -1722,7 +1722,17 @@ S_find_byclass(pTHX_ regexp * prog, cons
                                             " Scanning for legal start char...\n");
                                     }
                                 );            
-                                while ( uc <= (U8*)last_start  && !BITMAP_TEST(bitmap,*uc) ) {
+                                if(do_utf8) /* Skip continuation octets */
+                                 while (    /* for UTF8 strings */
+                                  uc <= (U8*)last_start
+                                  &&(
+                                   UTF8_IS_CONTINUATION(*uc)
+                                    ||
+                                   !BITMAP_TEST(bitmap,*uc)
+                                  )
+                                 ) uc++;
+                                else while ( uc <= (U8*)last_start
+                                          && !BITMAP_TEST(bitmap,*uc) ) {
                                     uc++;
                                 }
                                 s= (char *)uc;
diff -Nurp blead/t/re/pat_rt_report.t blead-70998/t/re/pat_rt_report.t
--- blead/t/re/pat_rt_report.t	2009-11-19 08:51:40.000000000 -0800
+++ blead-70998/t/re/pat_rt_report.t	2009-12-13 21:54:00.000000000 -0800
@@ -21,7 +21,7 @@ BEGIN {
 }
 
 
-plan tests => 2511;  # Update this when adding/deleting tests.
+plan tests => 2512;  # Update this when adding/deleting tests.
 
 run_tests() unless caller;
 
@@ -1076,6 +1076,42 @@ sub run_tests {
            '(?>) does not cause wrongness on long string with UTF-8';
     }
 
+    {
+	local $BugId = 70998;
+	local $Message
+	 = 'utf8 =~ /trie/ where trie matches a continuation octet';
+
+	# Catch warnings:
+	my $w;
+	local $SIG{__WARN__} = sub { $w .= shift };
+
+	# This bug can be reduced to
+	qq{\x{30ab}} =~ /\xab|\xa9/;
+	# but it's nice to have a more 'real-world' test. The original test
+	# case from the RT ticket follows:
+
+	my %conv = (
+		    "\xab"     => "&lt;",
+		    "\xa9"     => "(c)",
+		   );
+	my $conv_rx = '(' . join('|', map { quotemeta } keys %conv) . ')';
+	$conv_rx = qr{$conv_rx};
+	
+	my $x
+	 = qq{\x{3042}\x{304b}\x{3055}\x{305f}\x{306a}\x{306f}\x{307e}}
+	 . qq{\x{3084}\x{3089}\x{308f}\x{3093}\x{3042}\x{304b}\x{3055}}
+	 . qq{\x{305f}\x{306a}\x{306f}\x{307e}\x{3084}\x{3089}\x{308f}}
+	 . qq{\x{3093}\x{30a2}\x{30ab}\x{30b5}\x{30bf}\x{30ca}\x{30cf}}
+	 . qq{\x{30de}\x{30e4}\x{30e9}\x{30ef}\x{30f3}\x{30a2}\x{30ab}}
+	 . qq{\x{30b5}\x{30bf}\x{30ca}\x{30cf}\x{30de}\x{30e4}\x{30e9}}
+	 . qq{\x{30ef}\x{30f3}\x{30a2}\x{30ab}\x{30b5}\x{30bf}\x{30ca}}
+	 . qq{\x{30cf}\x{30de}\x{30e4}\x{30e9}\x{30ef}\x{30f3}};
+	
+	$x =~ s{$conv_rx}{$conv{$1}}eg;
+
+	iseq($w,undef);
+    }
+
 
     #
     # Keep the following tests last -- they may crash perl

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @demerphq

2009/12/20 Father Chrysostomos <sprout@​cpan.org>​:

This bug (which was caused by change 28373/07be1b8) can be reduced to​:

qq{\x{30ab}} =~ /\xab|\xa9/;

Malformed UTF-8 character (unexpected continuation byte 0xab, with no
preceding start byte) in pattern match (m//) at
/Users/sprout/Perl/5.8.7-regressions/70998.d/70998 copy line 2.

30ab is e3 82 ab in UTF-8.

The trie optimisation in S_find_byclass (added by change 28373/07be1b8)
searches in e3 82 ab for a char matching [\xab\xa9], and sets 2 (\xab) as
its starting position. The code it passes control to then stumbles across
this ‘lone’ \xab.

The attached patch fixes this.

Is this a problem/tested in blead?

I have a feeling that when i disabled certain trie functionality i
"fixed" this one.

Before this gets applied id like to know more.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @cpansprout

On Dec 21, 2009, at 2​:46 AM, demerphq wrote​:

2009/12/20 Father Chrysostomos <sprout@​cpan.org>​:

This bug (which was caused by change 28373/07be1b8) can be reduced
to​:

qq{\x{30ab}} =~ /\xab|\xa9/;

Malformed UTF-8 character (unexpected continuation byte 0xab, with no
preceding start byte) in pattern match (m//) at
/Users/sprout/Perl/5.8.7-regressions/70998.d/70998 copy line 2.

30ab is e3 82 ab in UTF-8.

The trie optimisation in S_find_byclass (added by change
28373/07be1b8)
searches in e3 82 ab for a char matching [\xab\xa9], and sets 2
(\xab) as
its starting position. The code it passes control to then stumbles
across
this ‘lone’ \xab.

The attached patch fixes this.

Is this a problem/tested in blead?

I have a feeling that when i disabled certain trie functionality i
"fixed" this one.

Before this gets applied id like to know more.

I hope this is a sufficient answer​:

$ perl5.11.3 -e 'qq{\x{30ab}} =~ /\xab|\xa9/'
Malformed UTF-8 character (unexpected continuation byte 0xab, with no
preceding start byte) in pattern match (m//) at -e line 1.

Whether mine is the best way to fix this I cannot say.

[p5pRT]

From @demerphq

2009/12/27 Father Chrysostomos <sprout@​cpan.org>​:

On Dec 21, 2009, at 2​:46 AM, demerphq wrote​:

2009/12/20 Father Chrysostomos <sprout@​cpan.org>​:

This bug (which was caused by change 28373/07be1b8) can be reduced to​:

qq{\x{30ab}} =~ /\xab|\xa9/;

Malformed UTF-8 character (unexpected continuation byte 0xab, with no
preceding start byte) in pattern match (m//) at
/Users/sprout/Perl/5.8.7-regressions/70998.d/70998 copy line 2.

30ab is e3 82 ab in UTF-8.

The trie optimisation in S_find_byclass (added by change 28373/07be1b8)
searches in e3 82 ab for a char matching [\xab\xa9], and sets 2 (\xab) as
its starting position. The code it passes control to then stumbles across
this ‘lone’ \xab.

The attached patch fixes this.

Is this a problem/tested in blead?

I have a feeling that when i disabled certain trie functionality i
"fixed" this one.

Before this gets applied id like to know more.

I hope this is a sufficient answer​:

$ perl5.11.3 -e 'qq{\x{30ab}} =~ /\xab|\xa9/'
Malformed UTF-8 character (unexpected continuation byte 0xab, with no
preceding start byte) in pattern match (m//) at -e line 1.

Whether mine is the best way to fix this I cannot say.

Thanks. I decided to fix it a different way, but I applied your test
code. Patches pushed to blead​:

commit d085b49
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Tue Nov 2 11​:29​:18 2010 +0100

  Fix RT-70998​: qq{\x{30ab}} =~ /\xab|\xa9/ produces warnings

commit aca5303
Author​: Father Chrysostomos <sprout@​cpan.org>
Date​: Tue Nov 2 11​:28​:33 2010 +0100

  Add test for rt-70998​: qq{\x{30ab}} =~ /\xab|\xa9/ produces warnings

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

@cpansprout - Status changed from 'open' to 'resolved'