A number of characters match both a posix class and its complement

[p5pRT]

Migrated from rt.perl.org#60156 (status was 'resolved')

Searchable as RT60156$

[p5pRT]

From Robin.Barker@npl.co.uk

Created by robin.barker@npl.co.uk

As I read the documentation, the pairs in perlrepod of
[[​:...​:]] and \p{Is....} are supposed to match the same.

But
\p{IsPrint} matches \011 \012 \013 \014 \015
which [[​:print​:]] does not

\p{IsPunct} does not match $ + < = > ^ ` | ~
which [[​:punct​:]] does match

Various \p{Is...} match characters in the range 128-256
but no [[​:...​:]] match characters in that range.

Perl Info

---
Flags:
    category=core
    severity=medium
---
Site configuration information for perl 5.10.0:

Configured by Robin Barker at Tue Dec 18 11:23:28 GMT 2007.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.18-1.2798.fc6, archname=i686-linux-64int
    uname='linux rain.npl.co.uk 2.6.18-1.2798.fc6 #1 smp mon oct 16 14:54:20 edt 2006 i686 i686 i386 gnulinux '
    config_args='-des -Dcc=gcc -Uinstallusrbinperl -Dcf_email=Robin.Barker@npl.co.uk -Dcf_by=Robin Barker -Dman1dir=none -Dman3dir=none -Doptimize=-O2 -g -Duse64bitint -Dotherlibdirs=/usr/lib/perl5:/usr/lib/perl5/site_perl:/usr/lib/perl5/vendor_perl -Dinc_version_list=5.8.8 5.8.7 5.8.6 5.8.5 5.8.4 5.8.3 5.8.2 5.8.1 5.8.0 5.6.1 5.6.0 5.005'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -g',
    cppflags='-DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='4.2.2', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.5.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'

Locally applied patches:
    

---
@INC for perl 5.10.0:
    /usr/local/lib/perl5/5.10.0/i686-linux-64int
    /usr/local/lib/perl5/5.10.0
    /usr/local/lib/perl5/site_perl/5.10.0/i686-linux-64int
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/local/lib/perl5/site_perl/5.8.8
    /usr/local/lib/perl5/site_perl/5.005
    /usr/local/lib/perl5/site_perl
    /usr/lib/perl5/5.8.8
    /usr/lib/perl5/5.8.7
    /usr/lib/perl5/5.8.6
    /usr/lib/perl5/5.8.5
    /usr/lib/perl5
    /usr/lib/perl5/site_perl/58.8
    /usr/lib/perl5/site_perl/5.8.7
    /usr/lib/perl5/site_perl/5.8.6
    /usr/lib/perl5/site_perl/5.85
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.8
    /usr/lib/perl5/vendor_perl/5.8.7
    /usr/lib/perl5/vendor_perl/5.8.6
    /usr/lib/perl5/vendor_perl/5.8.5
    /usr/lib/perl5/vendor_perl
    .

---
Environment for perl 5.10.0:
    HOME=/home/rmb1
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/lib:/usr/local/lib
    LOGDIR (unset)
    PATH=/home/rmb1/appl/script:/opt/perl/bin:/opt/gcc/bin:/opt/SUNWspci/bin/:/usr/rain.npl.co.uk/bin:/usr/local/bin:/usr/local/Admigration/exec:/usr/local/hotjava/bin:/usr/openwin/bin:/usr/dt/bin:/usr/bin:/bin
    PERL5LIB=
    PERL_BADLANG (unset)
    SHELL=/bin/tcsh

Robin Barker
Mathematics and Scientific Computing group
F1-A8
Ext: 7090



-------------------------------------------------------------------
This e-mail and any attachments may contain confidential and/or
privileged material; it is for the intended addressee(s) only.
If you are not a named addressee, you must not use, retain or
disclose such information.

NPL Management Ltd cannot guarantee that the e-mail or any
attachments are free from viruses.

NPL Management Ltd. Registered in England and Wales. No: 2937881
Registered Office: Serco House, 16 Bartley Wood Business Park,
                   Hook, Hampshire, United Kingdom  RG27 9UY
-------------------------------------------------------------------

[p5pRT]

From Robin.Barker@npl.co.uk

No progress in resolving this in code, so here is documentation patch.

Robin

--- pod/perlre.pod.orig 2008-01-30 20​:41​:06.000000000 +0000
+++ pod/perlre.pod
@​@​ -375,8 +375,8 @​@​
  digit IsDigit \d
  graph IsGraph
  lower IsLower
- print IsPrint
- punct IsPunct
+ print IsPrint (but see 2. below)
+ punct IsPunct (but see 3. below)
  space IsSpace
  IsSpacePerl \s
  upper IsUpper
@​@​ -385,6 +385,41 @​@​

For example C<[[​:lower​:]]> and C<\p{IsLower}> are equivalent.

+However, the equivalence between C<[[​:xxxxx​:]]> and C<\p{Xxxxx}> is not exact.
+
+=over 4
+
+=item 1.
+
+C<[[​:xxxxx​:]]> only matches characters in the range 0x00-0x7F.
+
+=item 2.
+
+C<\p{IsPrint}> matches characters 0x09-0x0d but C<[[​:print​:]]> does not.
+
+=item 3.
+
+C<[[​:punct​::]]> matches the following but C<\p{IsPunct}> does not,
+because they are classed as symbols in Unicode.
+
+=over 4
+
+=item C<$>
+
+Currency symbol
+
+=item C<+> C<< < >> C<=> C<< > >> C<|> C<~>
+
+Mathematical symbols
+
+=item C<^> C<`>
+
+Modifier symbols (accents)
+
+=back
+
+=back
+
If the C<utf8> pragma is not used but the C<locale> pragma is, the
classes correlate with the usual isalpha(3) interface (except for
"word" and "blank").

[p5pRT]

From Robin.Barker@npl.co.uk

perlre.patch

--- pod/perlre.pod.orig	2008-01-30 20:41:06.000000000 +0000
+++ pod/perlre.pod
@@ -375,8 +375,8 @@
     digit       IsDigit        \d
     graph       IsGraph
     lower       IsLower
-    print       IsPrint
-    punct       IsPunct
+    print       IsPrint		(but see 2. below)
+    punct       IsPunct		(but see 3. below)
     space       IsSpace
                 IsSpacePerl    \s
     upper       IsUpper
@@ -385,6 +385,41 @@
 
 For example C<[[:lower:]]> and C<\p{IsLower}> are equivalent.
 
+However, the equivalence between C<[[:xxxxx:]]> and C<\p{Xxxxx}> is not exact.
+
+=over 4
+
+=item 1.
+
+C<[[:xxxxx:]]> only matches characters in the range 0x00-0x7F.
+
+=item 2.
+
+C<\p{IsPrint}> matches characters 0x09-0x0d but C<[[:print:]]> does not.
+
+=item 3.
+
+C<[[:punct::]]> matches the following but C<\p{IsPunct}> does not,
+because they are classed as symbols in Unicode.
+
+=over 4
+
+=item C<$>
+
+Currency symbol
+
+=item C<+> C<< < >> C<=> C<< > >> C<|> C<~>
+
+Mathematical symbols
+
+=item C<^> C<`>
+
+Modifier symbols (accents)
+
+=back
+
+=back
+
 If the C<utf8> pragma is not used but the C<locale> pragma is, the
 classes correlate with the usual isalpha(3) interface (except for
 "word" and "blank").

[p5pRT]

From david@landgren.net

Robin Barker wrote​:

No progress in resolving this in code, so here is documentation patch.

Robin

--- pod/perlre.pod.orig 2008-01-30 20​:41​:06.000000000 +0000
+++ pod/perlre.pod
@​@​ -375,8 +375,8 @​@​
digit IsDigit \d
graph IsGraph
lower IsLower
- print IsPrint
- punct IsPunct
+ print IsPrint (but see 2. below)
+ punct IsPunct (but see 3. below)
space IsSpace
IsSpacePerl \s
upper IsUpper
@​@​ -385,6 +385,41 @​@​

For example C<[[​:lower​:]]> and C<\p{IsLower}> are equivalent.

+However, the equivalence between C<[[​:xxxxx​:]]> and C<\p{Xxxxx}> is not exact.
+
+=over 4
+
+=item 1.
+
+C<[[​:xxxxx​:]]> only matches characters in the range 0x00-0x7F.
+
+=item 2.
+
+C<\p{IsPrint}> matches characters 0x09-0x0d but C<[[​:print​:]]> does not.
+
+=item 3.
+
+C<[[​:punct​::]]> matches the following but C<\p{IsPunct}> does not,
+because they are classed as symbols in Unicode.

s/classed/classified/

considered?

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @Juerd

Robin Barker skribis 2008-03-31 21​:42 (+0100)​:

+C<[[​:xxxxx​:]]> only matches characters in the range 0x00-0x7F.

Not always true.

  juerd@​lanova​:~$ perl -CO -e'printf "[%s]\n", "foo\x{123}" =~ /([[​:print​:]]+)/'
  [fooģ]

See also Unicode​::Semantics on CPAN.
--
Met vriendelijke groet, Kind regards, Korajn salutojn,

  Juerd Waalboer​: Perl hacker <#####@​juerd.nl> <http​://juerd.nl/sig>
  Convolution​: ICT solutions and consultancy <sales@​convolution.nl>

[p5pRT]

From Robin.Barker@npl.co.uk

I've taken in to account comments about the last patch,
and increased my understanding of what is going on, and
present a revised documentation patch. With a patch to
t/op/pat.t to capture the current behaviour in a test.

Robin

diff -ur ../perl-current/pod/perlre.pod ./pod/perlre.pod
--- ../perl-current/pod/perlre.pod 2008-01-30 20​:41​:06.000000000 +0000
+++ ./pod/perlre.pod
@​@​ -375,20 +375,60 @​@​
  digit IsDigit \d
  graph IsGraph
  lower IsLower
- print IsPrint
- punct IsPunct
+ print IsPrint (but see [2] below)
+ punct IsPunct (but see [3] below)
  space IsSpace
  IsSpacePerl \s
  upper IsUpper
- word IsWord
+ word IsWord \w
  xdigit IsXDigit

For example C<[[​:lower​:]]> and C<\p{IsLower}> are equivalent.

+However, the equivalence between C<[[​:xxxxx​:]]> and C<\p{IsXxxxx}>
+is not exact.
+
+=over 4
+
+=item [1]
+
If the C<utf8> pragma is not used but the C<locale> pragma is, the
classes correlate with the usual isalpha(3) interface (except for
"word" and "blank").

+But if the C<locale> or C<encoding> pragmas are not used and
+the string is not C<utf8>, then C<[[​:xxxxx​:]]> (and C<\w>, etc.)
+will not match characters 0x80-0xff; whereas C<\p{IsXxxxx}> will
+force the string to C<utf8> and can match these characters
+(as Unicode).
+
+=item [2]
+
+C<\p{IsPrint}> matches characters 0x09-0x0d but C<[[​:print​:]]> does not.
+
+=item [3]
+
+C<[[​:punct​::]]> matches the following but C<\p{IsPunct}> does not,
+because they are classed as symbols (not punctuation) in Unicode.
+
+=over 4
+
+=item C<$>
+
+Currency symbol
+
+=item C<+> C<< < >> C<=> C<< > >> C<|> C<~>
+
+Mathematical symbols
+
+=item C<^> C<`>
+
+Modifier symbols (accents)
+
+=back
+
+=back
+
The other named classes are​:

=over 4
diff -ur ../perl-current/t/op/pat.t ./t/op/pat.t
--- ../perl-current/t/op/pat.t 2008-04-15 13​:46​:40.000000000 +0100
+++ ./t/op/pat.t
@​@​ -4604,6 +4604,32 @​@​
  iseq($te[0], '../');
}

+SKIP​: {
+ if (ordA == 193) { skip("Assumes ASCII", 4) }
+
+ my @​notIsPunct = grep {/[[​:punct​:]]/ and not /\p{IsPunct}/}
+ map {chr} 0x20..0x7f;
+ iseq( join('', @​notIsPunct), '$+<=>^`|~',
+ '[​:punct​:] disagress with IsPunct on Symbols');
+
+ my @​isPrint = grep {not/[[​:print​:]]/ and /\p{IsPrint}/}
+ map {chr} 0..0x1f, 0x7f..0x9f;
+ iseq( join('', @​isPrint), "\x09\x0a\x0b\x0c\x0d\x85",
+ 'IsPrint disagrees with [​:print​:] on control characters');
+
+ my @​isPunct = grep {/[[​:punct​:]]/ != /\p{IsPunct}/}
+ map {chr} 0x80..0xff;
+ iseq( join('', @​isPunct), "\xa1\xab\xb7\xbb\xbf", # ¡ « · » ¿
+ 'IsPunct disagrees with [​:punct​:] outside ASCII');
+
+ my @​isPunctLatin1 = eval q{
+ use encoding 'latin1';
+ grep {/[[​:punct​:]]/ != /\p{IsPunct}/} map {chr} 0x80..0xff;
+ };
+ if( $@​ ){ skip( $@​, 1); }
+ iseq( join('', @​isPunctLatin1), '',
+ 'IsPunct agrees with [​:punct​:] with explicit Latin1');
+}

# Test counter is at bottom of file. Put new tests above here.
@​@​ -4667,7 +4693,7 @​@​

# Don't forget to update this!
BEGIN {
- $​::TestCount = 4031;
+ $​::TestCount = 4035;
  print "1..$​::TestCount\n";
}

[p5pRT]

From Robin.Barker@npl.co.uk

perlre-3.patch

diff -ur ../perl-current/pod/perlre.pod ./pod/perlre.pod
--- ../perl-current/pod/perlre.pod	2008-01-30 20:41:06.000000000 +0000
+++ ./pod/perlre.pod
@@ -375,20 +375,60 @@
     digit       IsDigit        \d
     graph       IsGraph
     lower       IsLower
-    print       IsPrint
-    punct       IsPunct
+    print       IsPrint		(but see [2] below)
+    punct       IsPunct		(but see [3] below)
     space       IsSpace
                 IsSpacePerl    \s
     upper       IsUpper
-    word        IsWord
+    word        IsWord         \w
     xdigit      IsXDigit
 
 For example C<[[:lower:]]> and C<\p{IsLower}> are equivalent.
 
+However, the equivalence between C<[[:xxxxx:]]> and C<\p{IsXxxxx}>
+is not exact.
+
+=over 4
+
+=item [1]
+
 If the C<utf8> pragma is not used but the C<locale> pragma is, the
 classes correlate with the usual isalpha(3) interface (except for
 "word" and "blank").
 
+But if the C<locale> or C<encoding> pragmas are not used and
+the string is not C<utf8>, then C<[[:xxxxx:]]> (and C<\w>, etc.)
+will not match characters 0x80-0xff; whereas C<\p{IsXxxxx}> will
+force the string to C<utf8> and can match these characters
+(as Unicode).
+
+=item [2]
+
+C<\p{IsPrint}> matches characters 0x09-0x0d but C<[[:print:]]> does not.
+
+=item [3]
+
+C<[[:punct::]]> matches the following but C<\p{IsPunct}> does not,
+because they are classed as symbols (not punctuation) in Unicode.
+
+=over 4
+
+=item C<$>
+
+Currency symbol
+
+=item C<+> C<< < >> C<=> C<< > >> C<|> C<~>
+
+Mathematical symbols
+
+=item C<^> C<`>
+
+Modifier symbols (accents)
+
+=back
+
+=back
+
 The other named classes are:
 
 =over 4
diff -ur ../perl-current/t/op/pat.t ./t/op/pat.t
--- ../perl-current/t/op/pat.t	2008-04-15 13:46:40.000000000 +0100
+++ ./t/op/pat.t
@@ -4604,6 +4604,32 @@
     iseq($te[0], '../');
 }
 
+SKIP: {
+    if (ordA == 193) { skip("Assumes ASCII", 4) }
+
+    my @notIsPunct = grep {/[[:punct:]]/ and not /\p{IsPunct}/}
+			map {chr} 0x20..0x7f;
+    iseq( join('', @notIsPunct), '$+<=>^`|~',
+	'[:punct:] disagress with IsPunct on Symbols');
+
+    my @isPrint = grep {not/[[:print:]]/ and /\p{IsPrint}/}
+			map {chr} 0..0x1f, 0x7f..0x9f;
+    iseq( join('', @isPrint), "\x09\x0a\x0b\x0c\x0d\x85",
+	'IsPrint disagrees with [:print:] on control characters');
+
+    my @isPunct = grep {/[[:punct:]]/ != /\p{IsPunct}/}
+			map {chr} 0x80..0xff;
+    iseq( join('', @isPunct), "\xa1\xab\xb7\xbb\xbf",		# �� �� �� �� ��
+	'IsPunct disagrees with [:punct:] outside ASCII');
+
+    my @isPunctLatin1 = eval q{
+	use encoding 'latin1';
+	grep {/[[:punct:]]/ != /\p{IsPunct}/} map {chr} 0x80..0xff;
+    };
+    if( $@ ){ skip( $@, 1); }
+    iseq( join('', @isPunctLatin1), '', 
+	'IsPunct agrees with [:punct:] with explicit Latin1');
+} 
 
 
 # Test counter is at bottom of file. Put new tests above here.
@@ -4667,7 +4693,7 @@
 
 # Don't forget to update this!
 BEGIN {
-    $::TestCount = 4031;
+    $::TestCount = 4035;
     print "1..$::TestCount\n";
 }
 

[p5pRT]

From @rgs

2008/4/25 Robin Barker <Robin.Barker@​npl.co.uk>​:

I've taken in to account comments about the last patch,
and increased my understanding of what is going on, and
present a revised documentation patch. With a patch to
t/op/pat.t to capture the current behaviour in a test.

Thanks, applied as #33752

[p5pRT]

@rgs - Status changed from 'open' to 'resolved'

[p5pRT]

From @khwilliamson

This is a bug report for perl from public@​khwilliamson.com,
generated with the help of perlbug 1.39 running under perl 5.11.0.

---

use utf8;

print '©' =~ /[[​:graph​:]]/, "\n";
print '©' =~ /[[​:^graph​:]]/, "\n";

both print 1. This happens for various posix classes, and various
utf8 characters. Also for perl 5.8

---

---

Flags​:
  category=core
  severity=medium

---

This perlbug was built using Perl 5.11.0 - Wed Oct 22 19​:16​:44 MDT 2008
It is being executed now by Perl 5.11.0 - Fri Oct 24 11​:08​:58 MDT 2008.

Site configuration information for perl 5.11.0​:

Configured by khw at Fri Oct 24 11​:08​:58 MDT 2008.

Summary of my perl5 (revision 5 version 11 subversion 0 patch 34566)
configuration​:
  Platform​:
  osname=linux, osvers=2.6.24-21-generic, archname=i686-linux
  uname='linux karl 2.6.24-21-generic #1 smp mon aug 25 17​:32​:09 utc
2008 i686 gnulinux '
  config_args='-d -Dmksymlinks -Dprefix=/home/khw/myperl -Dusedevel
-DDEBUGGING=both'
  hint=previous, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64'
  ccversion='', gccversion='4.2.3 (Ubuntu 4.2.3-2ubuntu7)',
gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
  libc=/lib/libc-2.7.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.7'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib
-fstack-protector'

Locally applied patches​:
  DEVEL

---

@​INC for perl 5.11.0​:
  /home/khw/myperl/lib/5.11.0/i686-linux
  /home/khw/myperl/lib/5.11.0
  /home/khw/myperl/lib/site_perl/5.11.0/i686-linux
  /home/khw/myperl/lib/site_perl/5.11.0
  .

---

Environment for perl 5.11.0​:
  HOME=/home/khw
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)

PATH=/home/khw/bin​:/home/khw/print/bin​:/bin​:/usr/local/sbin​:/usr/local/bin​:/usr/sbin​:/usr/bin​:/sbin​:/usr/games​:/home/khw/cxoffice/bin
  PERL_BADLANG (unset)
  SHELL=/bin/ksh

[p5pRT]

From @khwilliamson

Tom Christiansen wrote​:

In-Reply-To​: Message from karl williamson <public@​khwilliamson.com>
of "Wed, 29 Oct 2008 10​:21​:33 MDT." <49088D8D.3050202@​khwilliamson.com>

Tom Christiansen wrote​:

PRESCRIPT​: Karl, my program at the end below should be good for
sniffing out \p and \P mutual-exclusion failure bugs
such as I believe you recently reported.

Thanks for the program. My bug reports have mostly come from
running something similar, but not with nearly as many of the
classes as you did.

I'm guessing you didn't run it past 127, because it fails immediately
with a Malformed UTF-8 character (fatal) at (eval 3) line 2.

Yes, that's because it intentionally assumed ASCII only. The amended
program and results follow. You should now be to use this program
directly for your testing.

I actually don't think there are any issues with the \p and \P,
because those go out and use auto-constructed files. The
problems I have found have been in the posix classes and
entirely in the 128-255 range.

It's not just there. It's a bug in negating of POSIX char classes.
I can elicit errors at codepoints <128, even with Unicode semantics on.

Specifically, just runniing POSIX charclass tests alone​:

REPORT​: ranging from U\+00 \(0\) \.\. U\+00007F \(127\) \[128 codepoints\]
failed 28 tests\, 3300 out of 3328 tests successful \(0\.991587%\)

The problems are with the [​:print​:] and [​:punct​:] properties​:

Trouble w/U\+007E​: TILDE​: Property "\[​:punct​:\]" failed
Trouble w/U\+007C​: VERTICAL LINE​: Property "\[​:punct​:\]" failed
Trouble w/U\+0060​: GRAVE ACCENT​: Property "\[​:punct​:\]" failed
Trouble w/U\+005E​: CIRCUMFLEX ACCENT​: Property "\[​:punct​:\]" failed
Trouble w/U\+003E​: GREATER\-THAN SIGN​: Property "\[​:punct​:\]" failed
Trouble w/U\+003D​: EQUALS SIGN​: Property "\[​:punct​:\]" failed
Trouble w/U\+003C​: LESS\-THAN SIGN​: Property "\[​:punct​:\]" failed
Trouble w/U\+002B​: PLUS SIGN​: Property "\[​:punct​:\]" failed
Trouble w/U\+0024​: DOLLAR SIGN​: Property "\[​:punct​:\]" failed
Trouble w/U\+000D​: CARRIAGE RETURN \(CR\)​: Property "\[​:print​:\]" failed
Trouble w/U\+000C​: FORM FEED \(FF\)​: Property "\[​:print​:\]" failed
Trouble w/U\+000B​: LINE TABULATION​: Property "\[​:print​:\]" failed
Trouble w/U\+000A​: LINE FEED \(LF\)​: Property "\[​:print​:\]" failed
Trouble w/U\+0009​: CHARACTER TABULATION​: Property "\[​:print​:\]" failed

The positive char class test is like this​:

(
( $U_char =~ /\A[[​:print​:]]\z/ )
==
( $U_char !~ /\A[[​:^print​:]]\z/ )
)
&&
(
( $U_char =~ /\A[[​:print​:]]\z/ )
==
( $U_char !~ /\A[^[​:print​:]]\z/ )
)

The negative char class test is this​:

\( $U\_char =~ /\\A\[\[​:^punct​:\]\]\\z/ \)
          ==
\( $U\_char =~ /\\A\[^\[​:punct​:\]\]\\z/ \)

...

I wondered why you were getting errors when my own (which I thought were
rather extensive) test cases were getting none in the ASCII range. It
turns out that it's because I hadn't thought to try the negation ^ in
the outer set of braces. Also, these problems don't occur when the
characters aren't utf8 (when they are packed as C instead of U). And
they fail in exactly the places that the posix classes don't match the
unicode ones. It is documented that [​:graph​:] includes the precise set
of symbols that have failures that \p{IsGraph} does not. Similarly for
[​:print​:] and the ones it has failures for.

I haven't looked at the code, but what likely what is going on is that
the complement of a complement loses these differences from unicode
(only when the utf8 flag is on, because otherwise the unicode classes
aren't looked at)

My goal is to fix all these problems in the 128-255 range. It's turning
out to be more work than I thought, essentially because there are a lot
of pre-existing errors and inconsistencies.

[p5pRT]

From @tonycoz

On Sun, Oct 26, 2008 at 03​:32​:53PM -0700, karl williamson wrote​:

-----------------------------------------------------------------
use utf8;

print '©' =~ /[[​:graph​:]]/, "\n";
print '©' =~ /[[​:^graph​:]]/, "\n";

both print 1. This happens for various posix classes, and various
utf8 characters. Also for perl 5.8
-----------------------------------------------------------------

This problem also occurs for \s vs \S inside a character class​:

sh-3.1$ /perl/blead34613/bin/perl5.11.0 -le '$x = chr(0xa0).chr(0x100); chop $x; print $x = /[\s]/; print $x =~ /[\S]/;'
1
1
sh-3.1$

The problem is that the character classes are compiled to a
combination of a bitmap and a Unicode property check, eg​:

[\s] becomes ANYOF[\11\12\14\15 +utf8​::IsSpacePerl]
[\S] becomes ANYOF[\0-\10\13\16-\37!-\377!utf8​::IsSpacePerl]

and similarly for [[​:foo​:]]​:

[[​:graph​:]] becomes ANYOF[!-~+utf8​::IsGraph]

[[​:^graph​:]] becomes ANYOF[\0- \177-\377!utf8​::IsGraph]

and for UTF scalars both the bitmap and the unicode property is
checked (S_reginclass in regexec.c).

The bitmap is generated using isSPACE, isGRAPH, etc, which without a
locale, returns false for characters \200-\x377.

I suspect a solution is going to involve not generating the bitmap for
named classes classes in character classes and using ANYOF_SPACE,
ANYOF_GRAPH etc in the character class node.

But I don't understand enough of the regular expression engine to want
to delve that deeply.

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @demerphq

2008/11/3 Tony Cook <tony@​develop-help.com>​:

On Sun, Oct 26, 2008 at 03​:32​:53PM -0700, karl williamson wrote​:

-----------------------------------------------------------------
use utf8;

print '(c)' =~ /[[​:graph​:]]/, "\n";
print '(c)' =~ /[[​:^graph​:]]/, "\n";

both print 1. This happens for various posix classes, and various
utf8 characters. Also for perl 5.8
-----------------------------------------------------------------

This problem also occurs for \s vs \S inside a character class​:

sh-3.1$ /perl/blead34613/bin/perl5.11.0 -le '$x = chr(0xa0).chr(0x100); chop $x; print $x = /[\s]/; print $x =~ /[\S]/;'
1
1
sh-3.1$

The problem is that the character classes are compiled to a
combination of a bitmap and a Unicode property check, eg​:

[\s] becomes ANYOF[\11\12\14\15 +utf8​::IsSpacePerl]
[\S] becomes ANYOF[\0-\10\13\16-\37!-\377!utf8​::IsSpacePerl]

and similarly for [[​:foo​:]]​:

[[​:graph​:]] becomes ANYOF[!-~+utf8​::IsGraph]

[[​:^graph​:]] becomes ANYOF[\0- \177-\377!utf8​::IsGraph]

and for UTF scalars both the bitmap and the unicode property is
checked (S_reginclass in regexec.c).

The bitmap is generated using isSPACE, isGRAPH, etc, which without a
locale, returns false for characters \200-\x377.

I suspect a solution is going to involve not generating the bitmap for
named classes classes in character classes and using ANYOF_SPACE,
ANYOF_GRAPH etc in the character class node.

But I don't understand enough of the regular expression engine to want
to delve that deeply.

Tony

I expected the attached patch to work out. Unfortunately it doesnt. I
get the following failures when i do a

make test-reonly

Test Summary Report

---

op/pat (Wstat​: 0 Tests​: 4035 Failed​: 5)
  Failed tests​: 1871, 1901-1904
op/regexp (Wstat​: 0 Tests​: 1359 Failed​: 4)
  Failed tests​: 603, 608, 611, 1310
op/regexp_noamp (Wstat​: 0 Tests​: 1359 Failed​: 4)
  Failed tests​: 603, 608, 611, 1310
op/regexp_notrie (Wstat​: 0 Tests​: 1359 Failed​: 4)
  Failed tests​: 603, 608, 611, 1310
op/regexp_qr (Wstat​: 0 Tests​: 1359 Failed​: 4)
  Failed tests​: 603, 608, 611, 1310
op/regexp_qr_embed (Wstat​: 0 Tests​: 1359 Failed​: 4)
  Failed tests​: 603, 608, 611, 1310
op/regexp_trielist (Wstat​: 0 Tests​: 1359 Failed​: 4)
  Failed tests​: 603, 608, 611, 1310
Files=29, Tests=21383, 59 wallclock secs ( 8.70 usr 0.22 sys + 49.73
cusr 0.76 csys = 59.41 CPU)
Result​: FAIL
Failed 7/29 test programs. 29/21383 subtests failed.

Which seems quite odd, like the swash logic is failing somehow, in
ways that I personally find surprising.

The issue here is that the bitmaps are constructed for non-utf8 and
utf8 semantics differ for utf8 and non-utf8 in the codepoint range
128-255. The logic ive added is to only use the bitmaps for codepoints
0..127 when doutf8 and then rely on the unicode charclass "swash"
logic to handle the rest, but this fails. I havent had sufficient
time, not access to a debugger i know well enough* to figure out why.

However im still poking.

Yves
* Id love to attend a 'gdb for perl core hackers' course should
someone put one together for the next YAPC​::EU/ or YAPC Europe Perl
Workshop. *HINT* HINT. :-)

--
perl -Mre=debug -e "/just|another|perl|hacker/"