Inconsistent and wrong handling of 8th bit set chars with no locale

[p5pRT]

Migrated from rt.perl.org#58182 (status was 'resolved')

Searchable as RT58182$

[p5pRT]

From @khwilliamson

This is a bug report for perl from corporate@​khwilliamson.com,
generated with the help of perlbug 1.36 running under perl 5.10.0.

---

Characters in the range U+0080 through U+00FF behave inconsistently
depending on whether or not they are part of a string which also
includes a character above that range, and in some cases they behave
incorrectly even when part of such a string. The problems I will
concentrate on in this report are those involving case.

I presume that they do work properly when a locale is set, but I haven't
tested that.

print uc("\x{e0}"), "\n"; # (a with grave accent)

yields itself instead of a capital A with grave accent (U+00C0). This
is true whether or not the character is part of a string which includes
a character not storable in a single byte. Similarly

print "\x{e0}" =~ /\x{c0}/i, "\n";

will print a null string on a line, as the match fails.

The same behavior occurs for all characters in this range that are
marked in the Unicode standard as lower case and have single letter
upper case equivalents.

The behavior that is inconsistent mostly occurs with upper case letters
being mapped to lower case.

print lcfirst("\x{c0}aaaaa"), "\n";

doesn't change the first character. But

print lcfirst("\x{c0}aaaaa\x{101}"), "\n";

does change it. There is something seriously wrong when a character
separated by an arbitrarily large distance from another one can affect
what case the latter is considered to be. Similarly,

print "\x{c0}aaaaaa" =~ /^\x{e0}/i, "\n";

will show the match failing, but

print "\x{c0}aaaaaa\x{101}" =~ /^\x{e0}/i, "\n";

will show the match succeeding. Again a character maybe hundreds of
positions further along in a string can affect whether the first
character in said string matches its lower case equivalent when case is
ignored.

The same behavior occurs for all characters in this range that are
marked in the Unicode standard as upper case and have lower case
equivalents, as well as U+00DF which is lower case and has an upper case
equivalent of the string 'SS'.

Also, the byte character classes inconsistently match characters in this
range, again depending on whether or not the character is part of a
larger string that contains a character greater than the range. So, for
example, for a non-breaking space,

print "\xa0" =~ /^\s/, "\n";

will show that the match returns false but

print "\xa0\x{101}" =~ /^\s/, "\n";

will show that the match returns true. But this behavior is sort-of
documented, and there is a work-around, which is to use the '\p{}'
classes instead. Note that calling them byte character classes is
wrong; they really are 7-bit classes.

From reading the documentation, I presume that the inconsistent behavior
is a result of the decision to have perl not switch to wide-character
mode in storing its strings unless necessary. I like that decision for
efficiency reasons. But what has happened is that the code points in
the range 128 - 255 have been orphaned, when they aren't part of strings
that force the switch. Again, I presume but haven't tested, that using
a locale causes them to work properly for that locale, but in the
absence of a locale they should be treated as Unicode code points (or
equivalently for characters in this range, as iso-8859-1). Storing as
wide-characters is supposed to be transparent to users, but this bug
belies that and yields very inconsistent and unexpected behavior.
(This doesn't explain the lower to upper case translation bug, which is
wrong even in wide-character mode.)

I am frankly astonished that this bug exists, as I have come to expect
perl to "Do the Right Thing" over the course of many years of using it.
I did see one bug report of something similar to this when searching for
this, but it apparently was misunderstood and went nowhere, and wasn't
in the perl bug data base

---

---

Flags​:
  category=core
  severity=high

---

Site configuration information for perl 5.10.0​:

Configured by ActiveState at Wed May 14 05​:06​:16 PDT 2008.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration​:
  Platform​:
  osname=linux, osvers=2.4.21-297-default,
archname=i686-linux-thread-multi
  uname='linux gila 2.4.21-297-default #1 sat jul 23 07​:47​:39 utc
2005 i686 i686 i386 gnulinux '
  config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads
-Ud_sigsetjmp -Uinstallusrbinperl -Ulocincpth= -Uloclibpth=
-Accflags=-DUSE_SITECUSTOMIZE -Duselargefiles
-Accflags=-DPRIVLIB_LAST_IN_INC -Dprefix=/opt/ActivePerl-5.10
-Dprivlib=/opt/ActivePerl-5.10/lib -Darchlib=/opt/ActivePerl-5.10/lib
-Dsiteprefix=/opt/ActivePerl-5.10/site
-Dsitelib=/opt/ActivePerl-5.10/site/lib
-Dsitearch=/opt/ActivePerl-5.10/site/lib -Dsed=/bin/sed -Duseshrplib
-Dcf_by=ActiveState -Dcf_email=support@​ActiveState.com'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DUSE_SITECUSTOMIZE -DPRIVLIB_LAST_IN_INC -fno-strict-aliasing -pipe
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DUSE_SITECUSTOMIZE -DPRIVLIB_LAST_IN_INC -fno-strict-aliasing -pipe'
  ccversion='', gccversion='3.3.1 (SuSE Linux)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =''
  libpth=/lib /usr/lib /usr/local/lib
  libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.3.2'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/opt/ActivePerl-5.10/lib/CORE'
  cccdlflags='-fPIC', lddlflags='-shared -O2'

Locally applied patches​:
  ACTIVEPERL_LOCAL_PATCHES_ENTRY
  33741 avoids segfaults invoking S_raise_signal() (on Linux)
  33763 Win32 process ids can have more than 16 bits
  32809 Load 'loadable object' with non-default file extension
  32728 64-bit fix for Time​::Local

---

@​INC for perl 5.10.0​:
  /opt/ActivePerl-5.10/site/lib
  /opt/ActivePerl-5.10/lib
  .

---

Environment for perl 5.10.0​:
  HOME=/home/khw
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)

PATH=/opt/ActivePerl-5.10/bin​:/home/khw/bin​:/home/khw/print/bin​:/bin​:/usr/local/sbin​:/usr/local/bin​:/usr/sbin​:/usr/bin​:/sbin​:/usr/games​:/home/khw/cxoffice/bin
  PERL_BADLANG (unset)
  SHELL=/bin/ksh

[p5pRT]

From @moritz

karl williamson wrote​:

# New Ticket Created by karl williamson
# Please include the string​: [perl #58182]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=58182 >

This is a bug report for perl from corporate@​khwilliamson.com,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
Characters in the range U+0080 through U+00FF behave inconsistently
depending on whether or not they are part of a string which also
includes a character above that range, and in some cases they behave
incorrectly even when part of such a string. The problems I will
concentrate on in this report are those involving case.

I presume that they do work properly when a locale is set, but I haven't
tested that.

print uc("\x{e0}"), "\n"; # (a with grave accent)

yields itself instead of a capital A with grave accent (U+00C0). This
is true whether or not the character is part of a string which includes
a character not storable in a single byte. Similarly

This is a known bug, and probably not fixable, because too much code
depends on it. See http​://search.cpan.org/perldoc?Unicode​::Semantics

A possible workaround is
my $x = "\x{e0}"; utf8​::upgrade($x); say uc($x); # yields À

CHeers,
Moritz

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @druud62

karl williamson schreef​:

The behavior that is inconsistent mostly occurs with upper case
letters being mapped to lower case.

print lcfirst("\x{c0}aaaaa"), "\n";

doesn't change the first character. But

print lcfirst("\x{c0}aaaaa\x{101}"), "\n";

does change it.

To me that is as expected.

print lcfirst substr "\x{100}\x{c0}aaaaa", 1;

Lowercasing isn't defined for as many characters in ASCII or Latin-1 as
it is in Unicode.
Unicode semantics get activated when a codepoint above 255 is involved.

--
Affijn, Ruud

"Gewoon is een tijger."

[p5pRT]

From @nothingmuch

On Thu, Aug 21, 2008 at 13​:22​:36 +0200, Dr.Ruud wrote​:

Unicode semantics get activated when a codepoint above 255 is involved.

Or a code point above 127 with use utf8 or use encoding

--
  Yuval Kogman <nothingmuch@​woobling.org>
http​://nothingmuch.woobling.org 0xEBD27418

[p5pRT]

From @nothingmuch

On Thu, Aug 21, 2008 at 14​:31​:40 +0300, Yuval Kogman wrote​:

Or a code point above 127 with use utf8 or use encoding

I should clarify that this is only in the context of the string
constants.

A code point above 127 will be treated as unicode if the string is
properly marked as such, and the way to achieve that for string
constants is 'use utf8'.

--
  Yuval Kogman <nothingmuch@​woobling.org>
http​://nothingmuch.woobling.org 0xEBD27418

[p5pRT]

From @khwilliamson

I'm the person who submitted this bug report. I think this bug should
be fixed in Perl 5, and I'm volunteering to do it. Towards that end, I
downloaded the Perl 5.10 source and hacked up an experimental version
that seems to fix it. And now I've joined this list to see how to
proceed. I don't know the protocol involved, so I'll just jump in, and
hopefully that will be all right.

To refresh your memory, the current implementation of perl on non-EBCDIC
machines is problematic for characters in the range 128-255 when no
locale is set.

The slides from the talk "Working around *the* Unicode bug" during
YAPC​::Europe 2007 in Vienna​:
http​://juerd.nl/files/slides/2007yapceu/unicodesemantics.html
give more cases of problems than were in my bug report.

The crux of the problem is that on non-EBCDIC machines, in the absence
of locale, in order to have meaningful semantics, a character (or code
point) has to be stored in utf8, except in pattern matching the \h, \H,
\v and \V or any of the \p{} patterns. (This leads to an anomaly with
the no-break space which is considered to be horizontal space (\h), but
not space (\s).) (The characters also always have base semantics of
having an ordinal number, and also of being not-a-anything (meaning that
they all pattern match \W, \D, \S, [[​:^punct]], etc.))

Perl stores characters as utf8 automatically if a string contains any
code points above 255, and it is trivially true for ascii code points.
That leaves a hole-in-the-doughnut of characters between 128 and 255
with behavior that varies depending on whether they are stored as utf8
or not. This is contrary, for example, to the Camel book​: "character
semantics are preserved at an abstract level regardless of
representation" (p.403). (How they get stored depends on how they were
input, or whether or not they are part of a longer string containing
code points larger than 255, or if they have been explicitly set by
using utf8​::upgrade or utf8​::downgrade.)

I know of three areas where this leads to problems.

The first is the pattern matching already alluded to. This is at least
documented (though somewhat confusingly). And one can use the \p{}
constructs to avoid the issue.

The second is case changing functions, like lcfirst() or \U in pattern
substitutions.

And the third is ignoring case in pattern matches.

There may be others which I haven't looked for yet. I think, for
example, that quotemeta() will escape all these characters, though I
don't believe that this causes a real problem.

One response I got to my bug report was that a lot of code depends on
things working the way they currently do. I'm wondering if that applies
to all three of the areas, or just the first?

Also, from reading the perl source, it appears to me that EBCDIC
machines may work differently (and more correctly to my way of thinking)
than Ascii-ish ones.

An idea I've had is to add a pragma like "use latin1", or maybe "use
locale unicode", or something else as a way of not breaking existing
application code.

Anyway, I'm hoping to get some sort of fix in for this. In my
experimental implementation (which currently doesn't change EBCDIC
handling), it is mostly just extending the existing definitions of ascii
semantics to include the 128..255 latin1 range. Code logic changes were
required only in the uc and ucfirst functions (to accommodate 3
characters which require special handling), and in the regular
expression compilation (to accommodate 2 characters which need special
handling). Obviously, in my ignorance, I may be missing things that
others can enlighten me on.

So I'd like to know how to proceed

Karl Williamson

[p5pRT]

From perl@nevcal.com

On approximately 9/20/2008 3​:52 PM, came the following characters from
the keyboard of karl williamson​:

I'm the person who submitted this bug report. I think this bug should
be fixed in Perl 5, and I'm volunteering to do it. Towards that end, I
downloaded the Perl 5.10 source and hacked up an experimental version
that seems to fix it. And now I've joined this list to see how to
proceed. I don't know the protocol involved, so I'll just jump in, and
hopefully that will be all right.

To refresh your memory, the current implementation of perl on non-EBCDIC
machines is problematic for characters in the range 128-255 when no
locale is set.

The slides from the talk "Working around *the* Unicode bug" during
YAPC​::Europe 2007 in Vienna​:
http​://juerd.nl/files/slides/2007yapceu/unicodesemantics.html
give more cases of problems than were in my bug report.

The crux of the problem is that on non-EBCDIC machines, in the absence
of locale, in order to have meaningful semantics, a character (or code
point) has to be stored in utf8, except in pattern matching the \h, \H,
\v and \V or any of the \p{} patterns. (This leads to an anomaly with
the no-break space which is considered to be horizontal space (\h), but
not space (\s).) (The characters also always have base semantics of
having an ordinal number, and also of being not-a-anything (meaning that
they all pattern match \W, \D, \S, [[​:^punct]], etc.))

Perl stores characters as utf8 automatically if a string contains any
code points above 255, and it is trivially true for ascii code points.
That leaves a hole-in-the-doughnut of characters between 128 and 255
with behavior that varies depending on whether they are stored as utf8
or not. This is contrary, for example, to the Camel book​: "character
semantics are preserved at an abstract level regardless of
representation" (p.403). (How they get stored depends on how they were
input, or whether or not they are part of a longer string containing
code points larger than 255, or if they have been explicitly set by
using utf8​::upgrade or utf8​::downgrade.)

I know of three areas where this leads to problems.

The first is the pattern matching already alluded to. This is at least
documented (though somewhat confusingly). And one can use the \p{}
constructs to avoid the issue.

The second is case changing functions, like lcfirst() or \U in pattern
substitutions.

And the third is ignoring case in pattern matches.

There may be others which I haven't looked for yet. I think, for
example, that quotemeta() will escape all these characters, though I
don't believe that this causes a real problem.

One response I got to my bug report was that a lot of code depends on
things working the way they currently do. I'm wondering if that applies
to all three of the areas, or just the first?

Also, from reading the perl source, it appears to me that EBCDIC
machines may work differently (and more correctly to my way of thinking)
than Ascii-ish ones.

An idea I've had is to add a pragma like "use latin1", or maybe "use
locale unicode", or something else as a way of not breaking existing
application code.

Anyway, I'm hoping to get some sort of fix in for this. In my
experimental implementation (which currently doesn't change EBCDIC
handling), it is mostly just extending the existing definitions of ascii
semantics to include the 128..255 latin1 range. Code logic changes were
required only in the uc and ucfirst functions (to accommodate 3
characters which require special handling), and in the regular
expression compilation (to accommodate 2 characters which need special
handling). Obviously, in my ignorance, I may be missing things that
others can enlighten me on.

So I'd like to know how to proceed

Karl Williamson

I applaud your willingness to dive in.

For compatibility reasons, as has been discussed on this list
previously, a pragma of some sort must be used to request the
incompatible enhancement (which you call a fix).

N.B. There are lots of discussions about it in the archive, some
recently, if you haven't found them, you should; if you find it hard to
find them, ask, and I (or someone) will try to find the starting points
for you, perhaps the summaries would be a good place to look to find the
discussions; I participated in most of them.

Those discussions are lengthy reading, unfortunately, but they do point
out an extensive list of issues, perhaps approaching completeness.

--
Glenn -- http​://nevcal.com/

A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

[p5pRT]

From @andk

On Sat, 20 Sep 2008 16​:52​:02 -0600, karl williamson <contact@​khwilliamson.com> said​:

  > I'm the person who submitted this bug report. I think this bug should
  > be fixed in Perl 5, and I'm volunteering to do it. Towards that end, I
  > downloaded the Perl 5.10 source and hacked up an experimental version
  > that seems to fix it. And now I've joined this list to see how to
  > proceed. I don't know the protocol involved, so I'll just jump in, and
  > hopefully that will be all right.

Thank you! As for the protocol​: do not patch 5.10, patch bleadperl
instead.

--
andreas

[p5pRT]

From @rgs

2008/9/21 karl williamson <contact@​khwilliamson.com>​:

The crux of the problem is that on non-EBCDIC machines, in the absence
of locale, in order to have meaningful semantics, a character (or code
point) has to be stored in utf8, except in pattern matching the \h, \H, \v
and \V or any of the \p{} patterns. (This leads to an anomaly with the
no-break space which is considered to be horizontal space (\h), but not
space (\s).) (The characters also always have base semantics of having an
ordinal number, and also of being not-a-anything (meaning that they all
pattern match \W, \D, \S, [[​:^punct]], etc.))

Perl stores characters as utf8 automatically if a string contains any
code points above 255, and it is trivially true for ascii code points.
That leaves a hole-in-the-doughnut of characters between 128 and 255
with behavior that varies depending on whether they are stored as utf8
or not. This is contrary, for example, to the Camel book​: "character
semantics are preserved at an abstract level regardless of
representation" (p.403). (How they get stored depends on how they were
input, or whether or not they are part of a longer string containing
code points larger than 255, or if they have been explicitly set by
using utf8​::upgrade or utf8​::downgrade.)

I know of three areas where this leads to problems.

The first is the pattern matching already alluded to. This is at least
documented (though somewhat confusingly). And one can use the \p{}
constructs to avoid the issue.

The second is case changing functions, like lcfirst() or \U in pattern
substitutions.

And the third is ignoring case in pattern matches.

There may be others which I haven't looked for yet. I think, for
example, that quotemeta() will escape all these characters, though I
don't believe that this causes a real problem.

This is a good summary of the issues.

One response I got to my bug report was that a lot of code depends on
things working the way they currently do. I'm wondering if that applies
to all three of the areas, or just the first?

In general, one finds that people write code relying on almost anything...

Also, from reading the perl source, it appears to me that EBCDIC
machines may work differently (and more correctly to my way of thinking)
than Ascii-ish ones.

That's in theory probable, but we don't have testers on EBCDIC
machines those days...

An idea I've had is to add a pragma like "use latin1", or maybe "use
locale unicode", or something else as a way of not breaking existing
application code.

I think that the current Unicode bugs are annoying enough to deserve
an incompatible change in perl 5.12.
However, for perl 5.10.x, something could be added to switch to a more
correct behaviour, if possible without slowing everything down...

Anyway, I'm hoping to get some sort of fix in for this. In my
experimental implementation (which currently doesn't change EBCDIC
handling), it is mostly just extending the existing definitions of ascii
semantics to include the 128..255 latin1 range. Code logic changes were
required only in the uc and ucfirst functions (to accommodate 3
characters which require special handling), and in the regular
expression compilation (to accommodate 2 characters which need special
handling). Obviously, in my ignorance, I may be missing things that
others can enlighten me on.

So I'd like to know how to proceed

If you're a git user, you can work on a branch cloned from
git​://perl5.git.perl.org/perl.git

Do not hesitate to ask questions here.

[p5pRT]

From @Juerd

Moritz Lenz skribis 2008-08-21 9​:50 (+0200)​:

This is a known bug, and probably not fixable, because too much code
depends on it.

It is fixable, and the backwards incompatibility has already been
announced in perl5100delta​:

| The handling of Unicode still is unclean in several places, where it's
| dependent on whether a string is internally flagged as UTF-8. This will
| be made more consistent in perl 5.12, but that won't be possible without
| a certain amount of backwards incompatibility.

It will be fixed, and it's wonderful to have a volunteer for that!
--
Met vriendelijke groet, Kind regards, Korajn salutojn,

  Juerd Waalboer​: Perl hacker <#####@​juerd.nl> <http​://juerd.nl/sig>
  Convolution​: ICT solutions and consultancy <sales@​convolution.nl>
1;

[p5pRT]

From @Juerd

Dr.Ruud skribis 2008-08-21 13​:22 (+0200)​:

Unicode semantics get activated when a codepoint above 255 is involved.

No, unicode semantics get activated when the internal encoding of the
string is utf8, even if it contains no character above 255, and even if
it only contains ASCII characters.

It's a bug. A known and old bug, but it must be fixed some time.
--
Met vriendelijke groet, Kind regards, Korajn salutojn,

  Juerd Waalboer​: Perl hacker <#####@​juerd.nl> <http​://juerd.nl/sig>
  Convolution​: ICT solutions and consultancy <sales@​convolution.nl>
1;

[p5pRT]

From @Juerd

karl williamson skribis 2008-09-20 16​:52 (-0600)​:

One response I got to my bug report was that a lot of code depends on
things working the way they currently do. I'm wondering if that applies
to all three of the areas, or just the first?

All three, but rest assured that this has already been discussed in
great detail, and that the pumpking's decision was that backwards
incompatibility would be better than keeping the bug.

This decision is clearly reflected in perl5100delta​:

| The handling of Unicode still is unclean in several places, where it's
| dependent on whether a string is internally flagged as UTF-8. This will
| be made more consistent in perl 5.12, but that won't be possible without
| a certain amount of backwards incompatibility.

Please proceed with fixing the bug. I am very happy with your offer to
smash this one.

Also, from reading the perl source, it appears to me that EBCDIC
machines may work differently (and more correctly to my way of thinking)
than Ascii-ish ones.

As always, I refrain from thinking about EBCDIC. I'd say​: keep the
current behavior for EBCDIC platforms - there haven't been *any*
complaints from them as far as I've heard.

An idea I've had is to add a pragma like "use latin1", or maybe "use
locale unicode", or something else as a way of not breaking existing
application code.

Please do break existing code, harsh as that may be. It is much more
likely that broken code magically starts working correctly, by the way.

Pragmas have problems, especially in regular expressions. And it's very
hard to load a pragma conditionally, which makes writing version
portable code hard. Besides that, any pragma affecting regex matches
needs to be carried in qr//, which in this case means new regex flags to
indicate the behavior for (?i​:...). According to dmq, adding flags is
hard.

Obviously, in my ignorance, I may be missing things that others can
enlighten me on.

Please feel free to copy the unit tests in Unicode​::Semantics!
--
Met vriendelijke groet, Kind regards, Korajn salutojn,

  Juerd Waalboer​: Perl hacker <#####@​juerd.nl> <http​://juerd.nl/sig>
  Convolution​: ICT solutions and consultancy <sales@​convolution.nl>
1;

[p5pRT]

From @Juerd

Glenn Linderman skribis 2008-09-20 16​:31 (-0700)​:

For compatibility reasons, as has been discussed on this list
previously, a pragma of some sort must be used to request the
incompatible enhancement (which you call a fix).

As the current behavior is a bug, the enhancement can rightfully be
called a fix.

What's this about the pragma that "must be used"? Yes, it has been
discussed, but no consensus has pointed in that direction.

In fact, perl5100delta clearly announces backwards incompatibility.
--
Met vriendelijke groet, Kind regards, Korajn salutojn,

  Juerd Waalboer​: Perl hacker <#####@​juerd.nl> <http​://juerd.nl/sig>
  Convolution​: ICT solutions and consultancy <sales@​convolution.nl>
1;

[p5pRT]

From @druud62

Juerd Waalboer schreef​:

Dr.Ruud​:

Unicode semantics get activated when a codepoint above 255 is
involved.

No, unicode semantics get activated when the internal encoding of the
string is utf8, even if it contains no character above 255, and even
if it only contains ASCII characters.

Yes, Unicode semantics get activated when a codepoint above 255 is
involved.

Yes, there are other ways too, like​:

perl -Mstrict -Mwarnings -Mencoding=utf8 -le'
  my $s = chr(65);
  print utf8​::is_utf8($s);
'
1

--
Affijn, Ruud

"Gewoon is een tijger."