perl lexer bug: eval broken after getting "Unrecognized character" in first eval

[p5pRT]

Migrated from rt.perl.org#47517 (status was 'resolved')

Searchable as RT47517$

[p5pRT]

From sylvain.beugin@orange-ftgroup.com

Created by sylvain.beugin@orange-ftgroup.com

After Eval a var with an latin1 accent , other evals don't work...
The bug is that an error within eval should not affect other evals.

For monks node see​:
http​://www.perlmonks.org/index.pl?node_id=651150 <http​://www.perlmonks.org/index.pl?node_id=651150>

For example $string should be "new eval done" and it is undef :

my $code1='my $var_with_é_accent;';

print "Eval buggy code\n";
eval $code1;
print "ERR = $@​" if ($@​);;
print "DONE\n";

my $string;
my $code2='$string="new eval done";';

print "Eval good code\n";
eval $code2;

print "ERR = $@​" if ($@​);
print "string is : $string\n";
print "DONE\n";

Conclusion​:
"The specific error being triggered in the first eval (Unrecognized character \xE9) has a special case in toke.c, and I can only think this early exit is leaving the tokenizer in an odd state".

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.8:
 
Configured by Debian Project at Wed Dec  6 23:17:41 UTC 2006.
 
Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.18.3, archname=i486-linux-gnu-thread-multi
    uname='linux saens 2.6.18.3 #1 smp sat nov 25 13:39:52 est 2006 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.8 -Dsitearch=/usr/local/lib/perl/5.8.8 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.1.2 20061115 (prerelease) (Debian 4.1.1-20)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.3.6.so, so=so, useshrplib=true, libperl=libperl.so.5.8.8
    gnulibc_version='2.3.6'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'
 
Locally applied patches:
    
 

@INC for perl v5.8.8:
    /etc/perl
    /usr/local/lib/perl/5.8.8
    /usr/local/share/perl/5.8.8
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    /usr/local/lib/perl/5.8.4
    /usr/local/share/perl/5.8.4
    .
 

Environment for perl v5.8.8:
    HOME=/home/wasac
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/local/lib:/lib:/usr/lib:/usr/ccs/lib:/usr/local/pgsql/lib:/oracle/lib
    LOGDIR (unset)
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/home/wasac/bin:/usr/local/bin:/usr/local/pgsql/bin:/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
    PERL_BADLANG (unset)
    SHELL=/bin/bash



*********************************
This message and any attachments (the "message") are confidential and intended solely for the addressees. 
Any unauthorised use or dissemination is prohibited.
Messages are susceptible to alteration. 
France Telecom Group shall not be liable for the message if altered, changed or falsified.
If you are not the intended addressee of this message, please cancel it immediately and inform the sender.
********************************

[p5pRT]

From @iabyn

On Fri, Nov 16, 2007 at 07​:53​:45AM -0800, BEUGIN Sylvain SCE wrote​:

After Eval a var with an latin1 accent , other evals don't work...
The bug is that an error within eval should not affect other evals.

For monks node see​:
http​://www.perlmonks.org/index.pl?node_id=651150 <http​://www.perlmonks.org/index.pl?node_id=651150>

For example $string should be "new eval done" and it is undef :

my $code1='my $var_with_é_accent;';

print "Eval buggy code\n";
eval $code1;
print "ERR = $@​" if ($@​);;
print "DONE\n";

my $string;
my $code2='$string="new eval done";';

print "Eval good code\n";
eval $code2;

print "ERR = $@​" if ($@​);
print "string is : $string\n";
print "DONE\n";

Conclusion​:
"The specific error being triggered in the first eval (Unrecognized
character \xE9) has a special case in toke.c, and I can only think this
early exit is leaving the tokenizer in an odd state".

Thanks for the report.
This is fixed in perl 5.9.5 onwards, probably by change #31203.
PL_in_my wasn't being re-initialised at the start of the eval, so
'$string' was being misinterpreted as 'my $string'.

--
A power surge on the Bridge is rapidly and correctly diagnosed as a faulty
capacitor by the highly-trained and competent engineering staff.
  -- Things That Never Happen in "Star Trek" #9

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

p5p@spam.wizbit.be - Status changed from 'open' to 'resolved'