Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Program gets stuck when trying to read an illegal iso-2022-jp file #16362

Open
p5pRT opened this issue Jan 16, 2018 · 6 comments
Open

Program gets stuck when trying to read an illegal iso-2022-jp file #16362

p5pRT opened this issue Jan 16, 2018 · 6 comments

Comments

@p5pRT
Copy link

@p5pRT p5pRT commented Jan 16, 2018

Migrated from rt.perl.org#132724 (status was 'open')

Searchable as RT132724$

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 16, 2018

From @hiroshi-manabe

This is a bug report for perl from manabe.hiroshi@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.22.0.

You can reproduce the bug with the following procedure​:
1. perl -e 'print "\x8f";' > temp.txt
2. perl -e 'open my $in, "<​:encoding(iso-2022-jp)", "temp.txt"; print <$in>;'


Flags​:
  category=library
  severity=high


Site configuration information for perl 5.22.0​:

Configured by hmanabe at Mon Oct 24 11​:28​:50 JST 2016.

Summary of my perl5 (revision 5 version 22 subversion 0) configuration​:
 
  Platform​:
  osname=linux, osvers=2.6.32-220.7.1.el6.x86_64, archname=x86_64-linux
  uname='linux rxl5051.mastar.nict.go.jp 2.6.32-220.7.1.el6.x86_64 #1 smp wed mar 7 00​:52​:02 gmt 2012 x86_64 x86_64 x86_64 gnulinux '
  config_args='-Dprefix=/home/hmanabe/.plenv/versions/5.22.0 -de -Dusedevel -A'eval​:scriptdir=/home/hmanabe/.plenv/versions/5.22.0/bin''
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
  optimize='-O2',
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.4.7 20120313 (Red Hat 4.4.7-16)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/home/hmanabe/lib /usr/local/lib /usr/lib /panfs/panmt/users/hmanabe/lib/../lib64 /panfs/panmt/users/hmanabe/lib64/../lib64 /lib/../lib64 /usr/lib/../lib64 /panfs/panmt/users/hmanabe/lib /panfs/panmt/users/hmanabe/lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.12.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.12'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'


@​INC for perl 5.22.0​:
  /home/hmanabe/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0/x86_64-linux
  /home/hmanabe/.plenv/versions/5.22.0/lib/perl5/site_perl/5.22.0
  /home/hmanabe/.plenv/versions/5.22.0/lib/perl5/5.22.0/x86_64-linux
  /home/hmanabe/.plenv/versions/5.22.0/lib/perl5/5.22.0
  .


Environment for perl 5.22.0​:
  HOME=/home/hmanabe
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH=/home/hmanabe/user/softwares/gcc-6.3.0/lib​:/home/hmanabe/user/softwares/gcc-6.3.0/lib64​:/panfs/panmt/users/hmanabe/lib​:/panfs/panmt/users/hmanabe/lib64
  LOGDIR (unset)
  PATH=/home/hmanabe/.plenv/versions/5.22.0/bin​:/home/hmanabe/.plenv/libexec​:/home/hmanabe/.plenv/plugins/perl-build/bin​:/home/hmanabe/user/softwares/gcc-6.3.0/bin​:/home/hmanabe/scripts​:/home/hmanabe/jre/bin​:/home/hmanabe/.linuxbrew/bin​:/home/hmanabe/.pyenv/shims​:/home/hmanabe/.pyenv/bin​:/home/hmanabe/.plenv/shims​:/home/hmanabe/.plenv/bin​:/home/hmanabe/.rbenv/shims​:/home/hmanabe/.rbenv/bin​:/home/hmanabe/user/srilm/bin​:/home/hmanabe/user/srilm/bin/i686-m64​:/home/hmanabe/bin​:/home/hmanabe/local/bin​:/usr/lib64/qt-3.3/bin​:/usr/local/bin​:/bin​:/usr/bin​:/usr/local/sbin​:/usr/sbin​:/sbin
  PERL_BADLANG (unset)
  SHELL=/usr/local/bin/bash

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 16, 2018

From @hiroshi-manabe

I re-reported as a bug in Encode module. https://rt.cpan.org/Public/Bug/Display.html?id=124094

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Sep 4, 2018

From @jkeenan

On Tue, 16 Jan 2018 04​:45​:00 GMT, manabe.hiroshi@​gmail.com wrote​:

This is a bug report for perl from manabe.hiroshi@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.22.0.

You can reproduce the bug with the following procedure​:
1. perl -e 'print "\x8f";' > temp.txt
2. perl -e 'open my $in, "<​:encoding(iso-2022-jp)", "temp.txt"; print
<$in>;'

I am unfamiliar with this encoding. Can you say what makes this file illegal?

(I agree that Perl gets stuck while attempting to read it.)

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Sep 4, 2018

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Sep 15, 2018

From @Leont

On Tue, Jan 16, 2018 at 7​:56 AM Hiroshi Manabe
<perlbug-followup@​perl.org> wrote​:

You can reproduce the bug with the following procedure​:
1. perl -e 'print "\x8f";' > temp.txt
2. perl -e 'open my $in, "<​:encoding(iso-2022-jp)", "temp.txt"; print <$in>;'

Given that this one-liner doesn't hang​:
  perl -MEncode -e 'decode("iso-2022-jp", "\x8f", Encode​::FB_CROAK |
Encode​::LEAVE_SRC)'
I'm pretty sure the issue is in PerlIO​::encoding, not in Encode. Which
means it's a bug on our side, not Encode's.

I assume "\x8f" is the start of a multi-byte character. :encoding
doesn't handle such errors at the end of a file well (I think there's
a already a ticket about this but I can't find it, though I'm sure
#84250 is related).

I think I know how to fix this though.

Leon

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Apr 4, 2019

From @khwilliamson

On Sat, 15 Sep 2018 09​:39​:31 -0700, LeonT wrote​:

On Tue, Jan 16, 2018 at 7​:56 AM Hiroshi Manabe
<perlbug-followup@​perl.org> wrote​:

You can reproduce the bug with the following procedure​:
1. perl -e 'print "\x8f";' > temp.txt
2. perl -e 'open my $in, "<​:encoding(iso-2022-jp)", "temp.txt"; print
<$in>;'

Given that this one-liner doesn't hang​:
perl -MEncode -e 'decode("iso-2022-jp", "\x8f", Encode​::FB_CROAK |
Encode​::LEAVE_SRC)'
I'm pretty sure the issue is in PerlIO​::encoding, not in Encode. Which
means it's a bug on our side, not Encode's.

I assume "\x8f" is the start of a multi-byte character. :encoding
doesn't handle such errors at the end of a file well (I think there's
a already a ticket about this but I can't find it, though I'm sure
#84250 is related).

I think I know how to fix this though.

Leon

Any progress on this?
--
Karl Williamson

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants