Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default IO Layer is ignored in 3-argument open call #8272

Closed
p5pRT opened this issue Jan 7, 2006 · 8 comments
Closed

Default IO Layer is ignored in 3-argument open call #8272

p5pRT opened this issue Jan 7, 2006 · 8 comments

Comments

@p5pRT
Copy link

@p5pRT p5pRT commented Jan 7, 2006

Migrated from rt.perl.org#38181 (status was 'resolved')

Searchable as RT38181$

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 7, 2006

From Nikolaus@rath.org

Created by Nikolaus@rath.org

This is a bug report for perl from Nikolaus@​rath.org,
generated with the help of perlbug 1.35 running under perl v5.8.7.

-----------------------------------------------------------------
Consider the following program​:

#!/usr/bin/perl -CSDA
use strict;
use warnings;
use utf8;
my($fh,$buf);
open($fh, "<", "utf8.txt");
#open($fh, "<utf8.txt");
#open($fh, "<​:utf8", "utf8.txt");
$buf = <$fh>;
print "'$buf'";
print "(is utf8)\n" if utf8​::is_utf8($buf);
exit;

Where utf8.txt contains some utf8 encoded characters. The first open
call doesn't work, $buf has no utf8 flag and the output is garbled. The
two commented open calls work.

Citing perldoc -f open​:

  will open the UTF-8 encoded file containing Unicode
  characters, see perluniintro. (Note that if layers are
  specified in the three-arg form then default layers set
  by the "open" pragma are ignored.)

Clearly I didn't specify an IO-layer in my first open call, yet the
default layer was ignored.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl v5.8.7:

Configured by Debian Project at Mon Dec 12 13:59:01 UTC 2005.

Summary of my perl5 (revision 5 version 8 subversion 7) configuration:
  Platform:
    osname=linux, osvers=2.6.10, archname=i486-linux-gnu-thread-multi
    uname='linux rothera 2.6.10 #1 smp fri may 13 09:24:22 utc 2005 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.7 -Dsitearch=/usr/local/lib/perl/5.8.7 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.7 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu9)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.3.5.so, so=so, useshrplib=true, libperl=libperl.so.5.8.7
    gnulibc_version='2.3.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    SPRINTF0 - fixes for sprintf formatting issues - CVE-2005-3962


@INC for perl v5.8.7:
    /etc/perl
    /usr/local/lib/perl/5.8.7
    /usr/local/share/perl/5.8.7
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    .


Environment for perl v5.8.7:
    HOME=/home/nikratio
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_ADDRESS=de_DE.UTF-8
    LC_COLLATE=de_DE.UTF-8
    LC_CTYPE=de_DE.UTF-8
    LC_IDENTIFICATION=en_US.UTF-8
    LC_MEASUREMENT=de_DE.UTF-8
    LC_MESSAGES=en_US.UTF-8
    LC_MONETARY=de_DE.UTF-8
    LC_NAME=de_DE.UTF-8
    LC_NUMERIC=de_DE.UTF-8
    LC_PAPER=de_DE.UTF-8
    LC_TELEPHONE=de_DE.UTF-8
    LC_TIME=de_DE.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=~/bin:~/Projekte/Scripts:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11
    PERL_BADLANG (unset)
    PERL_UNICODE=SDAL
    SHELL=/bin/bash

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 8, 2006

From BQW10602@nifty.com

This is a bug report for perl from Nikolaus@​rath.org,
generated with the help of perlbug 1.35 running under perl v5.8.7.

-----------------------------------------------------------------
Consider the following program​:

#!/usr/bin/perl -CSDA
use strict;
use warnings;
use utf8;
my($fh,$buf);
open($fh, "<", "utf8.txt");
#open($fh, "<utf8.txt");
#open($fh, "<​:utf8", "utf8.txt");
$buf = <$fh>;
print "'$buf'";
print "(is utf8)\n" if utf8​::is_utf8($buf);
exit;

Where utf8.txt contains some utf8 encoded characters. The first open
call doesn't work, $buf has no utf8 flag and the output is garbled. The
two commented open calls work.

Citing perldoc -f open​:

       will open the UTF\-8 encoded file containing Unicode
       characters\, see perluniintro\. \(Note that if layers are
       specified in the three\-arg form then default layers set
       by the "open" pragma are ignored\.\)

Clearly I didn't specify an IO-layer in my first open call, yet the
default layer was ignored.

Thank you for report. But "perl-current" (under development) works
even if open($fh, "<", "utf8.txt") as well as other two calls.

This problem seems to have been recently resolved​:
____________________________________________________________________________
[ 25925] By​: rgs on 2005/10/31 14​:27​:32
  Log​: Fix [perl #37533] open pragma ignored for all 3-arg opens
  Also, correct the documentation in open.pm that stated
  that open() did not override open.pm-given default encodings.
  Branch​: perl
  ! lib/open.pm perlio.c
____________________________________________________________________________

Regards,
SADAHIRO Tomoyuki

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 8, 2006

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 8, 2006

From BQW10602@nifty.com

#!/usr/bin/perl -CSDA
(snip)
open($fh, "<", "utf8.txt");
(snip)
Citing perldoc -f open​:

       will open the UTF\-8 encoded file containing Unicode
       characters\, see perluniintro\. \(Note that if layers are
       specified in the three\-arg form then default layers set
       by the "open" pragma are ignored\.\)

In this case the open pragma is not loaded but the -C switch is set.
Nevertheless the part of perlfunc does not mention -C.
Here is a doc patch (against perl@​26695) for open.pm and perlfunc.pod

P.S. In perlfunc.pod, some C<open> refer to the pragma and others
refer to the function, while pod2html makes a link to either.
(actually all the tags are linked to the function). In the patch,
B<open> is used for the pragma (but I'm not sure B<> is proper).

Regards,
SADAHIRO Tomoyuki

Inline Patch
diff -ur perl~/lib/open.pm perl/lib/open.pm
--- perl~/lib/open.pm	Mon Oct 31 23:56:43 2005
+++ perl/lib/open.pm	Sun Jan 08 23:25:39 2006
@@ -169,9 +169,8 @@
 "layers" (also known as "disciplines") for all I/O. Any two-argument
 open(), readpipe() (aka qx//) and similar operators found within the
 lexical scope of this pragma will use the declared defaults.
-Three-argument opens are not affected by this pragma since there you
-(can) explicitly specify the layers and are supposed to know what you
-are doing.
+Even three-argument opens may be affected by this pragma
+when they don't specify IO layers in MODE.
 
 With the C<IN> subpragma you can declare the default layers
 of input streams, and with the C<OUT> subpragma you can declare
@@ -180,7 +179,7 @@
 
 If you have a legacy encoding, you can use the C<:encoding(...)> tag.
 
-if you want to set your encoding layers based on your
+If you want to set your encoding layers based on your
 locale environment variables, you can use the C<:locale> tag.
 For example:
 
diff -ur perl~/pod/perlfunc.pod perl/pod/perlfunc.pod
--- perl~/pod/perlfunc.pod	Mon Jan 02 23:27:04 2006
+++ perl/pod/perlfunc.pod	Mon Jan 09 00:21:37 2006
@@ -3103,14 +3103,14 @@
 You may use the three-argument form of open to specify IO "layers"
 (sometimes also referred to as "disciplines") to be applied to the handle
 that affect how the input and output are processed (see L<open> and
-L<PerlIO> for more details). For example
+L<Encode::PerlIO> for more details). For example
 
   open(FH, "<:utf8", "file")
 
 will open the UTF-8 encoded file containing Unicode characters,
-see L<perluniintro>. (Note that if layers are specified in the
-three-arg form then default layers set by the C<open> pragma are
-ignored.)
+see L<perluniintro>. Note that if layers are specified in the
+three-arg form then default layers stored in ${^OPEN} (see L<perlvar>;
+usually set by the B<open> pragma or the switch B<-CioD>) are ignored.
 
 Open returns nonzero upon success, the undefined value otherwise.  If
 the C<open> involved a pipe, the return value happens to be the pid of
@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 8, 2006

From BQW10602@nifty.com

A reply to myself...

-L<PerlIO> for more details). For example
+L<Encode​::PerlIO> for more details). For example

Sorry this change is wrong. The reference here should be PerlIO.pm
(Thus no change is need here). Revised patch is below.

P.S. Why does a link to Encode/PerlIO.pm is produced from L<PerlIO>
on conversion from POD to HTML? I don't know....

Regards,
SADAHIRO Tomoyuki

Inline Patch
diff -ur perl~/lib/open.pm perl/lib/open.pm
--- perl~/lib/open.pm	Mon Oct 31 23:56:43 2005
+++ perl/lib/open.pm	Mon Jan 09 00:38:52 2006
@@ -169,9 +169,8 @@
 "layers" (also known as "disciplines") for all I/O. Any two-argument
 open(), readpipe() (aka qx//) and similar operators found within the
 lexical scope of this pragma will use the declared defaults.
-Three-argument opens are not affected by this pragma since there you
-(can) explicitly specify the layers and are supposed to know what you
-are doing.
+Even three-argument opens may be affected by this pragma
+when they don't specify IO layers in MODE.
 
 With the C<IN> subpragma you can declare the default layers
 of input streams, and with the C<OUT> subpragma you can declare
@@ -180,7 +179,7 @@
 
 If you have a legacy encoding, you can use the C<:encoding(...)> tag.
 
-if you want to set your encoding layers based on your
+If you want to set your encoding layers based on your
 locale environment variables, you can use the C<:locale> tag.
 For example:
 
diff -ur perl~/pod/perlfunc.pod perl/pod/perlfunc.pod
--- perl~/pod/perlfunc.pod	Mon Jan 02 23:27:04 2006
+++ perl/pod/perlfunc.pod	Mon Jan 09 01:19:16 2006
@@ -3108,9 +3108,9 @@
   open(FH, "<:utf8", "file")
 
 will open the UTF-8 encoded file containing Unicode characters,
-see L<perluniintro>. (Note that if layers are specified in the
-three-arg form then default layers set by the C<open> pragma are
-ignored.)
+see L<perluniintro>. Note that if layers are specified in the
+three-arg form then default layers stored in ${^OPEN} (see L<perlvar>;
+usually set by the B<open> pragma or the switch B<-CioD>) are ignored.
 
 Open returns nonzero upon success, the undefined value otherwise.  If
 the C<open> involved a pipe, the return value happens to be the pid of
#####  END OF PATCH
@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jan 8, 2006

From @rgarcia

On 1/8/06, SADAHIRO Tomoyuki <bqw10602@​nifty.com> wrote​:

A reply to myself...

-L<PerlIO> for more details). For example
+L<Encode​::PerlIO> for more details). For example

Sorry this change is wrong. The reference here should be PerlIO.pm
(Thus no change is need here). Revised patch is below.

Thanks, applied as #26739.

P.S. Why does a link to Encode/PerlIO.pm is produced from L<PerlIO>
on conversion from POD to HTML? I don't know....

A bug ? That should be the PerlIO module...

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 7, 2006

From Nikolaus@rath.org

This problem seems to have been recently resolved​:

____________________________________________________________________________

[ 25925] By​: rgs on 2005/10/31 14​:27​:32
Log​: Fix [perl #37533] open pragma ignored for all 3-arg opens
Also, correct the documentation in open.pm that stated
that open() did not override open.pm-given default encodings.
Branch​: perl
! lib/open.pm perlio.c

Is it possible to somehow work around this problem until the fix reaches
my distribution's packages? I'm thinking of using a custom 3-arg open
function that explicitly adds the default layers to the layer
specification before calling the builtin open. Unfortunately, I don't
know to get the list of default layers...

Thanks
Nikolaus

@p5pRT
Copy link
Author

@p5pRT p5pRT commented May 22, 2008

p5p@spam.wizbit.be - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant