Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use and require are affected by the open pragma #11541

Closed
p5pRT opened this issue Jul 31, 2011 · 31 comments
Closed

use and require are affected by the open pragma #11541

p5pRT opened this issue Jul 31, 2011 · 31 comments

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Jul 31, 2011

Migrated from rt.perl.org#96008 (status was 'resolved')

Searchable as RT96008$

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 31, 2011

From @Leont

This is a bug report for perl from fawaka@​gmail.com,
generated with the help of perlbug 1.39 running under perl 5.14.1.


use and require are affected by the open pragma, as seen here​:

% perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl
% perl '-Mopen=​:encoding(utf8)' -e 'require "x.pl"'
utf8 "\xF1" does not map to Unicode at x.pl line 1.



Flags​:
  category=core
  severity=medium


Site configuration information for perl 5.14.1​:

Configured by leon at Tue Jun 28 15​:59​:52 CEST 2011.

Summary of my perl5 (revision 5 version 14 subversion 1) configuration​:

  Platform​:
  osname=linux, osvers=2.6.38-8-generic, archname=x86_64-linux
  uname='linux leon-laptop 2.6.38-8-generic #42-ubuntu smp mon apr
11 03​:31​:24 utc 2011 x86_64 x86_64 x86_64 gnulinux '
  config_args='-de -Dprefix=/home/leon/perl5/perlbrew/perls/perl-5.14.1'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2',
  cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.6.1 20110409 (prerelease)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib /usr/lib/x86_64-linux-gnu
/lib64 /usr/lib64
  libs=-lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
  libc=, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.13'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
-fstack-protector'

Locally applied patches​:


@​INC for perl 5.14.1​:
  /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/site_perl/5.14.1/x86_64-linux
  /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/site_perl/5.14.1
  /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/5.14.1/x86_64-linux
  /home/leon/perl5/perlbrew/perls/perl-5.14.1/lib/5.14.1
  .


Environment for perl 5.14.1​:
  HOME=/home/leon
  LANG=en_US.utf8
  LANGUAGE=en_US​:en
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/home/leon/perl5/perlbrew/bin​:/home/leon/perl5/perlbrew/perls/perl-5.14.1/bin​:/home/leon/bin​:/usr/local/sbin​:/usr/local/bin​:/usr/sbin​:/usr/bin​:/sbin​:/bin​:/usr/games
  PERLBREW_HOME=/home/leon/.perlbrew
  PERLBREW_PATH=/home/leon/perl5/perlbrew/bin​:/home/leon/perl5/perlbrew/perls/perl-5.14.1/bin
  PERLBREW_PERL=perl-5.14.1
  PERLBREW_ROOT=/home/leon/perl5/perlbrew
  PERLBREW_VERSION=0.25
  PERL_BADLANG (unset)
  SHELL=/bin/bash

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From @iabyn

On Sun, Jul 31, 2011 at 02​:05​:23PM -0700, Leon Timmermans wrote​:

use and require are affected by the open pragma, as seen here​:

% perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl
% perl '-Mopen=​:encoding(utf8)' -e 'require "x.pl"'
utf8 "\xF1" does not map to Unicode at x.pl line 1.

Well, the pragma's docs says that it affects "all I/O" within the lexical
scope, so surely it's behaving as per docs?

--
Thank God I'm an atheist.....

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

The RT System itself - Status changed from 'new' to 'open'

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From @Leont

On Mon Aug 01 02​:27​:45 2011, davem wrote​:

On Sun, Jul 31, 2011 at 02​:05​:23PM -0700, Leon Timmermans wrote​:

use and require are affected by the open pragma, as seen here​:

% perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl
% perl '-Mopen=​:encoding(utf8)' -e 'require "x.pl"'
utf8 "\xF1" does not map to Unicode at x.pl line 1.

Well, the pragma's docs says that it affects "all I/O" within the lexical
scope, so surely it's behaving as per docs?

I really don't think loading a module is can be reasonably called «I/O
within the lexical scope». The encoding with which a module is loaded
with should not depend on a pragma set by the first module requiring it.
That's just plain evil.

Leon

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From @nwc10

On Mon, Aug 01, 2011 at 10​:27​:12AM +0100, Dave Mitchell wrote​:

On Sun, Jul 31, 2011 at 02​:05​:23PM -0700, Leon Timmermans wrote​:

use and require are affected by the open pragma, as seen here​:

% perl -e 'print "package F;\n # \xF1\n;1;"' > x.pl
% perl '-Mopen=​:encoding(utf8)' -e 'require "x.pl"'
utf8 "\xF1" does not map to Unicode at x.pl line 1.

Well, the pragma's docs says that it affects "all I/O" within the lexical
scope, so surely it's behaving as per docs?

$ cat >y.pl
require "x.pl";
$ ./perl -Ilib -e 'use open "​:encoding(utf8)"; require "y.pl"'
$

No warning.

perlfunc.pod says​:

  Otherwise, C<require> demands that a library file be included if it
  hasn't already been included. The file is included via the do-FILE
  mechanism, which is essentially just a variety of C<eval> with the
  caveat that lexical variables in the invoking script will be invisible
  to the included code. Has semantics similar to the following subroutine​:

and makes no comment about lexical pragmata. The y.pl example above shows
that the implementation doesn't propagate lexical pragmata *into* the
file that it's requiring.

So it looks like the current behaviour is "as if" [and probably actually
*is* - not groveled too far] that the open for the (internal) do is
being treated as being in the caller's lexical scope, before a new scope
is set up to actually process the file.

All things above being equal, I'm not sure if I think that this is a bug.
*But*​:

require only loads a file once. Hence if you have two modules which require
or use the same dependent module, one of which is

  use open "​:encoding(utf8)"
  require foo;

and the other is

  no open;
  require foo;

then what you actually get in %foo​:: is going to depend on the order that
the two snippets of code above happen to get wrong.

Which is action at a distance. And feels horribly like it will cause much
wailing, gnashing of teeth, pain, suffering, lost hours, and caustic blog
posts.

And caustic blog posts are bad marketing, even if we are so callous as to
disregard the waste of the hours.

Or at least, that's how somewhat sleep-deprived me sees it.

Nicholas Clark

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From @Leont

On Mon Aug 01 07​:36​:28 2011, nicholas wrote​:

So it looks like the current behaviour is "as if" [and probably
actually
*is* - not groveled too far] that the open for the (internal) do is
being treated as being in the caller's lexical scope, before a new
scope
is set up to actually process the file.

That's exactly what happens (see pp_require and check_type_and_open in
pp_ctl.c).

then what you actually get in %foo​:: is going to depend on the order
that
the two snippets of code above happen to get wrong.

Which is action at a distance. And feels horribly like it will cause
much
wailing, gnashing of teeth, pain, suffering, lost hours, and caustic
blog
posts.

Yeah, that is another complication of this. The open pragma seems to be
collecting issues :-(.

Leon

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From @Hugmeir

On Mon, Aug 1, 2011 at 11​:36 AM, Nicholas Clark <nick@​ccl4.org> wrote​:

$ cat >y.pl
require "x.pl";
$ ./perl -Ilib -e 'use open "​:encoding(utf8)"; require "y.pl"'
$

No warning.

perlfunc.pod says​:

Otherwise, C<require> demands that a library file be included if it
hasn't already been included. The file is included via the do-FILE
mechanism, which is essentially just a variety of C<eval> with the
caveat that lexical variables in the invoking script will be invisible
to the included code. Has semantics similar to the following
subroutine​:

and makes no comment about lexical pragmata. The y.pl example above shows
that the implementation doesn't propagate lexical pragmata *into* the
file that it's requiring.

So it looks like the current behaviour is "as if" [and probably actually
*is* - not groveled too far] that the open for the (internal) do is
being treated as being in the caller's lexical scope, before a new scope
is set up to actually process the file.

All things above being equal, I'm not sure if I think that this is a bug.
*But*​:

require only loads a file once. Hence if you have two modules which require
or use the same dependent module, one of which is

use open "​:encoding(utf8)"
require foo;

and the other is

no open;
require foo;

then what you actually get in %foo​:: is going to depend on the order that
the two snippets of code above happen to get wrong.

Which is action at a distance. And feels horribly like it will cause much
wailing, gnashing of teeth, pain, suffering, lost hours, and caustic blog
posts.

And caustic blog posts are bad marketing, even if we are so callous as to
disregard the waste of the hours.

Or at least, that's how somewhat sleep-deprived me sees it.

Nicholas Clark

Well, assuming this is considered a bug, here's a patch to resolve it : ) It
merely passes "​:raw" to PerlIO_openn instead of NULL - The latter which
forces the function to use the layers defined by use open, thus causing this
entire mess : )

Inline Patch
diff --git a/lib/open.t b/lib/open.t
index 1bf7dca..1d61086 100644
--- a/lib/open.t
+++ b/lib/open.t
@@ -186,18 +186,39 @@ SKIP: {
     eval q[use Encode::Alias;use open ":std", ":locale"];
     is($@, '', 'can use :std and :locale');

-    use open IN => ':non-existent';
-    eval {
-    require Symbol; # Anything that exists but we havn't loaded
-    };
-    like($@, qr/Can't locate Symbol|Recursive call/i,
-     "test for an endless loop in PerlIO_find_layer");
+#This test was using dubious behaviour - It relied on the open pragma to
propagate \+\#it's effect into the layer used for require\, which is no longer true\. \+\#See the next test as for why this was changed\. \+ \+\# use open IN => '​:non\-existent'; \+\# eval \{ \+\# require Symbol; \# Anything that exists but we havn't loaded \+\# \}; \+\# like\($@​\, qr/Can't locate Symbol|Recursive call/i\, \+\# "test for an endless loop in PerlIO\_find\_layer"\); \+ \+ \+ open my $out\_fh\, ">​:raw"\, "x\.pl"; \+ print \{ $out\_fh \} "package F;\\n \# \\xF1\\n;1;"; \+ close $out\_fh; \+ \+\#This require would've thrown "utf8 "\\xF1" does not map to Unicode" if require \+\#was affected by open\. \+ \{ \+ use open qw\( :encoding\(utf8\) \); \+ eval \{ \+ local @​INC = "\."; \+ require "x\.pl"; \+ \}; \+ is $@​\, ''\, "open does not propagate into require\(\)"; \+ \} \}

END {
  1 while unlink "utf8";
  1 while unlink "a";
  1 while unlink "b";
+ 1 while unlink "x.pl";
}

# the test cases beyond __DATA__ need to be executed separately

Inline Patch
diff --git a/pp_ctl.c b/pp_ctl.c
index d5924d8..7d5e58a 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@@ -3649,7 +3649,7 @@ S_check_type_and_open(pTHX_ SV *name)
     }

 #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
-    return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name); \+ return PerlIO\_openn\(aTHX\_ "​:raw"\, PERL\_SCRIPT\_MODE\, \-1\, 0\, 0\, NULL\, 1\, &name\); \#else   return PerlIO\_open\(p\, PERL\_SCRIPT\_MODE\); \#endif
@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From @Hugmeir

open_noprop.diff
diff --git a/lib/open.t b/lib/open.t
index 1bf7dca..1d61086 100644
--- a/lib/open.t
+++ b/lib/open.t
@@ -186,18 +186,39 @@ SKIP: {
     eval q[use Encode::Alias;use open ":std", ":locale"];
     is($@, '', 'can use :std and :locale');
 
-    use open IN => ':non-existent';
-    eval {
-	require Symbol; # Anything that exists but we havn't loaded
-    };
-    like($@, qr/Can't locate Symbol|Recursive call/i,
-	 "test for an endless loop in PerlIO_find_layer");
+#This test was using dubious behaviour - It relied on the open pragma to propagate
+#it's effect into the layer used for require, which is no longer true.
+#See the next test as for why this was changed.
+
+#    use open IN => ':non-existent';
+#    eval {
+#	require Symbol; # Anything that exists but we havn't loaded
+#    };
+#    like($@, qr/Can't locate Symbol|Recursive call/i,
+#	 "test for an endless loop in PerlIO_find_layer");
+
+
+    open my $out_fh, ">:raw", "x.pl";
+    print { $out_fh } "package F;\n # \xF1\n;1;";
+    close $out_fh;
+
+#This require would've thrown "utf8 "\xF1" does not map to Unicode" if require
+#was affected by open.
+    {
+        use open qw( :encoding(utf8) );
+            eval {
+                local @INC = ".";
+                require "x.pl";
+            };
+        is $@, '', "open does not propagate into require()";
+    }
 }
 
 END {
     1 while unlink "utf8";
     1 while unlink "a";
     1 while unlink "b";
+    1 while unlink "x.pl";
 }
 
 # the test cases beyond __DATA__ need to be executed separately
diff --git a/pp_ctl.c b/pp_ctl.c
index d5924d8..7d5e58a 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@@ -3649,7 +3649,7 @@ S_check_type_and_open(pTHX_ SV *name)
     }
 
 #if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
-    return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
+    return PerlIO_openn(aTHX_ ":raw", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
 #else
     return PerlIO_open(p, PERL_SCRIPT_MODE);
 #endif
@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 1, 2011

From tchrist@perl.com

Just because you call a function that does I/O doesn't mean
that that function should inherit its caller's I/O layers.
That requires dynamic not lexical scoping.

do provides a separate lexical scope.
Therefore, so does require.

A different question is $foo = `cat $file`.

--tom

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 2, 2011

From @Leont

On Mon, Aug 1, 2011 at 9​:29 PM, Brian Fraser <fraserbn@​gmail.com> wrote​:

On Mon, Aug 1, 2011 at 11​:36 AM, Nicholas Clark <nick@​ccl4.org> wrote​:
Well, assuming this is considered a bug, here's a patch to resolve it : ) It
merely passes "​:raw" to PerlIO_openn instead of NULL - The latter which
forces the function to use the layers defined by use open, thus causing this
entire mess : )

Perl modules/scripts are text files, not binary files. I don't think
"​:raw" really is the correct thing to do, though it may work out in
the end.

Leon

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 2, 2011

From @nwc10

On Mon, Aug 01, 2011 at 04​:29​:40PM -0300, Brian Fraser wrote​:

Well, assuming this is considered a bug, here's a patch to resolve it : ) It
merely passes "​:raw" to PerlIO_openn instead of NULL - The latter which
forces the function to use the layers defined by use open, thus causing this
entire mess : )

Except I *think* that the call​:

diff --git a/pp_ctl.c b/pp_ctl.c
index d5924d8..7d5e58a 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@​@​ -3649,7 +3649,7 @​@​ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
+ return PerlIO_openn(aTHX_ "​:raw", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1, &name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif

now opens the file then pushes "raw", and pushing "raw" does binmode() on the
file handle, which isn't what's desired on Win32.

I'm not sure how to call PerlIO in such a way as to override the open pragma,
but also honour the passed in mode.

PerlIO is not my favourite part of the core Perl source code, and hasn't been
since it was *written*, given that it was initially written without
consideration for threads. You would have thought that that mistake would
have already been learned.

Nicholas Clark

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 2, 2011

From @Leont

On Tue Aug 02 02​:55​:21 2011, nicholas wrote​:

I'm not sure how to call PerlIO in such a way as to override the open
pragma,
but also honour the passed in mode.

I think we should add a :default pseudo-layer. It should be fairly
trivial to implement in core, though I don't think the API allows it do
be done in a CPAN module.

PerlIO is not my favourite part of the core Perl source code

It's nobody's favorite part of the core, which is its main problem.
Without getting some love it will stay the way it is now.

Leon

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 2, 2011

From @cpansprout

On Tue Aug 02 02​:55​:21 2011, nicholas wrote​:

On Mon, Aug 01, 2011 at 04​:29​:40PM -0300, Brian Fraser wrote​:

Well, assuming this is considered a bug, here's a patch to resolve
it : ) It
merely passes "​:raw" to PerlIO_openn instead of NULL - The latter
which
forces the function to use the layers defined by use open, thus
causing this
entire mess : )

Except I *think* that the call​:

diff --git a/pp_ctl.c b/pp_ctl.c
index d5924d8..7d5e58a 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@​@​ -3649,7 +3649,7 @​@​ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0,
NULL, 1, &name);
+ return PerlIO_openn(aTHX_ "​:raw", PERL_SCRIPT_MODE, -1, 0, 0,
NULL, 1, &name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif

now opens the file then pushes "raw", and pushing "raw" does binmode()
on the
file handle, which isn't what's desired on Win32.

I'm not sure how to call PerlIO in such a way as to override the open
pragma,
but also honour the passed in mode.

How about temporarily clearing the hints (after using a SAVE* macro)?

PerlIO is not my favourite part of the core Perl source code, and
hasn't been
since it was *written*, given that it was initially written without
consideration for threads. You would have thought that that mistake
would
have already been learned.

Nicholas Clark

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 2, 2011

From @cpansprout

On Tue Aug 02 06​:21​:58 2011, sprout wrote​:

On Tue Aug 02 02​:55​:21 2011, nicholas wrote​:

On Mon, Aug 01, 2011 at 04​:29​:40PM -0300, Brian Fraser wrote​:

Well, assuming this is considered a bug, here's a patch to resolve
it : ) It
merely passes "​:raw" to PerlIO_openn instead of NULL - The latter
which
forces the function to use the layers defined by use open, thus
causing this
entire mess : )

Except I *think* that the call​:

diff --git a/pp_ctl.c b/pp_ctl.c
index d5924d8..7d5e58a 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@​@​ -3649,7 +3649,7 @​@​ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0,
NULL, 1, &name);
+ return PerlIO_openn(aTHX_ "​:raw", PERL_SCRIPT_MODE, -1, 0, 0,
NULL, 1, &name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif

now opens the file then pushes "raw", and pushing "raw" does binmode()
on the
file handle, which isn't what's desired on Win32.

I'm not sure how to call PerlIO in such a way as to override the open
pragma,
but also honour the passed in mode.

How about temporarily clearing the hints (after using a SAVE* macro)?

Or pushing the new scope before opening the file?

PerlIO is not my favourite part of the core Perl source code, and
hasn't been
since it was *written*, given that it was initially written without
consideration for threads. You would have thought that that mistake
would
have already been learned.

Nicholas Clark

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 2, 2011

From @jandubois

On Tue, 02 Aug 2011, Leon Timmermans wrote​:

On Mon, Aug 1, 2011 at 9​:29 PM, Brian Fraser <fraserbn@​gmail.com> wrote​:

On Mon, Aug 1, 2011 at 11​:36 AM, Nicholas Clark <nick@​ccl4.org> wrote​:
Well, assuming this is considered a bug, here's a patch to resolve it : ) It
merely passes "​:raw" to PerlIO_openn instead of NULL - The latter which
forces the function to use the layers defined by use open, thus causing this
entire mess : )

Perl modules/scripts are text files, not binary files. I don't think
"​:raw" really is the correct thing to do, though it may work out in
the end.

No, it won't. It will switch the DATA filehandle to :raw mode as well,
so you'll end up with "\r\n" line endings on Windows.

Perl actually used to read the source in :raw mode on Windows, but
(incompletely) turned on text mode when it reached the __DATA__ token.
This created quite a number of bugs, all merged into this ticket​:

https://rt-archive.perl.org/perl5/Public/Bug/Display.html?id=28106

For Perl 5.14 I changed the default to reading source code in text mode,
which fixed all those bugs. I'll soon remove the broken code that
unsuccessfully tries to switch on text-mode mid-stream because it
appears that nobody knows how to fix it (and it is unrelated to the open
pragma anyways).

Cheers,
-Jan

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 4, 2011

From @Hugmeir

On Tue, Aug 2, 2011 at 3​:36 PM, Jan Dubois <jand@​activestate.com> wrote​:

On Tue, 02 Aug 2011, Leon Timmermans wrote​:

Perl modules/scripts are text files, not binary files. I don't think
"​:raw" really is the correct thing to do, though it may work out in
the end.

No, it won't. It will switch the DATA filehandle to :raw mode as well,
so you'll end up with "\r\n" line endings on Windows.

Perl actually used to read the source in :raw mode on Windows, but
(incompletely) turned on text mode when it reached the __DATA__ token.
This created quite a number of bugs, all merged into this ticket​:

https://rt-archive.perl.org/perl5/Public/Bug/Display.html?id=28106

For Perl 5.14 I changed the default to reading source code in text mode,
which fixed all those bugs. I'll soon remove the broken code that
unsuccessfully tries to switch on text-mode mid-stream because it
appears that nobody knows how to fix it (and it is unrelated to the open
pragma anyways).

Cheers,
-Jan

Alright, then I think that Father C's suggestion is our winner here.

Unfortunately I can't get SAVEHINTS(); to play nice (my ignorance of PerlIO
is only surpassed by my inexperience with scope.c), so I can't volunteer
myself for this one. But if it helps, from my hamhanded testing, all you
need to do is clear PL_curcop->cop_hints of HINT_LEXICAL_IO_(IN|OUT) before
calling PerlIO_openn.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 7, 2011

From @Leont

On Tue, Aug 2, 2011 at 2​:47 PM, Leon Timmermans via RT
<perlbug-followup@​perl.org> wrote​:

On Tue Aug 02 02​:55​:21 2011, nicholas wrote​:

I'm not sure how to call PerlIO in such a way as to override the open
pragma,
but also honour the passed in mode.

I think we should add a :default pseudo-layer. It should be fairly
trivial to implement in core, though I don't think the API allows it do
be done in a CPAN module.

Attached the patch. It turned out to be quite simple.

Leon

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 7, 2011

From @Leont

default-perlio.patch
diff --git a/perlio.c b/perlio.c
index 72a71fe..c8d7141 100644
--- a/perlio.c
+++ b/perlio.c
@@ -837,6 +837,9 @@ PerlIO_get_layers(pTHX_ PerlIO *f)
  * XS Interface for perl code
  */
 
+#define PerlIO_default_top() PerlIO_default_layer(aTHX_ -1)
+#define PerlIO_default_btm() PerlIO_default_layer(aTHX_ 0)
+
 PerlIO_funcs *
 PerlIO_find_layer(pTHX_ const char *name, STRLEN len, int load)
 {
@@ -844,6 +847,8 @@ PerlIO_find_layer(pTHX_ const char *name, STRLEN len, int load)
     IV i;
     if ((SSize_t) len <= 0)
 	len = strlen(name);
+    if (len == 7 && strEQ(name, "default"))
+	return PerlIO_default_top();
     for (i = 0; i < PL_known_layers->cur; i++) {
 	PerlIO_funcs * const f = PL_known_layers->array[i].funcs;
 	if (memEQ(f->name, name, len) && f->name[len] == 0) {
@@ -1246,9 +1251,6 @@ PerlIO_default_layer(pTHX_ I32 n)
     return PerlIO_layer_fetch(aTHX_ av, n, PERLIO_FUNCS_CAST(&PerlIO_stdio));
 }
 
-#define PerlIO_default_top() PerlIO_default_layer(aTHX_ -1)
-#define PerlIO_default_btm() PerlIO_default_layer(aTHX_ 0)
-
 void
 PerlIO_stdstreams(pTHX)
 {
@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 21, 2011

From @Hugmeir

On Sun, Aug 7, 2011 at 6​:42 PM, Leon Timmermans <fawaka@​gmail.com> wrote​:

Attached the patch. It turned out to be quite simple.

With 5.15.2 out of the door, can this get committed?

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 21, 2011

From @cpansprout

On Sat Aug 20 21​:37​:43 2011, fraserbn@​gmail.com wrote​:

On Sun, Aug 7, 2011 at 6​:42 PM, Leon Timmermans <fawaka@​gmail.com> wrote​:

Attached the patch. It turned out to be quite simple.

With 5.15.2 out of the door, can this get committed?

If this adds a new pseudo-layer accessible from Perl space, then it
needs documentation, doesn’t it?

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 21, 2011

From @Hugmeir

On Sun, Aug 21, 2011 at 3​:37 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

If this adds a new pseudo-layer accessible from Perl space, then it
needs documentation, doesn’t it?

Would something like this do?

Inline Patch
diff --git a/lib/PerlIO.pm b/lib/PerlIO.pm
index 46e6e44..2c36d72 100644
--- a/lib/PerlIO.pm
+++ b/lib/PerlIO.pm
@@ -190,6 +190,15 @@ On Win32 platforms this I<experimental> layer uses the
nati rather than the unix\-like numeric file descriptor layer\. Known to be buggy as of perl 5\.8\.2\.

+=item :default
+
+When opening a stream without manually setting a layer, Perl implicitly
+uses a combination of several layers, which are dependent on build
+options and OS.
+As this behavior can be augmented (through the C<open> pragma, or
+by modifying C<${^OPEN}>), you can use this layer to ignore that and
+use the default instead.
+
=back

=head2 Custom Layers

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 21, 2011

From @Leont

On Sun, Aug 21, 2011 at 8​:37 AM, Father Chrysostomos via RT
<perlbug-followup@​perl.org> wrote​:

On Sat Aug 20 21​:37​:43 2011, fraserbn@​gmail.com wrote​:

On Sun, Aug 7, 2011 at 6​:42 PM, Leon Timmermans <fawaka@​gmail.com> wrote​:

Attached the patch. It turned out to be quite simple.

With 5.15.2 out of the door, can this get committed?

If this adds a new pseudo-layer accessible from Perl space, then it
needs documentation, doesn’t it?

At least as important, it needs tests. I wrote some, but I bumped into
some rather peculiar bugs in PerlIO that prevent it from working as
expected. It does exactly the same as "​:perlio" or "​:crlf" (depending
on platform) but the former subtly does the wrong thing​: adding a
second layer of buffering. In fact, in some cases it's even possible
to end up with three(!) layers of buffering. Haven't had time to
properly perlbug them though, will do so soon.

I think it'd be better to fix that first before applying this.

Leon

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Aug 22, 2011

From @cpansprout

On Thu Aug 04 01​:11​:45 2011, fraserbn@​gmail.com wrote​:

On Tue, Aug 2, 2011 at 3​:36 PM, Jan Dubois <jand@​activestate.com> wrote​:

On Tue, 02 Aug 2011, Leon Timmermans wrote​:

Perl modules/scripts are text files, not binary files. I don't think
"​:raw" really is the correct thing to do, though it may work out in
the end.

No, it won't. It will switch the DATA filehandle to :raw mode as well,
so you'll end up with "\r\n" line endings on Windows.

Perl actually used to read the source in :raw mode on Windows, but
(incompletely) turned on text mode when it reached the __DATA__ token.
This created quite a number of bugs, all merged into this ticket​:

https://rt-archive.perl.org/perl5/Public/Bug/Display.html?id=28106

For Perl 5.14 I changed the default to reading source code in text mode,
which fixed all those bugs. I'll soon remove the broken code that
unsuccessfully tries to switch on text-mode mid-stream because it
appears that nobody knows how to fix it (and it is unrelated to the open
pragma anyways).

Cheers,
-Jan

Alright, then I think that Father C's suggestion is our winner here.

Except it can’t work properly, because SAVEHINTS and PL_hints deal with
*compile-time* hints, whereas PerlIO looks at the hints in the
already-compiled op tree that is running (in PL_curcop, in other words).

PL_curcop->cop_hints can’t be modified, because that won’t be thread-safe.

Unfortunately I can't get SAVEHINTS(); to play nice (my ignorance of
PerlIO
is only surpassed by my inexperience with scope.c), so I can't volunteer
myself for this one. But if it helps, from my hamhanded testing, all you
need to do is clear PL_curcop->cop_hints of HINT_LEXICAL_IO_(IN|OUT)
before
calling PerlIO_openn.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2011

From @Hugmeir

On Mon, Aug 22, 2011 at 6​:34 PM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Thu Aug 04 01​:11​:45 2011, fraserbn@​gmail.com wrote​:

On Tue, Aug 2, 2011 at 3​:36 PM, Jan Dubois <jand@​activestate.com> wrote​:

On Tue, 02 Aug 2011, Leon Timmermans wrote​:

Perl modules/scripts are text files, not binary files. I don't think
"​:raw" really is the correct thing to do, though it may work out in
the end.

No, it won't. It will switch the DATA filehandle to :raw mode as well,
so you'll end up with "\r\n" line endings on Windows.

Perl actually used to read the source in :raw mode on Windows, but
(incompletely) turned on text mode when it reached the __DATA__ token.
This created quite a number of bugs, all merged into this ticket​:

https://rt-archive.perl.org/perl5/Public/Bug/Display.html?id=28106

For Perl 5.14 I changed the default to reading source code in text
mode,
which fixed all those bugs. I'll soon remove the broken code that
unsuccessfully tries to switch on text-mode mid-stream because it
appears that nobody knows how to fix it (and it is unrelated to the
open
pragma anyways).

Cheers,
-Jan

Alright, then I think that Father C's suggestion is our winner here.

Except it can’t work properly, because SAVEHINTS and PL_hints deal with
*compile-time* hints, whereas PerlIO looks at the hints in the
already-compiled op tree that is running (in PL_curcop, in other words).

PL_curcop->cop_hints can’t be modified, because that won’t be thread-safe.

Unfortunately I can't get SAVEHINTS(); to play nice (my ignorance of
PerlIO
is only surpassed by my inexperience with scope.c), so I can't volunteer
myself for this one. But if it helps, from my hamhanded testing, all you
need to do is clear PL_curcop->cop_hints of HINT_LEXICAL_IO_(IN|OUT)
before
calling PerlIO_openn.

Bit of a surprising discovery; Turns out that passing a single "​:" for the
layers skips the fetch from the context layers​:
perl -wE 'use open qw( :encoding(UTF-8) ); open my $fh, "<​:", "etc"; say
PerlIO​::get_layers($fh);'

That will only get the relevant default layers, while removing the colons
makes it work as usual -- So we can abuse this (mis)feature to fix the bug​:

Inline Patch
diff --git a/pp_ctl.c b/pp_ctl.c
index 3a2de27..a9c0afd 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@@ -3692,7 +3692,7 @@ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name);
+ return PerlIO_openn(aTHX_ "​:", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif

(tests in a previous mail)

Is this worth documenting somewhere?

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2011

From @rgarcia

On 22 September 2011 16​:29, Brian Fraser <fraserbn@​gmail.com> wrote​:

Bit of a surprising discovery; Turns out that passing a single "​:" for the
layers skips the fetch from the context layers​:
perl -wE 'use open qw( :encoding(UTF-8) ); open my $fh, "<​:", "etc"; say
PerlIO​::get_layers($fh);'
That will only get the relevant default layers, while removing the colons
makes it work as usual -- So we can abuse this (mis)feature to fix the bug​:
diff --git a/pp_ctl.c b/pp_ctl.c
index 3a2de27..a9c0afd 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@​@​ -3692,7 +3692,7 @​@​ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name);
+ return PerlIO_openn(aTHX_ "​:", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif
(tests in a previous mail)
Is this worth documenting somewhere?

I vote yes, because I find this accidental feature actually neat. And
doubly so if we use it in the core...

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2011

From tchrist@perl.com

Rafael Garcia-Suarez <rgs@​consttype.org> wrote
  on Thu, 22 Sep 2011 16​:35​:40 +0200​:

I vote yes, because I find this accidental feature actually neat. And
doubly so if we use it in the core...

My instinct says that you're right. It is always cool to find new uses
for our tools that their inventor never dreamt of.

I *REALLY* wish I understood layers better. I can do simple stuff,
but not fancy bits, and there are some things for which fanciness is
still called for. Is there any other/better documentation of all this
than what's in PerlIO(3) and the tiny bits in perlfunc?

thanks,

--tom

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2011

From @rgarcia

On 22 September 2011 17​:09, Tom Christiansen <tchrist@​perl.com> wrote​:

I *REALLY* wish I understood layers better.  I can do simple stuff,
but not fancy bits, and there are some things for which fanciness is
still called for.  Is there any other/better documentation of all this
than what's in PerlIO(3) and the tiny bits in perlfunc?

There's the scary low-level docs in perliol.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 23, 2011

From @cpansprout

On Thu Sep 22 07​:30​:00 2011, fraserbn@​gmail.com wrote​:

Bit of a surprising discovery; Turns out that passing a single "​:" for the
layers skips the fetch from the context layers​:
perl -wE 'use open qw( :encoding(UTF-8) ); open my $fh, "<​:", "etc"; say
PerlIO​::get_layers($fh);'

That will only get the relevant default layers, while removing the colons
makes it work as usual -- So we can abuse this (mis)feature to fix the
bug​:

diff --git a/pp_ctl.c b/pp_ctl.c
index 3a2de27..a9c0afd 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@​@​ -3692,7 +3692,7 @​@​ S_check_type_and_open(pTHX_ SV *name)
}

#if !defined(PERLIO_IS_STDIO) && !defined(USE_SFIO)
- return PerlIO_openn(aTHX_ NULL, PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name);
+ return PerlIO_openn(aTHX_ "​:", PERL_SCRIPT_MODE, -1, 0, 0, NULL, 1,
&name);
#else
return PerlIO_open(p, PERL_SCRIPT_MODE);
#endif

Thank you. Applied as 639dfab.

(tests in a previous mail)

I didn’t read that sentence until I had already written one. It turns
out that your test, however, passes even without the bug fix, because
the error you were testing for is actually a warning. So I went with my
test (commit 6406c52), as it was easier than fixing yours. :-)

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 23, 2011

From @cpansprout

On Thu Sep 22 07​:30​:00 2011, fraserbn@​gmail.com wrote​:

Is this worth documenting somewhere?

Yes, and I have documented it with commit c0fd9d2.

@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 23, 2011

@cpansprout - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this Sep 23, 2011
@p5pRT

This comment has been minimized.

Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 23, 2011

From @Leont

On Thu, Sep 22, 2011 at 5​:09 PM, Tom Christiansen <tchrist@​perl.com> wrote​:

I *REALLY* wish I understood layers better.  I can do simple stuff,
but not fancy bits, and there are some things for which fanciness is
still called for.  Is there any other/better documentation of likeall this
than what's in PerlIO(3) and the tiny bits in perlfunc?

Barely. perliol defines some internals but that's not useful for
anyone who isn't implementing a layer in C (which very few people do,
and even fewer do it correctly).

Most of it is reading the source and trying things out. That's also
why hardly anyone ever uses them, other than the unavoidable
:encoding.

Leon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.