Can open() to an in-memory scalar return false?

[p5pRT]

Migrated from rt.perl.org#131546 (status was 'resolved')

Searchable as RT131546$

[p5pRT]

From @epa

Created by @epa

perlfunc has some example code like

  open(MEMORY, ">", \$var)
  or die "Can't open memory file​: $!";

Under what circumstances would that open() call return false?
If it did return false, would the error really be in the $! variable?

I would have expected that opening a filehandle to a scalar would be
something that always succeeds, unless the scalar is readonly or
otherwise weird, in which case it just dies rather than returning
false. So the 'or die' is redundant and somewhat misleading.

If I'm wrong, and open() to a scalar can return false for some cases,
the docs should say what those cases are.

Perl Info

Flags:
    category=docs
    severity=low

Site configuration information for perl 5.22.2:

Configured by Red Hat, Inc. at Fri Nov  4 14:35:02 UTC 2016.

Summary of my perl5 (revision 5 version 22 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=4.7.9-200.fc24.x86_64, archname=x86_64-linux-thread-multi
    uname='linux buildvm-12.phx2.fedoraproject.org 4.7.9-200.fc24.x86_64 #1 smp thu oct 20 14:26:16 utc 2016 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Dldflags=-Wl,-z,relro  -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro  -Dlddlflags=-shared -Wl,-z,relro  -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.22.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='  -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fwrapv -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='5.3.1 20160406 (Red Hat 5.3.1-6)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-Wl,-z,relro  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib
    libs=-lpthread -lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lresolv -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.22'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro '
    cccdlflags='-fPIC', lddlflags='-shared -Wl,-z,relro  -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    Fedora Patch1: Removes date check, Fedora/RHEL specific
    Fedora Patch3: support for libdir64
    Fedora Patch4: use libresolv instead of libbind
    Fedora Patch5: USE_MM_LD_RUN_PATH
    Fedora Patch6: Skip hostname tests, due to builders not being network capable
    Fedora Patch7: Dont run one io test due to random builder failures
    Fedora Patch15: Define SONAME for libperl.so
    Fedora Patch16: Install libperl.so to -Dshrpdir value
    Fedora Patch22: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015)
    Fedora Patch26: Make *DBM_File desctructors thread-safe (RT#61912)
    Fedora Patch27: Make PadlistNAMES() lvalue again (CPAN RT#101063)
    Fedora Patch28: Make magic vtable writable as a work-around for Coro (CPAN RT#101063)
    Fedora Patch29: Fix duplicating PerlIO::encoding when spawning threads (RT#31923)
    Fedora Patch30: Do not let XSLoader load relative paths (CVE-2016-6185)
    Fedora Patch31: Avoid loading optional modules from default . (CVE-2016-1238)
    Fedora Patch32: Fix a crash in lexical scope warnings (RT#128597)
    Fedora Patch33: Do not mangle errno from failed socket calls (RT#128316)
    Fedora Patch34: Fix crash in "evalbytes S" (RT#129196)
    Fedora Patch35: Fix crash in "evalbytes S" (RT#129196)
    Fedora Patch36: Fix crash in "evalbytes S" (RT#129196)
    Fedora Patch37: Fix crash in splice (RT#129164, RT#129166, RT#129167)
    Fedora Patch38: Fix string overrun in Perl_gv_fetchmethod_pvn_flags (RT#129267)
    Fedora Patch39: Fix string overrun in Perl_gv_fetchmethod_pvn_flags (RT#129267)
    Fedora Patch40: Fix string overrun in Perl_gv_fetchmethod_pvn_flags (RT#129267)
    Fedora Patch41: Fix string overrun in Perl_gv_fetchmethod_pvn_flags (RT#129267)
    Fedora Patch42: Fix string overrun in Perl_gv_fetchmethod_pvn_flags (RT#129267)
    Fedora Patch43: Fix crash when matching UTF-8 string with non-UTF-8 substrings (RT#129350)
    Fedora Patch44: Fix parsing perl options in shell bang line (RT#129336)
    Fedora Patch45: Fix firstchar bitmap under UTF-8 with prefix optimization (RT#129950)
    Fedora Patch46: Avoid infinite loop in h2xs tool if enum and type have the same name (RT130001)
    Fedora Patch47: Fix stack handling when calling chdir without an argument (RT#129130)
    Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux
    Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux


@INC for perl 5.22.2:
    /home/eda/lib64/perl5/
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5


Environment for perl 5.22.2:
    HOME=/home/eda
    LANG=en_GB.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_GB.UTF-8
    LC_MESSAGES=en_GB.UTF-8
    LC_MONETARY=en_GB.UTF-8
    LC_NUMERIC=en_GB.UTF-8
    LC_TIME=en_GB.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/eda/bin:/home/eda/bin:/usr/local/bin:/usr/bin:/sbin:/usr/sbin:/sbin:/usr/sbin
    PERL5LIB=/home/eda/lib64/perl5/
    PERL_BADLANG (unset)
    SHELL=/bin/bash

Please ignore autogenerated disclaimer below this point. 

[p5pRT]

From @tonycoz

On Fri, 09 Jun 2017 09​:33​:58 -0700, eda@​waniasset.com wrote​:

perlfunc has some example code like

open(MEMORY, ">", \$var)
or die "Can't open memory file​: $!";

Under what circumstances would that open() call return false?
If it did return false, would the error really be in the $! variable?

I would have expected that opening a filehandle to a scalar would be
something that always succeeds, unless the scalar is readonly or
otherwise weird, in which case it just dies rather than returning
false. So the 'or die' is redundant and somewhat misleading.

If I'm wrong, and open() to a scalar can return false for some cases,
the docs should say what those cases are.

opening to a read-only scalar returns false (it doesn't die)​:

$ ./perl -Ilib -E 'say ">",open(my $fh, ">", \0), "<"; say $!';

<
Invalid argument

opening without truncating on a scalar with code points over 0xff can also fail​:

$ ./perl -Ilib -E '$x = "\x{100}"; say ">",open(my $fh, "&gt;&gt;", \$x), "&lt;"; say $!';

<
Invalid argument

opening with an unknown layer will also fail​:

$ ./perl -Ilib -E '$x = "\x{100}"; say ">",open(my $fh, "&gt;​:foo", \$x), "&lt;"; say $!';

<
No such file or directory

Only one of those is unique to scalars (writing to read-only scalars is conceptually the same as writing to read-only files) and they all produce reasonable warnings if warnings are enabled.

I've attached a patch against perlfunc documenting the code point limitation.

Tony

[p5pRT]

From @tonycoz

0001-perl-131546-note-code-point-limitation-for-in-memory.patch

From ddb23c4046193709d450aaa4a58a002f5273bd5d Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Mon, 12 Jun 2017 11:03:35 +1000
Subject: (perl #131546) note code point limitation for in-memory files

---
 pod/perlfunc.pod | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/pod/perlfunc.pod b/pod/perlfunc.pod
index bb0383f..b47039a 100644
--- a/pod/perlfunc.pod
+++ b/pod/perlfunc.pod
@@ -4530,6 +4530,10 @@ To (re)open C<STDOUT> or C<STDERR> as an in-memory file, close it first:
     open(STDOUT, ">", \$variable)
 	or die "Can't open STDOUT: $!";
 
+The scalars for in-memory files are treated as octet strings: unless
+the file is being opened with truncation the scalar may not contain
+any code points over 0xFF.
+
 See L<perliol> for detailed info on PerlIO.
 
 General examples:
-- 
2.1.4

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @epa

opening to a read-only scalar returns false (it doesn't die)​:

$ ./perl -Ilib -E 'say ">",open(my $fh, "&gt;", \0), "&lt;"; say $!';

<
Invalid argument

Actually, it depends​:

$ perl -MReadonly -E 'Readonly my $x; say ">",open(my $fh, "&gt;", \$x), "&lt;"; say $!';
Modification of a read-only value attempted at -e line 1

Can this be made more consistent so it either always dies or always returns false?

There is a further difference between your example and that in the manual page.
You take a reference to the constant 0. But the case I am asking about is a reference to a scalar variable
(sorry I did not make this clear enough in the initial report). So

  open $fh, '&gt;', \$x or die "can this ever be printed? $!";

I don't see how this can return false​: if $x is readonly then it fails before that point, as above.
Are there any circumstances where the error message wil be printed?

Thanks for your other two examples where you show that the '>>' or '>​:foo' file modes can return false given a scalar.
I am asking particularly about '>' here (but again, I didn't narrow down the question enough).

(writing to read-only scalars
is conceptually the same as writing to read-only files)

In that case shouldn't the error code be the same as the one you get for doing that?
I don't have a strong view but I will note that $!, which can only hold classical C library error codes, seems to be a poor fit here.
Some better way to report errors would be useful.

[p5pRT]

From @Abigail

On Sun, Jun 11, 2017 at 10​:57​:35PM -0700, Ed Avis via RT wrote​:

opening to a read-only scalar returns false (it doesn't die)​:

$ ./perl -Ilib -E 'say ">",open(my $fh, "&gt;", \0), "&lt;"; say $!';

<
Invalid argument

Actually, it depends​:

$ perl -MReadonly -E 'Readonly my $x; say ">",open(my $fh, "&gt;", \$x), "&lt;"; say $!';
Modification of a read-only value attempted at -e line 1

Can this be made more consistent so it either always dies or always returns false?

There is a further difference between your example and that in the manual page.
You take a reference to the constant 0. But the case I am asking about is a reference to a scalar variable
(sorry I did not make this clear enough in the initial report). So

open $fh\, '>'\, \\$x or die "can this ever be printed? $\!";

I don't see how this can return false​: if $x is readonly then it fails before that point, as above.
Are there any circumstances where the error message wil be printed?

  $ perl -E 'for (1) {open my $fh, "&gt;", \$_ or die "--&gt; $!";}'
  --> Invalid argument at -e line 1.
  $

Abigail

[p5pRT]

From @epa

Thanks Abigail, and to show it's not something special with $_​:

% perl -E 'for $x (1) {open my $fh, "&gt;", \$x or die "--&gt; $!";}'

So the error check is necessary in general and the example in perlfunc is correct. What about

  open my $fh, '>', \ my $x;

Can that fail?

[p5pRT]

From @tonycoz

On Sun, 11 Jun 2017 22​:57​:34 -0700, ed wrote​:

opening to a read-only scalar returns false (it doesn't die)​:

$ ./perl -Ilib -E 'say ">",open(my $fh, "&gt;", \0), "&lt;"; say $!';

<
Invalid argument

Actually, it depends​:

$ perl -MReadonly -E 'Readonly my $x; say ">",open(my $fh, ">", \$x),
"<"; say $!';
Modification of a read-only value attempted at -e line 1

Can this be made more consistent so it either always dies or always
returns false?

That's because Readonly​::Readonly() implements read-onlyness via a tie rather than by marking the SV read only​:

https://metacpan.org/source/SANKO/Readonly-2.05/lib/Readonly.pm#L409

(try Devel​::Peek's Dump() on your scalar)

If you use Readonly​::Scalar1 or SvREADONLY() instead​:

tony@​mars​:.../git/perl$ ~/perl/5.22.0/bin/perl -MReadonly=Scalar1 -wE 'Scalar1 my $x => undef; say ">", open( my $fh, ">", \$x); say "<"; '
Modification of a read-only value attempted at -e line 1.
Use of uninitialized value in say at -e line 1.

<
tony@​mars​:.../git/perl$ ./perl -Ilib -wE 'Internals​::SvREADONLY(my $x, 1); say ">", open( my $fh, ">", \$x); say "<"; '
Modification of a read-only value attempted at -e line 1.
Use of uninitialized value in say at -e line 1.

<

you get only a warning.

There is a further difference between your example and that in the
manual page.
You take a reference to the constant 0. But the case I am asking
about is a reference to a scalar variable
(sorry I did not make this clear enough in the initial report). So

open $fh, '&gt;', \$x or die "can this ever be printed? $!";

Abigail covered this.

I don't see how this can return false​: if $x is readonly then it fails
before that point, as above.
Are there any circumstances where the error message wil be printed?

Not using Readonly​::Readonly() fixes this.

Thanks for your other two examples where you show that the '>>' or
'>​:foo' file modes can return false given a scalar.
I am asking particularly about '>' here (but again, I didn't narrow
down the question enough).

(writing to read-only scalars
is conceptually the same as writing to read-only files)

In that case shouldn't the error code be the same as the one you get
for doing that?
I don't have a strong view but I will note that $!, which can only
hold classical C library error codes, seems to be a poor fit here.
Some better way to report errors would be useful.

The cases above all produce a warning, which can be captured with $SIG{__WARN__}.

I'll produce a patch for some better $! values.

Tony

[p5pRT]

From @petdance

I'll produce a patch for some better $! values.

I think the key to this ticket is that perlfunc should say that in-memory files can still fail on open, and that it would be good to explicitly say that. It’s not at all unreasonable for a user to quickly conclude “It’s opening in memory? That can never fail, so I don’t need to check the return from open."

[p5pRT]

From @tonycoz

On Wed, 21 Jun 2017 00​:09​:15 -0700, tonyc wrote​:

I'll produce a patch for some better $! values.

Tony

Attached, this includes the note I had in my previous patch, an extra note as suggested by Andy, and changing the $! set when the scalar is read only.

Tony

[p5pRT]

From @tonycoz

scalar-open.patch

From b8cabe26715d7db6eeec54f9928efec8d6beb553 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Mon, 12 Jun 2017 11:03:35 +1000
Subject: (perl #131546) some notes on in-memory files for open()

---
 pod/perlfunc.pod | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/pod/perlfunc.pod b/pod/perlfunc.pod
index bb0383f..c7dcbd0 100644
--- a/pod/perlfunc.pod
+++ b/pod/perlfunc.pod
@@ -4530,6 +4530,13 @@ To (re)open C<STDOUT> or C<STDERR> as an in-memory file, close it first:
     open(STDOUT, ">", \$variable)
 	or die "Can't open STDOUT: $!";
 
+The scalars for in-memory files are treated as octet strings: unless
+the file is being opened with truncation the scalar may not contain
+any code points over 0xFF.
+
+Opening in-memory files I<can> fail for a variety of reasons, as with
+any other open check the return value for success.
+
 See L<perliol> for detailed info on PerlIO.
 
 General examples:
-- 
2.1.4


From 0fd5ed42acad11d081f56f2c34903138626ce610 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 21 Jun 2017 17:07:02 +1000
Subject: (perl #131546) return a reasonable error code opening a read-only
 scalar

---
 ext/PerlIO-scalar/scalar.xs  | 2 +-
 ext/PerlIO-scalar/t/scalar.t | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/ext/PerlIO-scalar/scalar.xs b/ext/PerlIO-scalar/scalar.xs
index 4c5d2c1..5c2ff5f 100644
--- a/ext/PerlIO-scalar/scalar.xs
+++ b/ext/PerlIO-scalar/scalar.xs
@@ -43,7 +43,7 @@ PerlIOScalar_pushed(pTHX_ PerlIO * f, const char *mode, SV * arg,
 	     && mode && *mode != 'r') {
 		if (ckWARN(WARN_LAYER))
 		    Perl_warner(aTHX_ packWARN(WARN_LAYER), "%s", PL_no_modify);
-		SETERRNO(EINVAL, SS_IVCHAN);
+		SETERRNO(EACCES, RMS_PRV);
 		return -1;
 	    }
 	    s->var = SvREFCNT_inc(SvRV(arg));
diff --git a/ext/PerlIO-scalar/t/scalar.t b/ext/PerlIO-scalar/t/scalar.t
index 3dfcced..bd06d64 100644
--- a/ext/PerlIO-scalar/t/scalar.t
+++ b/ext/PerlIO-scalar/t/scalar.t
@@ -13,10 +13,11 @@ BEGIN {
 }
 
 use Fcntl qw(SEEK_SET SEEK_CUR SEEK_END); # Not 0, 1, 2 everywhere.
+use Errno qw(EACCES);
 
 $| = 1;
 
-use Test::More tests => 122;
+use Test::More tests => 123;
 
 my $fh;
 my $var = "aaa\n";
@@ -185,6 +186,7 @@ EOF
 
     my $ro = \43;
     ok(!(defined open(F, '>', $ro)), $!);
+    is($!+0, EACCES, "check we get a read-onlyish error code");
     close F;
     # but we can read from it
     ok(open(F, '<', $ro), $!);
-- 
2.1.4


From 29e859c47f784b17d3d38229098c0fe41787575e Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 21 Jun 2017 17:07:17 +1000
Subject: bump $PerlIO::scalar::VERSION

---
 ext/PerlIO-scalar/scalar.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ext/PerlIO-scalar/scalar.pm b/ext/PerlIO-scalar/scalar.pm
index bcbb56c..0276f21 100644
--- a/ext/PerlIO-scalar/scalar.pm
+++ b/ext/PerlIO-scalar/scalar.pm
@@ -1,5 +1,5 @@
 package PerlIO::scalar;
-our $VERSION = '0.27';
+our $VERSION = '0.28';
 require XSLoader;
 XSLoader::load();
 1;
-- 
2.1.4

[p5pRT]

From @petdance

On Jun 21, 2017, at 8​:33 PM, Tony Cook via RT <perlbug-followup@​perl.org> wrote​:

+Opening in-memory files I<can> fail for a variety of reasons, as with
+any other open check the return value for success.

The punctuation is a little misleading. It should be "Opening in-memory files I<can> fail for a variety of reasons. As with any other open, check the return value for success." (Or maybe "any other C<open>"?)

--
Andy Lester => www.petdance.com

[p5pRT]

From @tonycoz

On Wed, 21 Jun 2017 19​:16​:33 -0700, petdance wrote​:

On Jun 21, 2017, at 8​:33 PM, Tony Cook via RT <perlbug-
followup@​perl.org> wrote​:

+Opening in-memory files I<can> fail for a variety of reasons, as
with
+any other open check the return value for success.

The punctuation is a little misleading. It should be "Opening in-
memory files I<can> fail for a variety of reasons. As with any other
open, check the return value for success." (Or maybe "any other
C<open>"?)

Applied as 54e7f05, 45ef9d4 and aef23d3 with your suggested changes.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0, this and 185 other issues have been
resolved.

Perl 5.28.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists, feel free to reopen this ticket.