Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Scalar found where operator expected" on printing to indirect filehandle within s///e #16847

Closed
p5pRT opened this issue Feb 15, 2019 · 9 comments
Closed

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Feb 15, 2019

Migrated from rt.perl.org#133850 (status was 'pending release')

Searchable as RT133850$

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 15, 2019

From @Corion

Created by corion@corion.net

This is a bug report for perl from corion@​corion.net,
generated with the help of perlbug 1.40 running under perl 5.26.0.

-----------------------------------------------------------------
Hi all,

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

  perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the code​:

  Scalar found where operator expected at -e line 1, near "$fh $1"
  (Missing operator before $1?)
  Name "main​::fh" used only once​: possible typo at -e line 1.
  Use of uninitialized value $_ in substitution (s///) at -e line 1.
  Use of uninitialized value $_ in substitution (s///) at -e line 1.
  Can't use an undefined value as a symbol reference at -e line 1.

The code even outputs 'foo' in the below version​:

  perl -wle "my $fh = \*STDOUT; $_ = ''; s//print $fh 'foo'/e"
  String found where operator expected at -e line 1, near "$fh 'foo'"
  (Missing operator before 'foo'?)
  foo

The "fix" for this (highly unconventional) code is to use curly brackets
around the filehandle, forcing the "complicated expression" version of
print​:

  perl -wle "my $fh = \*STDOUT; $_ = ''; s//print {$fh} 'foo'/e"
  foo

I still think that the original code should parse without error, even
though it
is really weird and I'm no fan of side-effects during regexp evaluation.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.26.0:

Configured by strawberry-perl at Wed May 31 04:46:12 2017.

Summary of my perl5 (revision 5 version 26 subversion 0) configuration:

  Platform:
    osname=MSWin32
    osvers=6.3
    archname=MSWin32-x64-multi-thread
    uname='Win32 strawberry-perl 5.26.0.1 #1 Wed May 31 04:42:31 2017 x64'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE
-D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing
-mms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='7.1.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags ='-s -L"c:\strawberry-perl-5.26.0.1-64bit\perl\lib\CORE"
-L"c:\strawberry-perl-5.26.0.1-64bit\c\lib"'
    libpth=c:\strawberry-perl-5.26.0.1-64bit\c\lib
c:\strawberry-perl-5.26.0.1-64bit\c\x86_64-w64-mingw32\lib
c:\strawberry-perl-5.26.0.1-64bit\c\lib\gcc\x86_64-w64-mingw32\7.1.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr
-lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl526.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s
-L"c:\strawberry-perl-5.26.0.1-64bit\perl\lib\CORE"
-L"c:\strawberry-perl-5.26.0.1-64bit\c\lib"'



@INC for perl 5.26.0:
    C:/strawberry-perl-5.26.0.1-64bit/perl/site/lib/MSWin32-x64-multi-thread
    C:/strawberry-perl-5.26.0.1-64bit/perl/site/lib
    C:/strawberry-perl-5.26.0.1-64bit/perl/vendor/lib
    C:/strawberry-perl-5.26.0.1-64bit/perl/lib


Environment for perl 5.26.0:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\Program Files\Microsoft IntelliType Pro\;C:\Program
Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\bin;C:\Program Files\NVIDIA
GPU Computing
Toolkit\CUDA\v9.2\libnvvp;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
Files\Git\cmd;C:\Program Files\PuTTY;C:\Program Files\MiKTeX
2.9\miktex\bin\x64\;C:\strawberry-perl-5.20.1.1-x64\c\bin;C:\strawberry-perl-5.20.1.1-x64\perl\site\bin;C:\strawberry-perl-5.20.1.1-x64\perl\bin;C:\Program
Files (x86)\Skype\Phone\;C:\Program Files (x86)\NVIDIA
Corporation\PhysX\Common;C:\Program Files
(x86)\GtkSharp\2.12\bin;C:\Program Files
(x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\nodejs\;C:\Program
Files\NVIDIA Corporation\NVIDIA
NvDLISR;C:\Users\Corion\AppData\Local\Programs\Python\Python36\Scripts\;C:\Users\Corion\AppData\Local\Programs\Python\Python36\;C:\Program
Files\Docker
Toolbox;C:\ProgramData\chocolatey\bin;C:\strawberry-perl-5.26.0.1-64bit\perl\bin;c:\strawberry-perl-5.26.0.1-64bit\c\bin;C:\strawberry-perl-5.26.0.1-64bit\c\x86_64-w64-mingw32\bin;C:\Users\Corion\AppData\Roaming\npm
    PERL_BADLANG (unset)
    SHELL (unset)

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 19, 2019

From @tonycoz

On Fri, 15 Feb 2019 11​:37​:16 -0800, corion@​corion.net wrote​:

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the
code​:

Scalar found where operator expected at -e line 1, near "$fh $1"
(Missing operator before $1?)

This isn't a syntax error, but a warning (but still buggy).

The code inside the replacement part of s///e is handled with a sublex, and the code that decides whether an operator or term is expected after C<print $fh> is mostly skipped for a sublex.

The attached fixes that, doing the same checks for C<print $fh> in a sublex within brackets as for a normal parse.

This fixes the same problem for C<< "... ${code} ..." >>.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 19, 2019

From @tonycoz

0001-perl-133850-fix-parsing-hints-for-print-fh-foo-in-s-.patch
From 9e0f09ef2afa82ef1a96cba981d4fca094bba484 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 19 Feb 2019 15:38:39 +1100
Subject: (perl #133850) fix parsing hints for print $fh "foo" in s///e

The replacement code in s///e is parsed using a sublex, similarly
to double-quotish replacement.

The code that handles C< $id > tries to detect what the next token
should be, but only special cased code similar to C< print $fh ... >
in a non-sublex, so s/../print $fh $1/e set PL_expect to XOPERATOR
which caused the parser to complain when the $1 wasn't an operator.

The fix here handles two cases:

 s/.../code here/e

In this case the code is inserted into a do {} block like:

  do { code here }

so PL_lex_brackets ends up non-zero.

The second case is code like:

  "xxx${code here}"

which had the same problem, which this also fixes.
---
 t/lib/warnings/toke | 14 ++++++++++++++
 toke.c              |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index 1c85d7bc44..a500856283 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1675,3 +1675,17 @@ Execution of - aborted due to compilation errors.
 use utf8;
 qw���foo ��� ��� bar���
 EXPECT
+########
+# NAME [perl #133850] print $fh $1 in s///e expression
+use warnings;
+my $fh = \*STDOUT;
+$_ = "abc";
+s/(x)/ print $fh $1 /e;
+EXPECT
+########
+# NAME [perl #133850] side case
+use warnings;
+my $fh = \*STDOUT;
+my $y = "";
+my $x = "${print $fh $y; \'x'}";
+EXPECT
diff --git a/toke.c b/toke.c
index 5a3fe78472..0eb63fcb55 100644
--- a/toke.c
+++ b/toke.c
@@ -6772,7 +6772,7 @@ Perl_yylex(pTHX)
 	    }
 
 	    PL_expect = XOPERATOR;
-	    if (PL_lex_state == LEX_NORMAL && isSPACE((char)tmp)) {
+	    if ((PL_lex_state == LEX_NORMAL || PL_lex_brackets) && isSPACE((char)tmp)) {
 		const bool islop = (PL_last_lop == PL_oldoldbufptr);
 		if (!islop || PL_last_lop_op == OP_GREPSTART)
 		    PL_expect = XOPERATOR;
-- 
2.11.0

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 19, 2019

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 19, 2019

From @tonycoz

On Mon, 18 Feb 2019 20​:44​:26 -0800, tonyc wrote​:

This fixes the same problem for C<< "... ${code} ..." >>.

There's some other similar cases - i.e. where PL_lex_state is checked for LEX_NORMAL but no other checks are made. The simplest I was able to reproduce was​:

$ ./perl -we '%x = qw(a b c d); $_ = ""; $_ = @​x {a}'
Scalar value @​x{"a"} better written as $x{"a"} at -e line 1.
$ ./perl -we '%x = qw(a b c d); $_ = ""; s/^/ @​x {a} /e'
(no warning)

Kind of unlikely​:

tony@​mars​:.../git/perl$ ./perl -Ilib -we '{ my $x : shared = 1; }'
tony@​mars​:.../git/perl$ ./perl -Ilib -we '$_ = ""; s/^/ { my $x : shared = 1; } /e'
Unquoted string "shared" may clash with future reserved word at -e line 1.
syntax error at -e line 1, near "$x :"

Another​:

$ ./perl -we '$a = ${time}'
Ambiguous use of ${time} resolved to $time at -e line 1.
Name "main​::time" used only once​: possible typo at -e line 1.
$ ./perl -we '$_ = ""; s/^/ ${time} /e'
Name "main​::time" used only once​: possible typo at -e line 1.
Use of uninitialized value $time in substitution (s///) at -e line 1.
(no ambiguous warning)

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented May 29, 2019

From @tonycoz

On Mon, 18 Feb 2019 20​:44​:26 -0800, tonyc wrote​:

On Fri, 15 Feb 2019 11​:37​:16 -0800, corion@​corion.net wrote​:

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the
code​:

Scalar found where operator expected at -e line 1, near "$fh $1"
(Missing operator before $1?)

This isn't a syntax error, but a warning (but still buggy).

The code inside the replacement part of s///e is handled with a
sublex, and the code that decides whether an operator or term is
expected after C<print $fh> is mostly skipped for a sublex.

The attached fixes that, doing the same checks for C<print $fh> in a
sublex within brackets as for a normal parse.

This fixes the same problem for C<< "... ${code} ..." >>.

Rebased on blead, and fixes for the related issues.

I'll apply this in a few days.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented May 29, 2019

From @tonycoz

subparse-tokens.patch
From cb43f46efc62573c3470f5c2de01ee0e5b6ffd4a Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 19 Feb 2019 15:38:39 +1100
Subject: (perl #133850) fix parsing hints for print $fh "foo" in s///e

The replacement code in s///e is parsed using a sublex, similarly
to double-quotish replacement.

The code that handles C< $id > tries to detect what the next token
should be, but only special cased code similar to C< print $fh ... >
in a non-sublex, so s/../print $fh $1/e set PL_expect to XOPERATOR
which caused the parser to complain when the $1 wasn't an operator.

The fix here handles two cases:

 s/.../code here/e

In this case the code is inserted into a do {} block like:

  do { code here }

so PL_lex_brackets ends up non-zero.

The second case is code like:

  "xxx${code here}"

which had the same problem, which this also fixes.
---
 t/lib/warnings/toke | 14 ++++++++++++++
 toke.c              |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index 83641e5106..944e5d92f4 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1691,3 +1691,17 @@ EXPECT
 OPTION regex
 Malformed UTF-8 character: .*non-continuation.*
 The eval did not crash the program
+########
+# NAME [perl #133850] print $fh $1 in s///e expression
+use warnings;
+my $fh = \*STDOUT;
+$_ = "abc";
+s/(x)/ print $fh $1 /e;
+EXPECT
+########
+# NAME [perl #133850] side case
+use warnings;
+my $fh = \*STDOUT;
+my $y = "";
+my $x = "${print $fh $y; \'x'}";
+EXPECT
diff --git a/toke.c b/toke.c
index b0fa2c35ba..dc376f6beb 100644
--- a/toke.c
+++ b/toke.c
@@ -6838,7 +6838,7 @@ Perl_yylex(pTHX)
 	    }
 
 	    PL_expect = XOPERATOR;
-	    if (PL_lex_state == LEX_NORMAL && isSPACE((char)tmp)) {
+	    if ((PL_lex_state == LEX_NORMAL || PL_lex_brackets) && isSPACE((char)tmp)) {
 		const bool islop = (PL_last_lop == PL_oldoldbufptr);
 		if (!islop || PL_last_lop_op == OP_GREPSTART)
 		    PL_expect = XOPERATOR;
-- 
2.11.0


From 2b67caa9b5055eb7441998a9538af722639982af Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 29 May 2019 11:46:01 +1000
Subject: (perl #133850) warn on ${time} even in sub-parse

Code like:

  $a = ${time}

warns with:

  Ambiguous use of ${time} resolved to $time at ...

but:

  s/^/ ${time} /e

didn't, since the parser is in a special state in sub-parses.
---
 t/lib/warnings/toke | 8 ++++++++
 toke.c              | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index 944e5d92f4..eab22fbbc9 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1705,3 +1705,11 @@ my $fh = \*STDOUT;
 my $y = "";
 my $x = "${print $fh $y; \'x'}";
 EXPECT
+########
+# NAME [perl #133850] another case
+use warnings;
+my $time = 1;
+$_ = "";
+s/^/ ${time} /e
+EXPECT
+Ambiguous use of ${time} resolved to $time at - line 4.
diff --git a/toke.c b/toke.c
index dc376f6beb..c42d4fd802 100644
--- a/toke.c
+++ b/toke.c
@@ -9660,7 +9660,7 @@ S_scan_ident(pTHX_ char *s, char *dest, STRLEN destlen, I32 ck_uni)
 		PL_lex_state = LEX_INTERPEND;
 		PL_expect = XREF;
 	    }
-	    if (PL_lex_state == LEX_NORMAL) {
+	    if (PL_lex_state == LEX_NORMAL || PL_lex_brackets) {
 		if (ckWARN(WARN_AMBIGUOUS)
                     && (keyword(dest, d - dest, 0)
 		        || get_cvn_flags(dest, d - dest, is_utf8
-- 
2.11.0


From f623089c7b47c9de58667b70063be135948356bd Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 29 May 2019 13:47:36 +1000
Subject: (perl #133850) allow @x{...} with space in sub-parse

---
 t/lib/warnings/toke | 8 ++++++++
 toke.c              | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index eab22fbbc9..5dbef1ab62 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1713,3 +1713,11 @@ $_ = "";
 s/^/ ${time} /e
 EXPECT
 Ambiguous use of ${time} resolved to $time at - line 4.
+########
+# NAME [perl #133850] another case
+use warnings;
+%x = qw(a b c d);
+$_ = "";
+s/^/ @x {a} /e
+EXPECT
+Scalar value @x{"a"} better written as $x{"a"} at - line 4.
diff --git a/toke.c b/toke.c
index c42d4fd802..e1b163334f 100644
--- a/toke.c
+++ b/toke.c
@@ -6911,7 +6911,7 @@ Perl_yylex(pTHX)
 	if (!PL_tokenbuf[1]) {
 	    PREREF('@');
 	}
-	if (PL_lex_state == LEX_NORMAL)
+	if (PL_lex_state == LEX_NORMAL || PL_lex_brackets)
 	    s = skipspace(s);
 	if (   (PL_expect != XREF || PL_oldoldbufptr == PL_last_lop)
             && intuit_more(s, PL_bufend))
-- 
2.11.0


From d50d49c97634fa03d472f06bf4e70fe7dfc17fd2 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 29 May 2019 14:37:25 +1000
Subject: (perl #133850) allow attributes in sub-parses

---
 t/op/attrs.t | 3 +++
 toke.c       | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/t/op/attrs.t b/t/op/attrs.t
index 649525ce22..dd230b3fbd 100644
--- a/t/op/attrs.t
+++ b/t/op/attrs.t
@@ -507,4 +507,7 @@ BEGIN failed--compilation aborted at - line 1.
 EOS
               "attribute on our hash with sub of same name");
 
+fresh_perl_is('$_ = ""; s/^/ { my $x : shared = 1; } /e;', "", {},
+              "attributes in sub-parse");
+
 done_testing();
diff --git a/toke.c b/toke.c
index e1b163334f..0b3b76fee3 100644
--- a/toke.c
+++ b/toke.c
@@ -5991,7 +5991,7 @@ Perl_yylex(pTHX)
 
 	switch (PL_expect) {
 	case XOPERATOR:
-	    if (!PL_in_my || PL_lex_state != LEX_NORMAL)
+	    if (!PL_in_my || (PL_lex_state != LEX_NORMAL && !PL_lex_brackets))
 		break;
 	    PL_bufptr = s;	/* update in case we back off */
 	    if (*s == '=') {
-- 
2.11.0

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 11, 2019

From @tonycoz

On Tue, 28 May 2019 22​:08​:50 -0700, tonyc wrote​:

On Mon, 18 Feb 2019 20​:44​:26 -0800, tonyc wrote​:

On Fri, 15 Feb 2019 11​:37​:16 -0800, corion@​corion.net wrote​:

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the
code​:

Scalar found where operator expected at -e line 1, near "$fh $1"
(Missing operator before $1?)

This isn't a syntax error, but a warning (but still buggy).

The code inside the replacement part of s///e is handled with a
sublex, and the code that decides whether an operator or term is
expected after C<print $fh> is mostly skipped for a sublex.

The attached fixes that, doing the same checks for C<print $fh> in a
sublex within brackets as for a normal parse.

This fixes the same problem for C<< "... ${code} ..." >>.

Rebased on blead, and fixes for the related issues.

I'll apply this in a few days.

Applied as eec8822 through 82de236.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 11, 2019

@tonycoz - Status changed from 'open' to 'pending release'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.