Error producing ^\ (chr 28) with "\c\\"

[p5pRT]

Migrated from rt.perl.org#1806 (status was 'resolved')

Searchable as RT1806$

[p5pRT]

From newton@ficus.frogspace.net

It appears to be impossible to produce a ^\ character (ASCII 28)
using \c notation. "\x1c" and "\034" work fine, but "\c\\" gives
a two-character string (ASCII 28, 92 i.e. ^\ followed by backslash)
and "\c\" gives "Can't find string terminator '"' anywhere before
EOF".

Examples​:

$ perl -wle 'print join ", ", map ord, split //, "\c\"'
Can't find string terminator '"' anywhere before EOF at -e line 1.
$ perl -wle 'print join ", ", map ord, split //, "\c\\"'
28, 92

"\c\\" whould be the correct syntax in my opinion; however, it appears
that the \c logic sees two backslashes -- that the two backslashes
aren't first reduced to one (as per double-quoting usually) before \c
sees it.

Perl Info

Site configuration information for perl 5.00503:

Configured by frogleg at Sun Aug  8 13:32:51 EDT 1999.

Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration:
  Platform:
    osname=linux, osvers=5.2, archname=i686-linux
    uname='linux ficus.frogspace.net 2.2.6-ac3 #2 thu aug 5 09:35:04 edt 1999 i686 unknown '
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef useperlio=undef d_sfio=undef
  Compiler:
    cc='gcc', optimize='-O2', gccversion=2.7.2.3
    cppflags='-Dbool=char -DHAS_BOOL -I/usr/local/include'
    ccflags ='-Dbool=char -DHAS_BOOL -I/usr/local/include'
    stdchar='char', d_stdstdio=define, usevfork=false
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    alignbytes=4, usemymalloc=n, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lc -lposix -lcrypt
    libc=, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.00503:
    /usr/local/lib/perl5/5.00503/i686-linux
    /usr/local/lib/perl5/5.00503
    /usr/local/lib/perl5/site_perl/5.005/i686-linux
    /usr/local/lib/perl5/site_perl/5.005
    .


Environment for perl 5.00503:
    HOME=/home/newton
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/sbin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From [Unknown Contact. See original ticket]

,,, writes​:

"\c\\" whould be the correct syntax in my opinion; however, it appears
that the \c logic sees two backslashes -- that the two backslashes
aren't first reduced to one (as per double-quoting usually) before \c
sees it.

And what would "\c\c\c\c\\" do, in your opinion? ;-)

perlop/"Gory details..."

Ilya

[p5pRT]

From [Unknown Contact. See original ticket]

(cc'ed to Ilya Zakharevich)

On Fri, 19 Nov 1999, Ilya Zakharevich wrote​:

,,, writes​:

"\c\\" whould be the correct syntax in my opinion; however, it appears
that the \c logic sees two backslashes -- that the two backslashes
aren't first reduced to one (as per double-quoting usually) before \c
sees it.

And what would "\c\c\c\c\\" do, in your opinion? ;-)

My guess is ^\ + 'c' + ^\ + 'c' + '\\', i.e. ASCII 28, 99, 28, 99, 92. And
that's what Perl does.

This still doesn't explain to me, though, hough to produce a string
consisting solely of the character ^\.
--
Philip Newton <newton@​newton.digitalspace.net>

[p5pRT]

From [Unknown Contact. See original ticket]

On Fri, Nov 19, 1999 at 11​:17​:03AM -0500, Philip Newton wrote​:

And what would "\c\c\c\c\\" do, in your opinion? ;-)

My guess is ^\ + 'c' + ^\ + 'c' + '\\', i.e. ASCII 28, 99, 28, 99, 92. And
that's what Perl does.

So you want it to be parsed kleft-to-right. But you want \c\\ to be
parsed right-to-left. Choose one.

This still doesn't explain to me, though, hough to produce a string
consisting solely of the character ^\.

TIMTOWTDI. chr(ord('\\')-62) (or is it 32?) is one of them.

Ilya

[p5pRT]

From @ysth

In article <19991119124913.F20768@​monk.mps.ohio-state.edu>,
Ilya Zakharevich <ilya@​math.ohio-state.edu> wrote​:

On Fri, Nov 19, 1999 at 11​:17​:03AM -0500, Philip Newton wrote​:

And what would "\c\c\c\c\\" do, in your opinion? ;-)

My guess is ^\ + 'c' + ^\ + 'c' + '\\', i.e. ASCII 28, 99, 28, 99, 92. And
that's what Perl does.

So you want it to be parsed kleft-to-right. But you want \c\\ to be
parsed right-to-left. Choose one.

He stated he expected either "\c\\" or "\c\" to produce chr(28) and
found that neither of them did. He then said that in his opinion
"\c\\" should do it. I infer that what he *wants* is for either of
them to do it.

[D​:\susv2]perl -wlne "eval $_; print $@​ if $@​"
print length "\c\\"
2
print length "\c\"
Can't find string terminator '"' anywhere before EOF at (eval 2) line 1, <> chunk 2.
exit

I lean toward considering the second of these a bug.

[p5pRT]

From [Unknown Contact. See original ticket]

Yitzchak Scott-Thoennes writes​:

[D​:\susv2]perl -wlne "eval $_; print $@​ if $@​"
print length "\c\\"
2
print length "\c\"
Can't find string terminator '"' anywhere before EOF at (eval 2) line 1, <> chunk 2.
exit

I lean toward considering the second of these a bug.

Then read the docs.

Ilya

[p5pRT]

From @ysth

In article <199911210533.AAA01763@​monk.mps.ohio-state.edu>,
Ilya Zakharevich <ilya@​math.ohio-state.edu> wrote​:

Yitzchak Scott-Thoennes writes​:

[D​:\susv2]perl -wlne "eval $_; print $@​ if $@​"
print length "\c\\"
2
print length "\c\"
Can't find string terminator '"' anywhere before EOF at (eval 2) line 1, <> chunk 2.
exit

I lean toward considering the second of these a bug.

Then read the docs.

Thank you, I already did when you gave the doc reference before.

I realise that this behavior is perfectly in accord with perlop/"Gory
details"/"Finding the end". That doesn't mean it's not a bug. It
just means that if it is a code bug it is also a doc bug.

IMO, it also contradicts the beginning of perlop/"Gory details"​:

  When presented with something which may have several
  different interpretations\, Perl uses the principle DWIM
  \(expanded to Do What I Mean \- not what I wrote\) to pick up
  the most probable interpretation of the source\.

A​: "\c\"
B​: "\c\"more string here"

The question is, is the current incomprehesible behavior of string B
more important to maintain than following DWIM for "string" A?

[p5pRT]

From [Unknown Contact. See original ticket]

Yitzchak Scott-Thoennes writes​:

I realise that this behavior is perfectly in accord with perlop/"Gory
details"/"Finding the end". That doesn't mean it's not a bug. It
just means that if it is a code bug it is also a doc bug.

IMO, it also contradicts the beginning of perlop/"Gory details"​:

  When presented with something which may have several
  different interpretations\, Perl uses the principle DWIM
  \(expanded to Do What I Mean \- not what I wrote\) to pick up
  the most probable interpretation of the source\.

This is taken out of context.

A​: "\c\"
B​: "\c\"more string here"

The question is, is the current incomprehesible behavior of string B
more important to maintain than following DWIM for "string" A?

Perl uses a very simple rule to find an end of a quoted construct.
All one needs to do is to learn it.

Hope this help,
Ilya

[p5pRT]

From @ysth

Cc'd to​: ilya@​math.ohio-state.edu

In article <199911210756.CAA02439@​monk.mps.ohio-state.edu>,
Ilya Zakharevich <ilya@​math.ohio-state.edu> wrote​:

Yitzchak Scott-Thoennes writes​:

I realise that this behavior is perfectly in accord with perlop/"Gory
details"/"Finding the end". That doesn't mean it's not a bug. It
just means that if it is a code bug it is also a doc bug.

IMO, it also contradicts the beginning of perlop/"Gory details"​:

  When presented with something which may have several
  different interpretations\, Perl uses the principle DWIM
  \(expanded to Do What I Mean \- not what I wrote\) to pick up
  the most probable interpretation of the source\.

This is taken out of context.

I disagree. This reads to me like a "mission statement" for Perl
parsing. If one of the many detailed parsing rules that follow
violate this statement, then that is an indication that a modification
may be necessary.

A​: "\c\"
B​: "\c\"more string here"

The question is, is the current incomprehesible behavior of string B
more important to maintain than following DWIM for "string" A?

Perl uses a very simple rule to find an end of a quoted construct.
All one needs to do is to learn it.

Hope this help,
Ilya

Thanks, I thought I already said I understand the docs and behavior
for finding the end of a quoted construct are in agreement.

But application of these rules of finding the end, left-to-right
parse, etc. leaves the \c\ construct without a clear meaning.

One of the following should be true in all cases​:
1. \c\ produces a fatal exception
2. the second \ is a escape character and the following character is
  the 'argument' to \c
3. the \ is the 'argument' to \c and the following character is unaffected
4. the behavior is undefined and a warning is given

Currently none of these is true.
The behavior is pretty close to number 3 except for a couple of odd cases​:

[D​:\]perl -wlne "print map {length,'​:',map {;' ',ord} split //} eval"
"\c\\"
2​: 28 92
"\c\""
1​: 98
exit

In the first case, the string seems to contain two characters​: \c\ and \.
The anomaly is that the trailing backslash doesn't escape the ".

In the second case, the string should IMO end with the 2nd ". Instead,
the 2nd \ of \c\ escapes it, contrary to \c\ behavior elsewhere. Thus
the string becomes \c" -> chr((ord('"')-64) & 127) -> b.

[p5pRT]

From [Unknown Contact. See original ticket]

On Sun, Nov 21, 1999 at 10​:04​:34AM -0800, Yitzchak Scott-Thoennes wrote​:

But application of these rules of finding the end, left-to-right
parse, etc. leaves the \c\ construct without a clear meaning.

On the opposite, they *give* \c\ a clear meaning​: see qq[\c\c].

Ilya

[p5pRT]

From [Unknown Contact. See original ticket]

Yitzchak Scott-Thoennes <sthoenna@​efn.org> writes​:

A​: "\c\"
B​: "\c\"more string here"

The question is, is the current incomprehesible behavior of string B
more important to maintain than following DWIM for "string" A?

Yes.
print "this is \"important\"!\n";

--
Nick Ing-Simmons

[p5pRT]

From [Unknown Contact. See original ticket]

On Fri, 19 Nov 1999, Ilya Zakharevich wrote​:

On Fri, Nov 19, 1999 at 11​:17​:03AM -0500, Philip Newton wrote​:

And what would "\c\c\c\c\\" do, in your opinion? ;-)

My guess is ^\ + 'c' + ^\ + 'c' + '\\', i.e. ASCII 28, 99, 28, 99, 92. And
that's what Perl does.

So you want it to be parsed left-to-right. But you want \c\\ to be
parsed right-to-left. Choose one.

Not quite. I want '\\' to be translated to \ in a previous pass, before
the \c mechanism sees it. After the '\\' -> \ pass, I want left-to-right.

This still doesn't explain to me, though, hough to produce a string
consisting solely of the character ^\.

TIMTOWTDI. chr(ord('\\')-62) (or is it 32?) is one of them.

Well, and "\0x1c" and "\034" work, of course. I was just disappointed that
ctrl-\ seems to be the only character that's difficult to produce in \c
notation.

Cheers,
Philip

[p5pRT]

From [Unknown Contact. See original ticket]

On Tue, Nov 23, 1999 at 05​:00​:24AM -0500, Philip Newton wrote​:

So you want it to be parsed left-to-right. But you want \c\\ to be
parsed right-to-left. Choose one.

Not quite. I want '\\' to be translated to \ in a previous pass, before
the \c mechanism sees it. After the '\\' -> \ pass, I want left-to-right.

Then you are advising that "\\nc" and "\cc" should be parsed the same, right?

Ilya

[p5pRT]

From @TimToady

Philip Newton writes​:
: Not quite. I want '\\' to be translated to \ in a previous pass, before
: the \c mechanism sees it. After the '\\' -> \ pass, I want left-to-right.

It would have to be done in the same pass, by pretending that \c is a
funny kind of \. We go to great lengths to avoid doing multiple passes
in Perl, and when we do do multiple passes, we go to great lengths to
hide that fact. For instance, we pretend that regular expressions
are interpolated and interpreted just like double-quoted strings, but
in fact, the lexer must treat them entirely differently to preserve
that illusion, because the regular expression parser does a separate
pass after interpolation. Not only must the lexer pass backslashed
sequences through to the regular expression parser, but it has to decide
which dollar signs indicate something to be interpolated immediately​:

  /$foo/

and which have to be passed through to the regular expression engine​:

  /foo$/
  /(foo$|bar$)/
  /(?{ $foo += 1 })/

Actually, that last one doesn't need to pass $foo--it could conceivably
just pass a pointer to some precompiled code, but I don't think it
does. If I recall, it's more like an eval.

But anyway, we don't cavalierly add multiple passes to Perl. Multiple
passes tend to make things easier for the implementer, but harder for
the user. Perl's loyalties lie with the user.

Larry

[p5pRT]

From @ysth

Cc'd to​: larry@​wall.org

In article <199911231731.JAA16685@​kiev.wall.org>,
Larry Wall <larry@​wall.org> wrote​:

It would have to be done in the same pass, by pretending that \c is a
funny kind of \.

At last, someone with a glimmer of sense! \c *already* is a funny
kind of \. Except with respect to whichever character indicates the
end of the quoted string. This inconsistency is a *bug*.

For instance​:

$foo = 'bar'; print "\c$foo"; yields 'dfoo', not an error
$foo = 'bar'; print "\c\$foo"; yields chr(28).'bar', not 'dfoo'

From this, "\c\" should yield chr(28).
And "\c"" should yield 'b', just as qq'\c"' does.

Alternatively, \c\ should always apply the \c 'operator' to the
following character, so that "\c\"" works like qq'\c"' (as it
currently does) but it takes \c\\ to get a chr(28).