wrongly parsed double-quoted .lex names, .lex "foo\\o"

[rurban]

Needed by perl6: https://rt.perl.org/Public/Bug/Display.html?id=116643
Reportedly this worked in perl6-p before, but it could be a nqp change also, which put the statements into blocks, and thus the variables into lexicals.

The perl6 problem is with: .lex "&prefix:<\\o/>", $P103
and it works with .lex "&prefix:<\o/>", $P103 and .lex '&prefix:<\o/>', $P103.
Most pir strings are not properly unescaped, ' and " are treated just as ',
only when they are put into a register or with some other special cases (mk_sub_address_fromc)
the " is unescaped.
So the perl6 workaround is to put such lexicals in single-quotes until rurban/lexqnames-gh1095 has landed.

parrot:
In these roundtrips only the first 2 names survive:

.sub 'main' :main
    .lex 'bar\o', $P0
    $P1 = box 'ok 1'
    store_lex 'bar\o', $P1
    $P2 = find_lex 'bar\o'
    say $P2

    .lex "foo\\o", $P3       # imcc parses that as "foo\\\\o"
    $P1 = box 'ok 2'
    store_lex "foo\\o", $P1
    $P2 = find_lex "foo\\o"
    say $P2

    .lex "foo\o", $P4        # imcc parses that as "foo\\o"
    $P1 = box 'ok 3'
    store_lex "foo\o", $P1   # parrot str_unescape compresses that to "fooo"
    $P2 = find_lex "foo\o"
    say $P2
.end

=>

ok 1
ok 2
Lexical 'fooo' not found

For globals it is simplier, but the problem of ignoring Illegal escape sequences remains:

    $S1 = "foo\\o"
    $P1 = box 'ok 2'
    set_global "foo\\o", $P1   # ok, parsed as "foo\\o"
    $P2 = get_global "foo\\o"
    say $P2

    $S2 = "foo\o"
    $P1 = box 'ok 3'
    $S3 = "fooo"
    $P2 = box 'ok 4'
    set_global "foo\o", $P1    # wrong, compressed to "fooo" (Ignored Illegal escape sequence \o)
    $P3 = get_global "foo\o"
    say $P3

    $P3 = get_global "fooo"   # accesses $P1, not $P2
    say $P3

[rurban]

imcc: set_lexical() gets those wrongly quoted names from the parser

 .lex 'bar\o', $P0 => "bar\\o" # correct
 .lex "foo\\o", $P0 => "foo\\\\o" # wrong
 .lex "foo\o", $P0 => "foo\\o" # wrong

and this is also wrongly parsed:

store_lex "foo\o", P1 => "fooo" #wrong

[rurban]

There's more nonsense going on:

• A parsed name (a STRINGC) can be single or double-quoted, but both are treated as single-quoted (unescaped). But not when it's a
  • HLL: only " + unescaped
  • only mk_sub_address_fromc (sub, outer, sub_label_op_c) is correct, i.e. checks for " or ' and unescapes only "
  • loadlib only accepts " and unescapes
  • pmc_const accepts both, strips and treats both as unescaped (mk_pmc_const_named)
• all other names are stored as constant with the surrounding quotes. I.e. you can specify 'str' or "str" but both are different strings then.
• unknown escape sequences are ignored, e.g. "\o" will be "o", " will be " and more. I changed this with 940ede1 to die on unknown escape sequences for A-Za-z, but silently skip the \ with other chars, as before. To allow `", \``` and more. See GH Throw throw_illegal_escape, do not skip them #1103
• empty strings cannot occur in imcc, as they are either "" or ''

The encoding and quoting is later resolved in IMCC_string_from__STRINGC and IMCC_string_from_reg with proper unescaping, but the constants are at this time already interned in the symbol table.

e.g. annotated t/compilers/imcc/syn/clash_15.pir (added some tracings):

#    mk_const ''bar\o''
#    mk_const ''ok 1''
#    mk_const ''bar\o''
#    mk_const '"foo\\o"'
#    mk_const ''ok 2''
#    mk_const '"foo\\o"'
#    mk_const '"foo\\o"'
#    mk_const '"foo\o"'
#    mk_const ''ok 3''
#    mk_const '"fooo"'
#    mk_const ''ok 4''
#    mk_const '"foo\o"'
#    mk_const '"foo\o"'
#    mk_const '"fooo"'
#    mk_const '"()"'
0000 set S0, "bar\\o"
0003 box P0, "ok 1"
0006 set_global S0, P0
0009 get_global P1, "bar\\o"
000c say P1
ok 1
000e set S1, "foo\\o"
0011 box P0, "ok 2"
0014 set_global "foo\\o", P0
0017 get_global P1, "foo\\o"
001a say P1
ok 2
001c set S2, "fooo"
001f box P0, "ok 3"
0022 set S3, "fooo"
0025 box P1, "ok 4"
0028 set_global "fooo", P0
002b get_global P2, "fooo"
002e say P2
ok 3
0030 get_global P2, "fooo"
0033 say P2
ok 3
0035 set_returns PC0

and the same for the .lex testcase clash_14.pir:

#    mk_const 'bar\o'
#    .lex 'bar\o'
#    mk_const ''ok 1''
#    mk_const ''bar\o''
#    mk_const ''bar\o''
#    mk_const 'foo\\o'
#    .lex 'foo\\o'
#    mk_const ''ok 2''
#    mk_const '"foo\\o"'
#    mk_const '"foo\\o"'
#    mk_const 'foo\o'
#    .lex 'foo\o'
#    mk_const ''ok 3''
#    mk_const '"foo\o"'
#    mk_const '"foo\o"'
#    mk_const '"()"'
0000 box P1, "ok 1"
0003 store_lex "bar\\o", P1
#    store_lex_sc_p 'bar\o'
0006 find_lex P2, "bar\\o"
#    lexpad.get 'bar\o'
#    find_lex_p_sc 'bar\o'
0009 say P2              
ok 1
000b box P1, "ok 2"      
000e store_lex "foo\\o", P1
#    store_lex_sc_p 'foo\o'
0011 find_lex P2, "foo\\o" 
#    lexpad.get 'foo\o'
#    find_lex_p_sc 'foo\o'
0014 say P2                
ok 2
0016 box P1, "ok 3"        
0019 store_lex "fooo", P1  
#    store_lex_sc_p 'fooo'
Lexical 'fooo' not found

[rurban]

Clarification, as the fix will need some time: The workaround for perl6 and other HLLs is to use single-quotes for .lex names. Single quotes are preferred, only some special ops require double-quotes: HLL and loadlib.

[rurban]

Merged a documentation warning into 6.10.0:

Limitation: For now use only single-quotes for lexical variable
names! Double-quoted lexical names in .lex are treated as already
escaped, i.e. only as single-quoted and are not unescaped.

[rurban]

Took the easy road, and just unescaped double-quoted .lex string constants for now in smoke-me/lexqnames-gh1095

[rurban]

smoked fine on nqp and perl6 also.