Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

\" escape does not work with string interpolation #6476

Closed
scabug opened this issue Oct 4, 2012 · 48 comments · Fixed by scala/scala#8830
Closed

\" escape does not work with string interpolation #6476

scabug opened this issue Oct 4, 2012 · 48 comments · Fixed by scala/scala#8830
Assignees
Labels
Milestone

Comments

@scabug
Copy link

scabug commented Oct 4, 2012

scala> val a="42"
a: String = 42

scala> "a=\"$a\""
res33: String = a="$a"    // ok

scala> s"a=\"$a\""
<console>:1: error: ';' expected but string literal found.
       s"a=\"$a\""
                ^         // oops

scala> s"a=\'$a\'"
res34: String = a='42'    // ok with \'

scala> """a=\"$a\""""
res35: String = a=\"$a\"  // escape not interpreted, ok

scala> s"""a=\"$a\""""
res36: String = a="42"    // escape interpreted, should it ?
@scabug
Copy link
Author

scabug commented Oct 4, 2012

Imported From: https://issues.scala-lang.org/browse/SI-6476?orig=1
Reporter: Denis Petrov (denis)
Affected Versions: 2.10.0-M6

@scabug
Copy link
Author

scabug commented Oct 4, 2012

@paulp said:
You can see this being declared wontfix here: https://groups.google.com/d/topic/scala-sips/d2K23f__6b0/discussion

@scabug
Copy link
Author

scabug commented Oct 4, 2012

Denis Petrov (denis) said (edited on Oct 4, 2012 3:24:35 PM UTC):
That's bad.

Without interpolation, single quote literals could represent any string using escapes (triple-quoted cannot represent """)
With interpolation enabled, there are still literals which cannot be represented using both single- and triple- quoted sequences.

Thus, a code generator (i.e. template engine) cannot use triple quoted in the emitted code, as literals are not known in advance.
It can use single quotes, though, and escape any literal into them.

But as result of this escaping rules, a code generator cannot use string interpolation at all.

@scabug
Copy link
Author

scabug commented Oct 10, 2012

Denis Petrov (denis) said:
As '$' is the escape symbol in interpolated strings (think s"$$" == "$"), it may have sense to escape '"' also via $-prefixed escape sequence (say s"$"" == """).

@scabug
Copy link
Author

scabug commented Dec 12, 2012

@retronym said:
Closing as "Won't fix" as the behaviour is in accordance with the spec.

I know it's a little clunky, but your code generator could create:

scala> s"${'"'}"
res1: String = "

@scabug
Copy link
Author

scabug commented Dec 18, 2012

Denis Petrov (denis) said:
Hopefully, s"\042" works (but s"\u0022" does not).

@scabug
Copy link
Author

scabug commented Aug 8, 2013

Radim Kolar (hsn) said:
Its realistic to get spec updated and this issue fixed?

@scabug
Copy link
Author

scabug commented Sep 6, 2013

@jrudolph said:
I'll try to summarize the case in other words: Scala doesn't want to make any choices about escaping in interpolated strings. Especially, backslash is should be freed as escaping character on the scala grammar level. This means that \" can't work because it would mean that backslash itself would need to be escaped.

I agree to what Denis said: as there is a new escape character $ in interpolated strings, it would make sense to escape double-quotes with $". Any counter-arguments?

@scabug
Copy link
Author

scabug commented Sep 6, 2013

@retronym said:
+1 to s"$"". Because it doesn't compile today, we don't risk changing the meaning of existing programs.

@scabug
Copy link
Author

scabug commented Sep 6, 2013

@densh said:
Non-working escaping bugged me a bit before I got used to """. I guess this is quite logical to let people escape quotes in their string interpolators. e.g.

q"method.call($"mystring in a quasiquote$")"

@scabug
Copy link
Author

scabug commented Sep 19, 2013

@retronym said (edited on Sep 19, 2013 4:03:44 PM UTC):
WIP:

Update Scala parser:

retronym/scala@scala:master...ticket/6476

Update IntelliJ parser:

https://github.com/retronym/intellij-scala/compare/topic/SI-6476

Starting a [scala-internals] discussion: https://groups.google.com/d/msg/scala-internals/_DnZGO-4NXc/xNoiOUoCf58J

@scabug
Copy link
Author

scabug commented Feb 28, 2014

Jason Swartz (swartzrock) said:
Re comments from Martin and others here make it appear that string interpolation should not interpret the backspace:
https://groups.google.com/forum/#!topic/scala-sips/d2K23f__6b0/discussion

A triple-quoted string interpolation should preserve the backspace.
s"""a="$a""""
expected result: a="42"
actual resultl: a="42"

@scabug
Copy link
Author

scabug commented Feb 28, 2014

@som-snytt said:
The language is that the parser preserves the escapes in an interpolated string, so that the interpolator can do whatever it wants.

Clarifying the previous comment:

scala> implicit class `literally the string`(val sc: StringContext) {
     | def lit(args: Any*): String = {
     |   sc.checkLengths(args)
     |   sc.standardInterpolator(identity, args)
     | }}
defined class literally$u0020the$u0020string

scala> val a = "42" ; s"""a=\"$a\";"""
a: String = 42
res4: String = a="42";

scala> val a = "42" ; lit"""a=\"$a\";"""
a: String = 42
res5: String = a=\"42\";

Clarifying a previous suggestion to use octal escapes, they are deprecated somewhat:

scala> implicit class `strictly ess`(val sc: StringContext) {
     | def S(args: Any*): String = {
     |   sc.checkLengths(args)
     |   sc.standardInterpolator(StringContext.processEscapes, args)
     | }}
defined class strictly$u0020ess

scala> val a = "42" ; s"\042$a\042"
a: String = 42
res0: String = "42"

scala> val a = "42" ; S"\042$a\042"
scala.StringContext$InvalidEscapeException: invalid escape character at index 0 in "\042"
  at scala.StringContext$.treatEscapes0(StringContext.scala:208)
  at scala.StringContext$.processEscapes(StringContext.scala:189)
  at strictly$u0020ess$$anonfun$S$1.apply(<console>:10)
  at strictly$u0020ess$$anonfun$S$1.apply(<console>:10)
  at scala.StringContext.standardInterpolator(StringContext.scala:121)
  at strictly$u0020ess.S(<console>:10)
  ... 32 elided

scala> :se +deprecation

scala> val a = "42" ; f"\042$a\042"
<console>:11: warning: Octal escape literals are deprecated, use \u0022 instead.

And of course as noted previously, that advice is only good for triple-quotes. There's a comment in the code to upgrade the advice to dollar-quote when that is merged someday.

@scabug
Copy link
Author

scabug commented Apr 30, 2014

@som-snytt said (edited on Jul 9, 2014 5:12:18 AM UTC):
I got antsy, so @retronym's patch is on scala/scala#3870, which is under the aegis of SIP-24.

https://groups.google.com/d/msg/scala-sips/rem6D3XTkPY/08v2X0ZpmgsJ

The literal quote is what @retronym would call the least controversial bit.

@scabug
Copy link
Author

scabug commented Jun 6, 2015

@som-snytt said:
I defected and proposed

scala/scala#4308

and I never understood the confusion over this topic.

I find the arguments against the interpolator receiving the raw text unconvincing. (That escapes must be processed at one layer as meta-escapes and so on.)

Everyone wants to say s"\"hello,\" world".

I think the compiler should satisfy that customer.

@scabug
Copy link
Author

scabug commented Jul 6, 2015

Kevin Lee (kevinlee) said:
I'm confused about s""" """ case. So is s"""a\nb""" becoming

a
b

a bug or not?

@scabug
Copy link
Author

scabug commented Jul 6, 2015

@som-snytt said:
The doc for the s-interpolator is clear about handling standard escapes, which has nothing to do with this issue.

http://www.scala-lang.org/api/2.11.7/index.html#scala.StringContext@s%28args:Any*%29:String

@scabug
Copy link
Author

scabug commented Jul 6, 2015

Kevin Lee (kevinlee) said:
Thank you. Got it now.

@scabug
Copy link
Author

scabug commented Aug 6, 2015

Isiah Meadows (impinball) said:
Progress? This still currently fails in 2.11.5 AFAICT (through IntelliJ, but the plugin's own parser correctly parses this, so it's likely not an IDE bug):

s"\"wrapped\""

@scabug
Copy link
Author

scabug commented Aug 7, 2015

@som-snytt said:
This change is perceived to harbor complexity, so it has been kicked back to the SIP process. It is unlikely to progress until next year's vacations and exam periods are over.

@scabug
Copy link
Author

scabug commented Oct 30, 2015

@som-snytt said:
Not likely to progress, period.

@scabug
Copy link
Author

scabug commented Jan 5, 2016

@SethTisue said:
discussion on a possible SLIP on this: scala/slip#10

@scabug
Copy link
Author

scabug commented Jan 5, 2016

@som-snytt said:
Episode 8 is tentatively titled, "Revenge of the SIP".

@scabug
Copy link
Author

scabug commented Jun 6, 2016

@som-snytt said:
They still leave hate comments on the doc page http://docs.scala-lang.org/overviews/core/string-interpolation.html

@scabug
Copy link
Author

scabug commented Aug 11, 2016

@som-snytt said:
They still backlog this.

@scabug
Copy link
Author

scabug commented Aug 11, 2016

Denis Petrov (denis) said:
Another inconsistency with triple quotes (which perhaps deserves its own ticket as not related to the interpolation, or it is just a thing to consider if one day the issue will be addressed). The same way as I have to write \042 instead of ", I also sometimes have to write \u005C instead of \ because there is no other way to encode \u (two chars, '\' then 'u') in triple-quote string literal:

scala> "c:\\user"
res0: String = c:\user

scala> """c:\\user"""
res1: String = c:\\user

scala> """c:\user"""
<console>:1: error: error in unicode escape
       """c:\user"""
              ^

scala> """c:\u005Cuser"""
res2: String = c:\user

@scabug
Copy link
Author

scabug commented Aug 11, 2016

@som-snytt said:
Workaround, where the escape is processed by the macro:

scala> f"""c:\\user"""
res3: String = c:\user

@scabug
Copy link
Author

scabug commented Mar 15, 2017

Daniel Gordon (Dgordon) said:
I'm in the middle of writing my own mini regex library, and I stumbled upon this issue. Here is the basic code that does not compile:

val clientRegex= "search for this \d"
val regexString = s""".*+[2-3]$clientVal""" // only clientVal is relevant here
    // returns 'invalid escape - use '\\'

So in my code - no problem, I can use two backslashes (even though that defeats the whole purpose of triple quotes). However, code like this will be in a method that I expose for clients to use.

Now if a client decides to use my method, and passes some regex string that contains any escape like \d - then I'll fail. So my only workaround here is to use the old "some string" + someVal + "more string" syntax - which I hate.

Or I'll have to parse the user regex and if there is any backslashes in it, then create a string using two backslashes instead.

I've had this come up many times when using Regex in scala. I'm surprised that something like this is sitting on the backlog.

@scabug
Copy link
Author

scabug commented Mar 15, 2017

@som-snytt said:
There's a tweak to the docs to recommend raw"\d".r.

But ultimately, stringly composition is a losing battle.

I think there's probably a misunderstanding in Daniel's example? Interpolated text is never interpreted or escaped. There's no danger that the following might interpolate a newline. I didn't count the backslashes.

scala> val x = raw"\\\\\\\"
x: String = \\\\\\\

scala> s"${x}n"
res1: String = \\\\\\\n

@scabug scabug added this to the Backlog milestone Apr 7, 2017
@eed3si9n
Copy link
Member

eed3si9n commented Mar 26, 2020

Actually the current 1.3.5 as well as the original Google Doc says:

interpolatedString ::= alphaid ‘"’ {printableChar \ (‘"’ | ‘$’) | escape} ‘"’ 
                         |  alphaid ‘"""’ {[‘"’] [‘"’] char \ (‘"’ | ‘$’) | escape} {‘"’} ‘"""’

Maybe it's missing some vertical bars, but should I read that:

  • standard interpolated string permits printableChar, \", \$, or escape sequence, and
  • multiline interpolated string permits "", char, \$, or escape sequence

in the scanner?

@som-snytt
Copy link

som-snytt commented Mar 26, 2020

Not being able to write s"\"hello\", $person" is just bad.

That is a matter of opinion or taste.

If a default interpolator were supported for ordinary string syntax, then fixing it would not be optional.

Recall that the initial judgment was that interpolation wasn't needed because you can write

"\"hello\", "+ person

where the lack of a space between the quote and operator makes its visually parseable. So it's a matter of opinion or taste.

I don't think there was a great deal of incentive to make interpolation especially nice.

@som-snytt
Copy link

som-snytt commented Mar 26, 2020

For a bug, the spec specifically disallows triple single quote for a char: "The character can be any Unicode character except the single quote delimiter", but accepted it anyway until it was fixed.

I don't think appeal to the spec means anything except in so far as the spec converges upon a desired description, and the implementation either follows or leads.

eed3si9n added a commit to eed3si9n/scala that referenced this issue Mar 26, 2020
Fixes scala/bug#6476

SIP-11 specification says

```
interpolatedString ::= alphaid `"` {printableChar \ (`"` | `$`) | escape} `"`
```

My reading of this is that the scanner should support the character sequence `\"`, but pass them as is. The Standard (s) interpolator would take care of the evaluation of `\"` as `"`.
This would fix the long standing issue of people trying to write `s"a=\"$a\""` and running into mysterious `';' expected but string literal found.`.

This change should be relatively safe since neither `s"\"foo\""` nor `s"foo\"` previously compiled.
@SethTisue
Copy link
Member

SethTisue commented Mar 26, 2020

I don't see any possibility of changing this in Scala 2, for compatibility reasons. Given the necessity of supporting cross-compilation between Scala 2.13 and 3, it seems to me that the next change opportunity is Scala 3.1.

Under the circumstances, I think we should just close the ticket, as this is working-as-designed in Scala 2.

For anyone seriously interested in changing this, in whatever Scala version — it isn't enough to just wish — or even to prove! — that a different decision had been made initially. A lot of people wish that. To actually change it now, the full implications, the spec changes, the implementation changes, and the social aspects of getting the change through all need to be worked through.

@eed3si9n
Copy link
Member

eed3si9n commented Mar 26, 2020

Here's my PR - scala/scala#8830. No spec change, and hopefully no change in existing-and-working code.

@OlegYch
Copy link

OlegYch commented Mar 26, 2020

@SethTisue we can always make it not compile in a minor version without any serious compatibility implications

@SethTisue
Copy link
Member

SethTisue commented Mar 26, 2020

I would be delighted to be forced to admit that I was wrong (about the chances of this moving forward in Scala 2).

As I just commented on Eugene's PR,

I think we need to actually know what the safety level is. Either it's absolutely safe, or there are unsafe cases that need to be precisely characterized in order to know whether we're comfortable with advancing this.

@OlegYch
Copy link

OlegYch commented Mar 26, 2020

specifically i'm referring to s"""a=\"$a\""""
this is obviously a bug in current scala and should not compile if we can't fix it

eed3si9n added a commit to eed3si9n/scala that referenced this issue Mar 27, 2020
Fixes scala/bug#6476

SIP-11 specification says

```
interpolatedString ::= alphaid `"` {printableChar \ (`"` | `$`) | escape} `"`
```

My reading of this is that the scanner should support the character sequence `\"`, but pass them as is. The Standard (s) interpolator would take care of the evaluation of `\"` as `"`.
This would fix the long standing issue of people trying to write `s"a=\"$a\""` and running into mysterious `';' expected but string literal found.`.

This change should be relatively safe since neither `s"\"foo\""` nor `s"foo\"` previously compiled.
@eed3si9n
Copy link
Member

eed3si9n commented Mar 27, 2020

@OlegYch It's surprising for sure, but I think s"""a=\"$a\"""" works as designed. As a capturing mechanism all String interpolators act as triple quoted String literal, and in case of s-interpolator the implementation is doing the backslash escaping. Therefore s"""a=\"$a\"""" should evaluate as a="42".

Perhaps one way of looking at it is that s-interpolator emulates standard String literal "abc" and raw-interpolator emulates the multiline String literal """abc""".

@OlegYch
Copy link

OlegYch commented Mar 27, 2020

if it's surprising then it should not compile
"""a=\"qwe\"""" and s"""a=\"qwe\"""" should produce equal strings
i'd suggest disallowing escape sequences in raw interpolated strings if there is no way to make it so

@martinKindall
Copy link

martinKindall commented Jul 10, 2020

Using Scala 2.12 and got this error, this issues explains it nicely. Thanks.

eed3si9n added a commit to eed3si9n/scala that referenced this issue Mar 7, 2021
Fixes scala/bug#6476

SIP-11 specification says

```
interpolatedString ::= alphaid `"` {printableChar \ (`"` | `$`) | escape} `"`
```

My reading of this is that the scanner should support the character sequence `\"`, but pass them as is. The Standard (s) interpolator would take care of the evaluation of `\"` as `"`.
This would fix the long standing issue of people trying to write `s"a=\"$a\""` and running into mysterious `';' expected but string literal found.`.

This change should be relatively safe since neither `s"\"foo\""` nor `s"foo\"` previously compiled.
lrytz pushed a commit to eed3si9n/scala that referenced this issue Mar 8, 2021
Fixes scala/bug#6476

SIP-11 specification says

```
interpolatedString ::= alphaid `"` {printableChar \ (`"` | `$`) | escape} `"`
```

My reading of this is that the scanner should support the character sequence `\"`, but pass them as is. The Standard (s) interpolator would take care of the evaluation of `\"` as `"`.
This would fix the long standing issue of people trying to write `s"a=\"$a\""` and running into mysterious `';' expected but string literal found.`.

This change should be relatively safe since neither `s"\"foo\""` nor `s"foo\"` previously compiled.
lrytz pushed a commit to eed3si9n/scala that referenced this issue Mar 15, 2021
Fixes scala/bug#6476

SIP-11 specification says

```
interpolatedString ::= alphaid `"` {printableChar \ (`"` | `$`) | escape} `"`
```

My reading of this is that the scanner should support the character sequence `\"`, but pass them as is. The Standard (s) interpolator would take care of the evaluation of `\"` as `"`.
This would fix the long standing issue of people trying to write `s"a=\"$a\""` and running into mysterious `';' expected but string literal found.`.

This change should be relatively safe since neither `s"\"foo\""` nor `s"foo\"` previously compiled.
@dwijnand dwijnand modified the milestones: Backlog, 2.13.6 Mar 31, 2021
@SethTisue
Copy link
Member

SethTisue commented Mar 31, 2021

@SethTisue wrote:

I don't see any possibility of changing this in Scala 2

It seems I will be having my own hat for lunch today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.