Newlines in multiline strings are converted to \n

[scabug]

On Windows, where line.separator is \r\n, the scala compiler 2.8.0.Beta1 converts the platform newlines in multiline strings to \n.

This is wrong.

Steps to reproduce:
Create nl.scala with:

println("""
""".getBytes.length)

Make sure it has Windows line separator \r\n.

scala-2.8.0.Beta1-prerelease:

scala nl.scala
1

Expected results (scala 2.7.7):

scala nl.scala
2

I was also able to reproduce this also with scala-2.8.0-20100214.025735-327 (latest from maven snaphots repo).

Specs way to reproduce:

"""
""" must_== "\r\n"

Note you can force the windows line separators in file from SVN by setting the svn:eol-style property to CRLF.

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-3101?orig=1
Reporter: Pavol Vaskovic (pali-at-pali.sk)
Attachments:

• nl.patch (created on Feb 25, 2010 9:20:55 AM UTC, 1684 bytes)

[scabug]

@jrudolph said:
Could you get the output of the file in binary/hex form? What are the line terminator bytes in nl.scala?

What is your expectation for the Scala compiler to do when it finds special characters inside a multiline string? IMO it should just treat them verbatim and apply no processing to them whatsoever.

[scabug]

Pavol Vaskovic (pali-at-pali.sk) said:
Replying to [comment:2 jrudolph]:

Could you get the output of the file in binary/hex form? What are the line terminator bytes in nl.scala?

As written in original report:

Make sure it has Windows line separator \r\n.

What is your expectation for the Scala compiler to do when it finds special characters inside a multiline string? IMO it should just treat them verbatim and apply no processing to them whatsoever.

Even though I am not sure what you mean by special characters, I would say the behavior should be same as in 2.7. Breaking changes in compiler need strong rationale, because they are very painful.

[scabug]

@jrudolph said:
Sorry, you are right. But this is no issue with Windows at all. This happens in linux exactly the same.

I'll restate the problem: In 2.8 the Scala compiler handles newlines in multiline strings specially: The character sequence "\r\n" is replaced by "\n". This changed from 2.7 which IMO showed the correct behaviour: Multiline strings could contain any characters and were converted verbatim on the character-level into a string constant.

The Scala langugage reference says:

Characters must not necessarily be printable; newlines or other control characters are also permitted. Unicode escapes work as everywhere else, but none of the escape sequences in (�1.3.6) is interpreted.

[scabug]

@jrudolph said:
A possible patch, completely untested

[scabug]

@odersky said:
(In r21017) closed #3101. Review by community.

[scabug]

Pavol Vaskovic (pali-at-pali.sk) said:
Verified with 2.8.0-20100303.025923-333.
Thanks Martin!

BTW, looking at the fix in r21017, I see no testcases committed. Are there any?

[scabug]

@SethTisue said:
This bit me today. On the one hand, I can see the argument for preserving the \r characters. But it's pretty annoying for the meaning of source code to change depending on what OS I was on when I checked the source out from version control.

[scabug]

@jrudolph said:
Yes, I agree. I didn't even remember that I took any stance in this issue in the past. We've been bitten by this issue as well, so that we had to introduce a special stripMarginWithNewline in tests to avoid failures: https://github.com/akka/akka/blob/release-2.3-dev/akka-http-core/src/test/scala/akka/http/impl/engine/server/HttpServerSpec.scala#L126

The problem here is that version control systems like git automatically adapt line endings of source files to the OS they run on when checking code out or in.

But maybe this kind of inadvertent "fuzzing" just demonstrates the real issue: that you need to be explicit about what kind of line endings a multiline string should produce in the first place. So, we actually like the stripMarginWithNewline("\r\n") as it makes it very clear what line ending in the resulting string should look like.