Wrong line delimiter when enclosing multi-line strings with double quotes.

[miklossy]

After checking out a fresh Xtend workspace on windows, there are several java files in xtend-gen folder marked as dirty:

As far as I can see, the reason for the change is that multi line Strings enclosed by double quotes are translated differently on windows and on mac.

Hint: by multi line String enclosed by ''' the newLine() method is used in the generated *.java file. I suggest to also use this approach for multi-line Strings enclosed by ".

[miklossy]

Small example to reproduce:
The xtend code

package foo

class Foo {
	val a = "
		line1
		line2
	"
	
	val b = '''
		line1
		line2
	'''
}

is compiled to the following java code:

package foo;

import org.eclipse.xtend2.lib.StringConcatenation;
import org.eclipse.xtext.xbase.lib.Functions.Function0;

@SuppressWarnings("all")
public class Foo {
  private final String a = "\r\n\t\tline1\r\n\t\tline2\r\n\t";
  
  private final String b = new Function0<String>() {
    @Override
    public String apply() {
      StringConcatenation _builder = new StringConcatenation();
      _builder.append("line1");
      _builder.newLine();
      _builder.append("line2");
      _builder.newLine();
      return _builder.toString();
    }
  }.apply();
}

[miklossy]

For RichStrings, the XtendCompiler._toJavaExpression(RichString richString, ITreeAppendable b) method is called, for String Literals the XtendCompiler._toJavaExpression(XStringLiteral expr, ITreeAppendable b) method is called. I think we should make a difference if the XStringLiteral is a multi-line string and handle such cases similar to RichStrings.

cc @cdietrich , @szarnekow @ArneDeutsch What do you think about this idea?

[szarnekow]

Instead of imposing runtime semantics, I’d rather align this with Java’s multiline strings and always use unix line delimiters.

[ArneDeutsch]

I had a look about this. At first I thought one could simply fix XtendCompiler#_toJavaStatement(RichString richString, ITreeAppendable b, boolean isReferenced). replacing line 412 (b.append("();");) with b.append("(\"\\n\");"); should do the trick. But it hasn't resolved the issues. I dig deeper. There is post processing logic that replaces the line endings with what is configured for the project.

It seems for me locally I could fix the line ending issues by:

1. Update to latest git (from 2.12 to 2.24.1)
2. Install it such that linux line endings are used (autocrlf=false)
3. Renormalize line breaks for all repositories (disable autocrlf for all repositiories and call git add . --renormalize, afterwards replace all changes with latest from head)

At least inside eclipse context, not sure about outside this scope, Xtext/Xtend seems to work right. At least if environment is configured properly. Perhaps this is just an issue to preconfigure our distro appropriatly and document all this in better way. I will keep my installation as is for now and experiment with it.

[szarnekow]

Are there any settings among the ones you described, that we can automatically apply with Oomph when cloning the repos?

[ArneDeutsch]

I have no experience in configuration of Oomph, but I will have a look. Probably @kthoms can help?

[ArneDeutsch]

The autocrlf setting can be applied by creating a .gitattributes file per repository.

https://help.github.com/en/github/using-git/configuring-git-to-handle-line-endings

If we do so and set text eol=lf everyone should get unix line endings. I would propose to do so. But might have side effects for existing users. I can imagine that we have to renormalize the line endings for local existing repositories (at least windows users).

[miklossy]

The xtext-extras repository is also affected, since the org.eclipse.xtext.xbase.tests makes heavy use of multline strings enclosed by "

[miklossy]

Instead of imposing runtime semantics, I’d rather align this with Java’s multiline strings and always use unix line delimiters.

This could be achieved by converting the line delimiters to Unix style in the LiteralsCompiler.

However, this would affect all Xbase languages, not only Xtend.

[LorenzoBettini]

Instead of imposing runtime semantics, I’d rather align this with Java’s multiline strings and always use unix line delimiters.

@szarnekow, I had also suggested this in another issue I cannot find right now, which may have been lost during the transition to the mono-repo.

I had proposed something similar to #2293 (comment)

I seem to remember that there's an earlier place where the parsed string can be "sanitized" by simply removing \r.

[szarnekow]

If the compiler always emits Unix linebreaks, the model should never contain Windows line endings, either. Consequently, the value converter would be the right service to sanitize the strings.

[LorenzoBettini]

@szarnekow would this org.eclipse.xtext.conversion.impl.STRINGValueConverter.Implementation.convertFromJavaString(String, boolean, int, StringBuilder) be the right place to remove Windows EOLs? Or is it too much?

[szarnekow]

I'd use a subtype specifically used for Xbase string literal values.

[LorenzoBettini]

@szarnekow OK, thanks for the suggestion!

I've updated the PR #3004

If I understood correctly, 8ea116b#diff-71a02996356cc4a288de10e1aa3233fdc96cbdcda67302af2d119480c4bc4960R259 this should do the trick; locally the tests are still green.

This way, neither the LiteralsCompiler nor the XbaseInterpreter need adjustments.