[21] JEP 430 String Templates (First Preview)

[mpalat]

JEP 430 at https://openjdk.org/jeps/430
Ref https://cr.openjdk.java.net/~gbierman/jep430/latest for more details on LS changes

At the time of creating this issue, this is not yet targeted but it may be targeted - keeping this in the radar for 20

[mpalat]

This was moved out of JAVA 20 and hence moving t0 BETA_JAVA21

[jarthana]

As per the jdk issue, this is targeted now for 21

[jarthana]

The first goal should be to come up with scanner changes that allow us to recognize and differentiate String literals and templates, preferably using the same code. As per the JEP, I see that the only difference in a template being the presence of an embedded expression that is of the format \{name}.

I have a draft here - https://github.com/jarthana/eclipse.jdt.core/tree/jep430_grammar
But putting it on hold since the latest JDK 21 doesn't seem to have the TemplateProcessor available.

[mpalat]

retargeting out of BETA_JAVA21
(opportunistic)

[akurtakov]

Initial work on the topic seems to have started in #1494

[jarthana]

Initial work on the topic seems to have started in #1494

Yep, I am fairly close to completion of this, although I am still studying and comparing two possible approaches wrt grammar. The PR is probably not updated. Will keep posted.

[paulholder]

Is there another PR where the changes will be made to have the now "globally known" STR, FMT and RAW types defined, exposed into the code if the preview feature is enabled? Also, those constants presumably need to me reserved now and a warning issued if they're used when the preview feature is not enabled.

[jarthana]

On suggestion from @srikanth-sankaran (after I ran into grammar issues with the previous PR), I experimented an alternate approach. The approach is to, on the first pass, scan the entire string template and return a TokenNameStringTemplate. During this step, the scanner doesn't bother with the embedded expressions, except for marking the boundaries of the top level embedded expressions. Later, when the parser encounter the said token, we will spawn new parser+scanner with the boundaries stored and

1. Create the string fragments
2. Invoke Parser#parseExpression() and create embedded expressions for each
3. Put them all together and create the string template.

This is a bit more involved on the scanner side, but makes the grammar lot simpler and avoid the need for multiple synthetic tokens. One of the challenges is getting the format of the text block working, especially when there is a nested text block involved. The JEP doesn't seem to say anything about the influence of the indentation of the outer block on the inner text block.

[srikanth-sankaran]

"The JEP doesn't seem to say anything about the influence of the indentation of the outer block on the inner text block" -- is this a concern only for the alternative design where the scanner returns a single string template token ??

[jarthana]

"The JEP doesn't seem to say anything about the influence of the indentation of the outer block on the inner text block" -- is this a concern only for the alternative design where the scanner returns a single string template token ??

I think it will be something we need to take care of in both designs. The fact that the formatting of a text block can only be done with all lines at hand (and not line by line) means it is even more difficult with the other design where we may be creating multiple text blocks for each fragment. I think this one here gives us more control on the formatting.

[szarnekow]

The approach is to, on the first pass, scan the entire string template and return a TokenNameStringTemplate.

Will this work with text blocks that have embedded expressions that in turn use text blocks?

[jarthana]

The approach is to, on the first pass, scan the entire string template and return a TokenNameStringTemplate.

Will this work with text blocks that have embedded expressions that in turn use text blocks?

It does, with the known issue of formatting I mentioned. This is WIP, but feel free to try out. I have some tests already added in StringTemplateTest

[jarthana]

It does, with the known issue of formatting I mentioned. This is WIP, but feel free to try out. I have some tests already added in StringTemplateTest

This is in decent shape for review. However, there are still known issues or design decisions that need to be discussed.

1. The grammar only works for Name '.' Template. However, this is supposed to work for any expression. This requires some not so simply grammar tweaks.
2. Javac seems to simplify a template expression with STR to a simple string concatenation before code generation. I haven't bothered to do that as I don't know the implication. The current patch creates a method invocation with the processor as the receiver and the template as the only argument. This generic way should work with all kind of processors.
3. There's simply too much happening when it comes to handling the template/string delimiters for text blocks inside scanForTextBlock(). This is something we have been living with since the introduction of text blocks. This was cloned and modified from scanForStringLiteral(). Perhaps, when times permits, we should merge these two.
4. Since the last patch, I also made some changes to completely strip off the text of the embedded expression from the text of the outer text block. This ensures that the formatting of the enclosing text block is not affected by the embedded expressions.

[paulholder]

alternate approach. The approach is to, on the first pass, scan the entire string template and return a TokenNameStringTemplate.

In relation to the question about the importance of spaces, this was covered in the text blocks https://openjdk.org/jeps/378 that are components that StringTemplates are made from. I think your approach to scanning should be to first recognize the Text Block, then scan it for any indication of \{} internally, and if found, change it from a Text Block to a StringTemplate, and then further process that into the proper code to define the StringTemplate instance.

Also someone made reference to just STR but RAW and FMT are also globally defined in the Java code, and users can define their own.

[srikanth-sankaran]

@jarthana can you share some examples of text block string templates where the formatting/indentation is unclear - also documenting what Javac is doing.

I can review the relevant portions of the spec to see if I can help clarify

[jarthana]

@jarthana can you share some examples of text block string templates where the formatting/indentation is unclear - also documenting what Javac is doing.

I can review the relevant portions of the spec to see if I can help clarify

String s = STR."""
....xyz
....\{
.."""
..abc
..def
.."""}
....""";

Javac produces this:

xyz
abc
def

Question is, how does the line that contains the beginning of the inner text block (""") affect the indentation of the outer block?

[jarthana]

alternate approach. The approach is to, on the first pass, scan the entire string template and return a TokenNameStringTemplate.

In relation to the question about the importance of spaces, this was covered in the text blocks https://openjdk.org/jeps/378 that are components that StringTemplates are made from. I think your approach to scanning should be to first recognize the Text Block, then scan it for any indication of {} internally, and if found, change it from a Text Block to a StringTemplate, and then further process that into the proper code to define the StringTemplate instance.

Yes, that's what this patch does. Just that, instead of grammar taking care of it, it's mostly hand-written. I am not sure how that (handling the rules in the grammar) will handle delimiters as unicode characters.

Also someone made reference to just STR but RAW and FMT are also globally defined in the Java code, and users can define their own.
The reference, that I made, was about how Javac seems to be handling STR specially, rather than in a generic way. This patch doesn't differentiate between a STR or other kind of processors.

Other quirk, or perhaps even a less than ideal thing to do, is that when we encounter anoter string literal (or text block) inside an embedded expression, we just spawn another instance of scanner and invoke scanForStringLiteral(). This is just to scan past that literal and the literal is discarded. Perhaps, wasteful and can be optimized.

[mpalat]

@jarthana - please ref - https://cr.openjdk.org/~gbierman/jep459/latest/ captured in #1397 - to track the delta and probably. fyi.