[XQuery] String Value Templates

[rhdunn]

A string value template (SVT) is a StringLiteral that supports enclosed expression values and entities. It is written as either T"..." or T'...', where the T stands for "template".

Note: An SVT is similar to an attribute value template or text value template in XSLT.

For instance, the following expression:

for $s in ("one", "two", "red", "blue")
return T"{$s} fish"

evaluates to the sequence ("one fish", "two fish", "red fish", "blue fish").

Note: A string value template T"xyz" is equivalent to the expression <svt t="xyz"/>/@t/string().

Grammar

PrimaryExpr ::= ... | StringValueTemplate
StringValueTemplate ::= ('T"' (EscapeQuot | QuotAttrValueContent)* '"')
                      | ("T'" (EscapeApos | AposAttrValueContent)* "'")

Note: The T" and T' are a single token/unit (i.e. no whitespace/comments are allowed between the characters), just like the Q{ in BracedURILiterals.

[ChristianGruen]

This reminds me of the existing string constructor:

for $s in ("one", "two", "red", "blue")
return ``[`{$s}` fish]``

Are there any differences?

[rhdunn]

The main motivation for this is making string constructor style constructs easier to read, per discussions on the #xpath-ng slack channel on Dec 18 (https://xmlcom.slack.com/archives/C01GVC3JLHE/p1608283860102200).

The key difference in logic is that this processes entities while string constructors don't (although I'm open to making it work the same as string constuctors). -- The motivation for supporting entities was to a) keep the parallel with attribute/text value templates (as StringLiterals support entities), and b) keep the syntax/grammar simple (i.e. re-using the attribute syntax).

Note: In that thread, Michael Kay proposed the `{$key}={$value}` syntax, although I don't know if he intended that to support entities or not.

[benibela]

I have implemented something like that In Xidel. I call them extended strings, and have the prefix x:

for $s in ("one", "two", "red", "blue")
return x"{$s} fish"

[ChristianGruen]

Thanks for the clarification; sounds helpful!

[benibela]

Btw, I had suggested a bunch of them: https://www.w3.org/Bugs/Public/show_bug.cgi?id=29360

[liamquin]

@rhdunn i wonder if there's a way to make the existing string constructors do what you want? For example, ......{xxx}... would not conflict with ....`{...`}.... in practice (unless you wanted a string starting with ` and then users would have to construct it with concatenation i expect) but then it might be easier for people to remember. Iwish we had .... in XSLT too, but then, i wish for.... had grouping and sorting in XPath and hence XSLT too, mostly because i tend to forget it doesn't! But maybe this is revisiting that awful discussion at the face to face we had about this syntax!

[rhdunn]

That's an interesting idea. The issue is with backward compatibility when a StringLiteral contains { and } characters (which currently does not work as an EnclosedExpr.

I wasn't at those discussions, so I can't make that call. Also, we now have implementation experience on string constructors in XQuery, and text value templates in XSLT.

[line-o]

👍🏼✖️💯 for easy to use template strings!
I also like the idea of prepending it with something like the capital T, x or something but I could also live with not having that.
I still do not get the difference between StringConstructors and the SVTs proposed here.
Can you give an entity processing example, @rhdunn ?
As I understand it, SVTs should replace StringConstructors completely. Otherwise there will confusion and just-another-way-to-get-there.
On top of that, I would like to see SVTs to be enclosed in backticks rather than single or double quotes because rarely these will be in the content and have to be escaped.

Would the below example render a JSON-ish representation of the map $m?

T`\{
{
  map:for-each($m, function ($k, $v) { T`"{$k}": "{$v}"` }) 
  => string-join(",
")
}
\}`

[liamquin]

👍🏼heavy_multiplication_x100 for easy to use template strings!

I still do not get the difference between StringConstructors and the SVTs proposed here.

I am not sure that there is a significant difference. The minor difference is that in string constructors, { } are not special - when i proposed string constructors i had in mind a use case such as generating CSS or JavaScript with very occasional embedded values. We ended up with different (probably much better) syntax than i had proposed, after a lot more discussion than i had expected.

Making \ special in XQuery or XPath should be avoided if possible - regular expressions are already hard enough, especially when the query is wholly or partly embedded e.g. in a PHP or Java or JavaScript string.

map:for-each($m, function($k, $v) { ["{ $k }" : "{ $v `}" } => string-join($newline)
seems slightly clearer to me. And,
let $string := function($contents) { '"' || my:escape-quote($contents) || '"' }
return map:for-each($m, function($k, $v) { $string($k) || ":" || $string($v) } => string-join($newline)
seems much easier to read.

But @benibela is doing a lot of complex string processing - or Xidel users are, rather - and have specific needs.

[rhdunn]

The intention behind this idea is to make use of the existing attribute value template syntax and infrastructure to apply that to string literals. Entity processing is therefore the same, so "1&2" would be the string "1&2" (just like with current StringLiteral), but with this proposal, you could write T"{$x} + {$y} = {$x + $y}" and have it consistent with <a b="{$x} + {$y} = {$x + $y}"/>.

[line-o]

What does one have to do in order to get literal curlies in attribute content, then?

<a selected="{{usedInJSFrameworks}}" >{{likePolymer}}</a>

[line-o]

Oh my @benibela what a thread! And I stepped into the trap of wanting single backticks not knowing what I got into ...

[liamquin]

To include curly brackets use e.g. <a data-curly-brace="{ '{' }" /> i.e. the string value. The XSLT 3 specification doesn't seem to talk about other ways to escape the braces.

…

-- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org

[line-o]

The below works in existdb

• {{ => { and
• "" => "

<a href="{{{{link}}}}">{{name}}</a>

Analog to that we could have

T"{{""{$key}"": {$value + 1}}}"

to get to

{"a_key": 2}

[rhdunn]

Yes, XQuery supports {{ and }} to escape braces. -- See 3.9.1.1 Attributes step 1.a. and 1.b.

https://www.w3.org/TR/xquery-30/#prod-xquery30-CommonContent

[michaelhkay]

I'm inclined to support this - mainly because the existing StringConstructor syntax in XQuery is hopelessly unmemorable and unreadable, and I don't want to introduce it to XPath or XSLT in its current form.

I would go for using the syntax ${There were {$n} green bottles}. (The leading ${ is used in many templating languages, though there is a risk that people might confuse ${n} with {$n})

Within the outer curly braces, the rules are as for XSLT AVTs and TVTs. The only characters that need escaping are curly braces, which are escaped by doubling.

What about & escape sequences (entities)? We could follow the precedent of string literals, where entity and character references are recognized in XQuery but not in XPath (which means they can be used in XSLT, but are handled by the XML parser, not by the XPath parser).

[line-o]

I also see the risk of ${n} being confused with {$n}.
A string template literal that solely prints the variable $n would become ${{$n}}. Wouldn't the double curlies not even add to the confusion?

[rhdunn]

Would $"$n = {$n}" (and $'$n = {$n}') work? (Specifically mirroring the existing string syntax.) That should avoid the confusion.

[rhdunn]

Regarding the use of {{ and }} to escape curly quotes, this is already present in XQuery for attribute values, along with "" and '' for escaping those quotes in strings where the string quote style matches the quote needing escaping.

[liamquin]

When i first proposed this, i experimented with <<EOF....EOF (like the Unix shell, Perl, etc) and ${.....} and @{.....}. I proposed ${...} and it was rejected (actually,, ironically, most vociferously by someone now supporting it) out of fears of confusion with {$foo}. It's true i was working in XQuery at the time and hadn't see text value templates, which uses something like "...{ exprsingle }...".

The ``[ ... ]`` syntax was a compromise that's similar to markdown. But we've got it now, and i'm not sure i'd want to have something different and incompatible in XSLT or other uses of XPath.

I would like to see interpolated strings in XPath (i was surprised they only ended up in XQuery in fact; probably my fault for making the original proposal for XQuery).

[ChristianGruen]

I agree with Liam: If we introduce yet another string template/constructor syntax, we'll need to justify why we provide different solutions (at least in XQuery) that are more or less identical.

[line-o]

I really like the idea to create something that resembles an attribute value (like @rhdunn proposed).
That way it would not introduce a new concept, there is plenty of test cases available and it can be communicated easily.

attr="{ expr }"

T"{ expr }"

[michaelhkay]

A suggestion that tries to improve the StringConstructors in XQuery 3.1, rather than abandoning them and doing something completely unrelated.

The current syntax is:

[177] StringConstructor ::= "``["  StringConstructorContent "]``"
[180] StringConstructorInterpolation ::= "`{"  Expr? "}`"

I'd suggest supplementing this with

[177b] SimpleStringConstructor ::= "`"  SimpleStringConstructorContent "`"
[180b] SimpleStringConstructorInterpolation ::= "{"  Expr? "}"

so the spec example

for $s in ("one", "two", "red", "blue")
return ``[`{$s}` fish]``

simplifies to

for $s in ("one", "two", "red", "blue")
return `{$s} fish`

In the simplified version, backticks and curly braces can be doubled to escape them. The content must be non-empty so "``" is not allowed

[ChristianGruen]

I like this simplified proposal a lot. I assume (hope) that entity references in the original string are not going to be resolved?

[rhdunn]

I like this as well. It's in line with the intent of my original proposal.

[line-o]

That's a great proposal @michaelhkay

[joewiz]

Yes, yes, yes!

[liamquin]

The use case i wanted to solve with these string constructors was being able to generate CSS, JavaScript, and other syntaxes containing curly brackets, while still being able to interpolate values. Certainly i have heard from people using them in this way. It's true that embedding a `{ or `} is fiddly in the current mechanism - e.g. can be done by breaking the string and joining it with a literal, rather as one has to do to include ]]> in a CDATA section. So although i like `...` as you propose, it doesn't meet the original needs, so presumably both syntaxes would coexist?

…

-- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org

[michaelhkay]

I propose resolution of this issue as follows:

(a) the current StringConstructor in XQuery is retained unchanged, and remains XQuery-only.

(b) the justification for introducing something new (if we need one) is that for many simple cases it is simpler, and more suitable for scenarios in which XPath is embedded in a language such as XSLT. It has drawbacks, however, when constructing text that is rich in curly braces, for example JSON, CSS, or Java code. Or for that matter, text containing backticks.

(c) for both XPath and XQuery we introduce the new primary expression

StringTemplate ::= "`" StringTemplateContent "`"
StringTemplateContent ::= (StringTemplateFixedPart | StringTemplateVariablePart)*
StringTemplateFixedPart ::= ((Char - ('{', '}', '`') | '{{' | '}}' | '``')*
StringTemplateVariablePart ::= EnclosedExpr

The semantics are identical to XSLT Value Templates (XSLT 3.0 §5.6) with the additional rule that a fixed part must not contain an unescaped backtick, and a literal backtick can be expressed by doubling it.

Note that this rule means that a StringTemplate cannot start with the sequence tick-tick-[, so there is no ambiguity with the existing StringConstructor syntax.

[michaelhkay]

The feature has now been implemented in the spec.