Add support for string literal attributes with textual substitution

[Jermolene]

A common use case for macros is to construct a string from component parts. For example:

\define my-url()
https://$(host)$/$(path)$
\end

<$let host="example.com" path="/files">
  <a href=<<my-url>>>
    ...
  </a>
</$let>

It is proposed to introduce a shortcut syntax to cover these simple use cases.

The shortcut syntax uses backticks to quote the string, with the following substitutions:

• $(variable-name)$ is replaced with the value of the variable
• ${filter}$ is replaced with the first result of evaluating the filter

The example above could be expressed as follows using the new syntax:

<$let host="example.com" path="/files">
  <a href=`https://$(host)$/$(path)$`>
    ...
  </a>
</$let>

An example using filters:

<div style.width=`${ [<width>multiply[3]] }$px`>
...

An example of transcluding a tiddler:

<div style.width=`${ [{mytiddler!!myfieldfield}] }$px`>

[CodaCodr]

style.widget? Is that doc'd somewhere?

[Jermolene]

style.widget? Is that doc'd somewhere?

New in v5.2.2. See https://tiddlywiki.com/#HTML%20in%20WikiText, scroll down to the heading "Style Attributes".

[CodaCodr]

Oh, did you mean style.width? "widget" was my confusion. Looking at the example, I assume it's width.

[Jermolene]

Thanks @CodaCodr it was a typo 🙇

[AnthonyMuscio]

I just want to add the ability to include field content, or from a text reference, into "textural substitutions" is also important.

[pmario]

One question popped up, while reading the spec. How can we dynamically create the substitution string.

<div style.width=`${ [<width>multiply[3]] }$px`>
...
<div style.width=`this text should constructed using variables`>

[Jermolene]

I just want to add the ability to include field content, or from a text reference, into "textural substitutions" is also important.

For example:

<div style.width=`${ [{mytiddler!!myfieldfield}] }$px`>

How can we dynamically create the substitution string.

I'm not sure what you mean.

[AnthonyMuscio]

I just want to add the ability to include field content, or from a text reference, into "textural substitutions" is also important.

For example:

<div style.width=`${ [{mytiddler!!myfieldfield}] }$px`>

that would be the way, within the filter.

I could not see it illustrated. Thanks

[pmario]

I think a simple example would be as follows, where \define is just a placeholder for a more complex function, that returns a string. ... May be I'm only not seeing it.

\define action() multiply

<div style.width=`${ [<width>${action}$[3]] }$px`>

[Jermolene]

I think a simple example would be as follows, where \define is just a placeholder for a more complex function, that returns a string. ... May be I'm only not seeing it.

\define action() multiply

<div style.width=`${ [<width>${action}$[3]] }$px`>

Hi @pmario that's a very interesting example – you've got two layers of substitution nested inside each other. I'm not sure that we'd want to recursively apply the substitutions because of the ease of setting up an infinite loop.

[AnthonyMuscio]

A passing depending on how you handle this would the use of text references be possible?

All one needs to test is there is a leading !!, internal or trailing"!!" It could be included in the syntax, the question is can it be processed?

$(!!fieldname)$

$(tiddlername!!fieldname)$

$(tiddlername!!)$ defaults to text, separates it from variable names.

For example;

<$let host="example.com" path="/files">
  <a href=`https://$(host)$/$(path)$#$(!!tiddler-name)$`>
    ...
  </a>
</$let>

Rather than needing the filter as follows.

<$let host="example.com" path="/files">
  <a href=`https://$(host)$/$(path)$#$([{!!tiddler-name}])$`>
    ...
  </a>
</$let>

This above example $([{!!tiddler-name}])$ causes a form of dyslexia where it is hard for the eyes to parse the braces.

[Jermolene]

Hi @AnthonyMuscio

A passing depending on how you handle this would the use of text references be possible?

Using the presence of !! (or presumably ##) to distinguish between text references and variables is not consistent with how we resolve similar problems elsewhere in the design of TiddlyWiki. Generally, we use different types of quotes to distinguish the contents.

EDIT 1st May 2022: added the critical word "not" above

It would be more consistent to recognise the existing {{title!!field}} syntax, but there are a couple of good reasons not to do that:

• The double curly braces are a wikitext syntax, which might give the impression that the templated string is wikified, which of course it is not (and there is no intention to make it do so for performance reasons)
• Using double curly braces would make it impossible to construct templated strings for attributes that happen to contain wikitext including transclusions. For example, something like this would be ambiguous:

<$list ... emptyMessage=`The answer is {{answertiddler}}`

The problem is that the transclusion would be handled by the textual substitution mechanism, and the result would be passed to the emptyMessage attribute, and hence get wikified when rendered. That would mean that we'd end up wikifying the result of the text reference, which would not be intended.

[AnthonyMuscio]

Jeremy,

Regardless of the validity of the approach I suggested to permit fields to be included as references, inside these concatinations, is there a way to address this need "using field values" without forcing the use of filters such as $( [{!!fieldname}] )$ or
$( [all[current]getfieldname]] )$ ?

Or is there already a way I have missed?

[Jermolene]

Hi @AnthonyMuscio the proposal at the moment doesn't include provision for textual substitution of text references. The rationale is to keep things simple with a single new mechanism for substituting filter results, only keeping the variable substitution syntax for backwards compatibility.

As it stands, the syntax for the examples you give would be ${ [{!!fieldname}] }$ and
${ [all[current]get[fieldname]] }$.

There is a way to address the need that you raise, which would be to introduce a third substitution syntax (perhaps ${textreference}$ and ${{filter}}$). But I am raising the concern that introducing a third syntax makes things more complicated because it means that users will have to think further about which kind of substitution they are doing.

So, the question is whether it is worth having a special syntax for textual substitution of text references, and the considerations are the complexity that it brings.

[AnthonyMuscio]

As it stands, the syntax for the examples you give would be ${ [{!!fieldname}] }$ and
${ [all[current]get[fieldname]] }$.

This is sufficient and quite powerful as it stands where we can effectively use "any filter" to generate a "string" that will be substituted, and through that even a "text reference" $( [{tiddlername!!fieldname}] )$.

As a result I agree lets not complicate it but instead document it as fields and text references can be substituted through the filter $( filter )$ form.

From my naïve perspective can we say effectively that

any variable can have its value substituted using $(varname)$ with this change not only inside macros (as before) but anywhere

Filters can now be used inside the substitution $( filter )$ and its result will be substitute in its place.

[saqimtiaz]

any variable can have its value substituted using $(varname)$ with this change not only inside macros (as before) but anywhere

Not anywhere, only inside string literal attributes.

Filters can now be used inside the substitution $( filter )$ and its result will be substitute in its place.

Please play closer attention to the syntax. Filter substitutions use curly braces:
${ filter }$

[AnthonyMuscio]

A question; in this manufactured case it only works if you use the wikify widget

\define test() <$text text={{!!fieldname}}/>
<$wikify name=test text="<<test>>">
<$list filter="[<test>match[CamelCase]]">
  go it
</$list>
</$wikify>

If I follow the new substitution's correctly will this or something similar work?

\define test() <$text text={{!!fieldname}}/>
<$list filter="[[${test}$]match[CamelCase]]">
  go it
</$list>

If so that would be wonderful.

[saqimtiaz]

> \define test() <$text text={{!!fieldname}}/>
> <$list filter="[[${test}$]match[CamelCase]]">
>   go it
> </$list>

@AnthonyMuscio no, that will not work. Firstly your example does not use string literal attributes. Secondly, you are passing wikitext where a filter is expected. As Jeremy mentioned earlier in this thread, there is no intention to support automatic wikification due to performance considerations. Wikifying some wikitext to return a string will always be slower than generating it directly via a filter, where possible.

[pmario]

It is proposed to introduce a shortcut syntax to cover these simple use cases.

The shortcut syntax uses backticks to quote the string, with the following substitutions:

• $(variable-name)$ is replaced with the value of the variable
• ${filter}$ is replaced with the first result of evaluating the filter

I think $(filter)$ isn't needed and the limitation to only use the first result shouldn't be there. Same reasoning as in " [IDEA] :foreach filter run prefix" #6781

IMO <$set name=myVar filter="[[a b]] my filter +[nth[2]]"> would do just fine.

It should be possible to even tell the macro, that it should treat the variable as a tiddlywiki list.

<$set name=myVar filter="[[a b]] my filter">
<$text text=<<jsstring "some new text $(myVar)$. " isList=yes>>/>
</$set>

the output may be: (I'm not sure if the macro should iterate over an array )

some new text [[a b]]. some new text my. some new text filter. 

isList defaults to "no", so the string is treated as simple string.

If brackets are a problem the filter can look like this: [[a my]] filter +[join[ ]], which will remove them.

<$set name=myVar filter="[[a b]] my filter">
<$text text=<<jsstring "[[$(myVar)]]$\n" isList=yes>>/>
</$set>

would create

[[a b]]
[[a]]
[[b]]

and so on. ... No code written yet. Just some thoughts.

[Jermolene]

Thanks @pmario

I think $(filter)$ isn't needed

Why?

The idea is to make it easy to insert expressions like today's date is ${ [<now>] }$.

The overall goal here is to make a better alternative to macros for textual substitution tasks, hence the desire to make it as powerful as is appropriate.

the limitation to only use the first result shouldn't be there. Same reasoning as in " [IDEA] :foreach filter run prefix" #6781

I don't see how the reasoning in #6781 applies here. The point there is that there is no workaround available to give the ability to return multiple results from the "filter" prefix. That doesn't apply here.

But it still may be reasonable to accept the entire list that the ${x}$ syntax returns. Presumably we'd just concatenate them.

IMO <$set name=myVar filter="[[a b]] my filter +[nth[2]]"> would do just fine.

Here we're trying to improve on that syntax; it's indirect and verbose.

It should be possible to even tell the macro, that it should treat the variable as a tiddlywiki list.

<$set name=myVar filter="[[a b]] my filter">
<$text text=<<jsstring "some new text $(myVar)$. " isList=yes>>/>
</$set>

I don't think that looks very good. Generally if we've got a choice between two apparently mutually exclusive choices, the worst option is to support both. That means that users have to grapple with the complexity of both. Good system design is about keeping things conceptually simple for users.

I think a macro is not the best approach for what you're trying to do because we can't use computed parameters within a filter.

The approach I was thinking was that the string literal syntax would be a shortcut for a filter operator alternative. For example:

<div title=`a string $(sub)$`>

would be a shortcut for:

<div title={{{ [substitute[a string $(sub)$]] }}}>

(Important to note that the shortcut syntax would be able to cope with double square brackets and other character combinations that would be illegal in the filter operator form).

In the filter operator form, it might be nice to also support $1, $2 for optional positional parameters:

<div title={{{ [substitute[a $1 string],<adverb>] }}}>

Mind you, there would be nothing to stop us supporting parameters with the shortcut syntax too:

<div title=`a $1 string`,<adverb>>

[saqimtiaz]

In the filter operator form, it might be nice to also support $1, $2 for optional positional parameters:

I would very much like to see this implemented. I do something similar with my custom printf filter operator which is a must in every single one of my wikis.

[AnthonyMuscio]

I understand this is to be part of TW 5.3.0?

• Will it be in the pre-release?

I understand the back tick format is also used for inline code highlighting.

• Will previous uses to highlight and stop wikification be backwardly compatible?

Use the `$(currentTiddler)$`variable in macros
<$let var=`$(currentTiddler)$`>
now <<var>>
</$let>

[Jermolene]

I understand this is to be part of TW 5.3.0?

No, this enhancement is not planned for v5.3.0. This ticket is part of showing the roadmap for how we will eventually be able to deprecate macros.

• Will previous uses to highlight and stop wikification be backwardly compatible?

No, the context is different. Here the backticks are used as special quotation symbols for attributes, not as freestanding syntax like code sections.

[pmario]

I wanted to solve exactly that problem with \function and \procedure using PR #6666 as follows. It would work fine if the link-widget would have the same functionality as the wikitext links eg:

[[local link]] .. [ext[external link]]

<$link to="local link" /> .. <$link ext="external link" />

Proposal

Add an ext parameter to the link-widget so it would look like this

\procedure URL(protocol:"https", host, path, tiddler)
<$link ext=<<f-URL>> />
\end

or even better

<$link ext=<<dynURL host:"tiddlywiki.com" tiddler:"HelloThere">> />

Example

A very common usecase is the dynamic creation of URLs, where our users always have problems and the naive implementation creates several technical problems

Code with Function Definition

\function dynURL(protocol:"https", host, path, tiddler) [<protocol>][[://]][<host>][<path>][[#]][<tiddler>] +[join[]]

Eg: create a dynamic link to the HelloThere tiddler at tiddlywiki.com

<<dynURL host:"tiddlywiki.com" tiddler:"HelloThere">> -> Creates a concatenated string

Rendered as: https://tiddlywiki.com#HelloThere -> Creates a concatenated string

.But it should be a link. We need to create external links using the A HTML tag, which is complicated and error-prone, because the HTML code needs additional parameters like "class", "rel" and "target"

\function f-URL() [<protocol>][[://]][<host>][<path>][[#]][<tiddler>] +[join[]]

\procedure URL(protocol:"https", host, path, tiddler)
<a class="tc-tiddlylink-external" href=<<f-URL>> rel="noopener noreferrer" target="_blank"><<f-URL>></a>
\end

[pmario]

Edit: I did add the following intro text to the first post

I wanted to solve exactly that problem with \function and \procedure using PR #6666 as follows. It would work fine if the link-widget would have the same functionality as the wikitext links eg:

[[local link]] .. [ext[external link]]

<$link to="local link" /> .. <$link ext="external link" />

[Jermolene]

Hi @pmario as things stand, the link widget is only used for tiddler links, and is not involved in rendering external links, even if they were made using the same double square bracket syntax as pretty links. This is useful; it means, for example, that it is easy to scan a parse tree looking for tiddler links. The relationship between tiddler links and external links is that tiddler links are layered over the top of the implementation of external links.

Adding the facility to make external links would be smooshing together functionality that is really quite different, making a frankenstein widget with two only tenuously related areas of functionality.

I think the red flag here is that it is an attempt to optimise for brevity at the widget layer. I think that that is almost always a bad idea because it inevitably leads to more complex semantics for the widgets that are the primitives in our system. The more complex that those semantics become the harder it is to reason about how the primitives combine together.

In this case, it sounds like your use case could be served with a custom <$$extlink> widget that hides the messy details of the class, rel and target attributes.

[pmario]

In this case, it sounds like your use case could be served with a custom <$$extlink> widget that hides the messy details of the class, rel and target attributes.

That's right. I'll test that

[pmario]

Just to be sure we need to be able to do stuff a shown in an even simpler way. https://talk.tiddlywiki.org/t/how-text-substitution-can-look-like-with-v5-3-0-prerelease/6607

\function sub() "[" [<operator>] ":" [<suffix>] "<term>]" +[join[]]

\procedure search-macro(f, operator, suffix, term)
  debug: <$text text=<<sub>>/>
  <$list filter="[subfilter<f>] +[subfilter<sub>] +[limit[1]]"/>
\end

<<search-macro "[all[tiddlers]!is[system]]" "regexp" "title" "\.[a-zA-Z]{2,4}$">>

----

<<search-macro "[!is[system]limit[250]]" "search" "text" "operator">>

---

slightly related: #6826

[AnthonyMuscio]

I think $(filter)$ isn't needed and the limitation to only use the first result shouldn't be there. Same reasoning as in " [IDEA] :foreach filter run prefix" #6781

From @pmario

With 5.3.0 pending, and this feature "Add support for string literal attributes with textual substitution
#6663" possibly for the next release, I would like to restate the desire not to return only the first result

an example may be;

<div class=${ [subfilter] }$>

• Allowing a parameter to be fed a list

Although perhaps {{{ [subfilter<class-list>] }}} would work?

• but it too, is limited to one value when in use as a parameter.

I understand the motivation for the first value, but it seems an unnecessary handicap.

• Or perhaps an override?
  An uninformed guess

<div class=`${ =[subfilter<class-list>] }$`>

[saqimtiaz]

@Jermolene A couple of questions as I work on an implementation:

1. Can you think of a good name for the new style widget attributes denoted by backticks?
   The working name I have is substituted attributes but I think we might need something more intuitive.

2. Similarly, the operator equivalent has a working name of substitute[] and I wonder if we can do better.

[Jermolene]

Hi @saqimtiaz I think substitute is a good name. The other term I see used in other languages is "interpolated string", but that seems a it obscure for us.

A separate point, should we perhaps allow escape sequences within substituted strings?

<div attr=`A linebreak is \n; a letter z is encoded as \x7A or  \u007A or \u{7A}`>

Another point, can we allow line breaks within substituted strings? It would be quite useful in some situations.

<div attr=`This string contains
a linebreak`>

Also, I wonder if we might support arbitrary numbers of backticks to demarcate the string. So it would be possible to do:

<div attr=``This is a string that includes this ` backtick within it``>

[saqimtiaz]

Another point, can we allow line breaks within substituted strings? It would be quite useful in some situations.
Yes, this is already a given.

Also, I wonder if we might support arbitrary numbers of backticks to demarcate the string.

Good idea, that makes perfect sense.

A separate point, should we perhaps allow escape sequences within substituted strings?

Are you thinking in terms of allowing escape sequences to pass through without processing?

[Jermolene]

Are you thinking in terms of allowing escape sequences to pass through without processing?

I was wondering about additionally substituting escape sequences with the character string they represent, in the same way that JavaScript allows escape sequences in template literals.

[pmario]

Just as a reference.

Jeremy

<div attr=`A linebreak is \n; a letter z is encoded as \x7A or \u007A or \u{7A}`>

Saq

Are you thinking in terms of allowing escape sequences to pass through without processing?

I think having double backtick to cover "single backticks" is a nice to have. Markdown does the same thing, which can be tested here in the editor too.

I did have a look at: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#description and did search for "secape". ... It said:

To escape a backtick in a template literal, put a backslash (\) before the backtick.

`\`` === "`"; // true

Not sure, if the dollar sign is important for us.

Dollar signs can be escaped as well to prevent interpolation.

`\${1}` === "${1}"; // true

Later in the text it references "normal" string-escape sequences: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#escape_sequences

So \\ gives us the \ ... BUT my concern is, that all the js-like escaping may be confusing for our users.

[saqimtiaz]

There is a draft implementation at #7526, I will continue work on it after the weekend.
There is a question there regarding variable naming for which feedback would be welcome.

[Jermolene]

BUT my concern is, that all the js-like escaping may be confusing for our users.

Yes. The main way that people might bump into it is if they are writing text that includes backslash characters. In practice, those are rare in English, but when they do occur the escaping is going to be nightmarish.

Something to consider is the impact on regular expressions – if we support escaping then users have to write this:

^\\\\(function|procedure|widget)\\s+([^(\\s]+)\\((\\s*([^)]*))?\\)(\\s*\\r?\\n)?

instead of

^\\(function|procedure|widget)\s+([^(\s]+)\((\s*([^)]*))?\)(\s*\r?\n)?

[pmario]

... In practice, those are rare in English, but when they do occur the escaping is going to be nightmarish.

Yes, rare in English prose text, but not if you use windows\paths in your descriptions. So we should avoid that one.

[Jermolene]

So we should avoid that one.

Do you mean that we shouldn't perform escaping of substitution templates? In other words, that we should leave backslash sequences alone.

[pmario]

Do you mean that we shouldn't perform escaping of substitution templates? In other words, that we should leave backslash sequences alone.

TLDR;
I would be happy, if we would find a way to get around it, because I want to avoid something like this: \\\\\\\\server\\\\share which imo is a nightmare without a regexp but would be needed for dynamic docs for paths in Windows.

I do not have a good idea, but I'll think about it.

---

Reasoning

With unix systems and forward slashes paths are easy. But windows is a different beast here. eg:

C:\Users\Test is easy.

The fun part start with network mounts that use UNC paths eg: \\NAS-XX\Public

or

\\localhost\c$\Users\Test the Test user directory as an UNC path which contains a $ char which is forbidden in user paths but valid here.

WSL Windows Subsystem for Linux is a new feature in Windows, but mounting drives can be tricky.

There are 2 doc paragraphs, which show how to mount a windows drive in a WSL unix instance

• Mounting DrvFs
  • ... This one uses forward slashes which is no problem
• Mounting network locations
  • ... Network mounts are problematic

It says:

Mounting network locations

When you wish to mount a network location, you can of course create a mapped network drive in Windows and mount that as indicated above. However, it's also possible to mount them directly using a UNC path:

$ sudo mount -t drvfs '\\server\share' /mnt/share

Note the single quotes around the UNC path; these are necessary to prevent the need to escape the backslashes. If you don't surround the UNC path with single quotes, you need to escape the backslashes by doubling them (e.g. \\\\server\\share).

So now imagine I want to dynamically create the path in the (e.g. \\\\server\\share). .... If we do need backslash escapes I would need to use \\\\\\\\server\\\\share to create a string to describe it. That's 100% error prone.

[Jermolene]

Hi @pmario Windows file paths are definitely relevant as an example of typing backslashes. However, the context here isn't the general content that might be written by users, but rather the type of strings that might need to be used with textual substitution like this. So, I might have a wiki entirely filled with Windows file system documentation that makes extensive use of backslashes, but that doesn't mean that we'd find any backslashes inside template literal strings.

[pmario]

So, I might have a wiki entirely filled with Windows file system documentation that makes extensive use of backslashes, but that doesn't mean that we'd find any backslashes inside template literal strings.

IMO the chance is high, that windows paths are dynamically created using template literals even for documentation strings. It will be a pita. I'm pretty sure about that.

[Jermolene]

Hi @pmario I did a brief search on stack overflow to see if there were a lot of people complaining about escaping path separators in JavaScript template literals, and couldn't find evidence of widespread problems.

Maybe we could resolve the problem by adding support for tagged template literals (as JS calls them; it might not be a helpful name for us to use). This just means that the quoted string is preceded by the name of a function that is used to transform the string. That function can replace the escaped string with the raw input string, and thus we could provide a raw option like:

<$text text=raw`\Users\$(username)$\folders\$1$`/>

[pmario]

I do like the idea of "tagged templates" if that's doable. I think it would definitely be worth exploring. Then for the default behaviour we could use default escape rules and for windows-paths we could have a tagged template that is more relaxed using backsalshes.

[Jermolene]

What do you think @saqimtiaz? One disadvantage of the raw idea is that it requires us to maintain a global variable called raw which seems likely to clash with user variable names.

[pmario]

IMO the tag could still be raw but the internal variable could be tv-raw or something similar. So users already know that they should not create tv-xx variables, because they belong to the core. ... Just a thought

[saqimtiaz]

I'm away until Monday afternoon with just my phone so a more considered reply will follow then.

My immediate instinct is that we should consider that the new attribute form is driven by the need to provide an easier form of text substitution for our end users, who are largely not programmers. As such, I think the default behaviour should be that text that is provided in the new attributes is parsed just as text in double quotes currnelty is with regards to escaping, and that any alternate behaviour is opted into with an extension of the syntax/operator prefix.

[saqimtiaz]

My current opinion regarding escape characters and sequences is that if needed, this can be accomplished either via filter operators, or via adding support for tagged templates in the future. The default behaviour regarding escaping should be identical to other types of attributes.

[Jermolene]

My current opinion regarding escape characters and sequences is that if needed, this can be accomplished either via filter operators, or via adding support for tagged templates in the future. The default behaviour regarding escaping should be identical to other types of attributes.

That makes sense. We could introduce operators for "jsescape" and "jsunescape".