Parameterised transclusions

[Jermolene]

Introduction

This PR introduces a number of improvements and new features related to some of TiddlyWiki's most fundamental components: macros, widgets, operators and transclusion.

The motivation is to fix one of TiddlyWiki 5's early design flaws: the reliance on macros using textual substitution as the primary way to modularise and reuse wikitext and filters.

Experience has shown that while macros are a good match for a small number of tasks, they are brittle and error prone for many common operations. Over the years we have introduced mitigations for the worst problems but these have come at a cost of increased complexity.

The changes in this PR provide powerful new ways to achieve common tasks, and unlock completely new capabilities that were previously impossible in wikitext.

• Procedures, which are essentially what macros should have been; they work in exactly the same way except that parameters are exposed as simple variables (without the double underscores) and no textual substitution takes place
• Custom widgets, allowing the creation of widgets in wikitext, and the redefinition of built-in widgets
• Functions, a new way to encapsulate filter expressions with named parameters
• Custom Filter Operators, allowing functions to be used as custom filter operators
• Parameterised transclusions, allowing strings and wikitext trees to be passed to transclusions

Some smaller improvements are also needed to enable these new features:

• New filter operators for reading JSON data – superseded by JSON Filter Operators (Revised Attempt) #6936
• A new <$genesis> widget that permits any other widget to be dynamically created, with dynamic attributes – cherrypicked to Introduce genesis widget #6961
• A new <$error> widget for displaying core error messages – cherrypicked into Improve recursion detection #6970
• More effective protection against infinitely recursive functions and procedures – cherrypicked into Improve recursion detection #6970

All of these changes are intended to be backwards compatible, and should not affect existing functionality. While they represent a new field of opportunities for wikitext authors, equally it is entirely possible for authors to ignore all these new features and continue to use TiddlyWiki 5 in the way that they have always done.

Background

TiddlyWiki 5 macros were originally based on the technique we call "textual substitution": the string values of the parameters provided when calling a macro would be plugged into the macro definition before it was wikified in the usual way.

A typical example of the approach in early versions of TiddlyWiki 5:

\define mymacro(title)
<$codeblock code={{$title$}}/>
\end

The technique worked well enough to get the basics of the TiddlyWiki 5 user interface up and running, but it was clear from the start that it was annoyingly brittle. For example, the macro above would fail with tiddler titles containing double closing curly braces. Trying to use it with the title foo}}bar would lead to the macro being expanded to the following invalid syntax:

<$codeblock code={{foo}}bar}}/>

As a result, for a long time, the TiddlyWiki 5 user interface failed if a variety of combinations of special characters were found in tiddler titles. Long time users will remember a warning that popped up in the edit template whenever a potentially troublesome character was detected.

Over the years we've mitigated almost all of these issues, particularly by providing access to the macro parameters as variables. For backwards compatibility, this was done without affecting the existing syntax, which required us to adopt the clumsy protocol of wrapping the parameter name in double underscores to get the name of the corresponding variable.

This has all worked well enough for us to fix the UI issues with special characters in tiddler titles, but is very inconsistent and complex, requiring users to grasp multiple mutually exclusive conceptual models for what is going on.

New Features and Improvements

The approach taken by this PR is to add new functionality by extending and augmenting the system without disturbing existing functionality.

This lays the groundwork for macros and related features to be deprecated, which is the point at which users are advised not to use old features, and instead given clear pointers to the equivalent modern functionality.

The new transclusion architecture is not by itself sufficient to enable us to fully deprecate macros yet. To handle the remaining use cases we propose a new backtick quoted attribute format that allows for the substitution of variable values. See #6663 for details.

Procedures

Procedures are the modern replacement for macros. The key difference is that the parameters are only available as wikitext variables (without requiring double underscores), and that textual substitution does not occur.

The syntax for defining a procedure closely resembles the existing syntax for defining a macro:

\procedure hello(when:"afternoon")
<$list filter="[<when>match[morning]]">
	Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
	Good afternoon or evening!
</$list>
\end

Procedures are invoked using the same shortcut syntax that is used for macros:

<<hello>>
<<hello "morning">>
<<hello when:"afternoon">>

Procedures can also be invoked as transcluded variable attributes using the syntax <div class=<<hello>>>. Note that the plain, unwikified text of the procedure will be used, without any parameter processing.

Custom Widgets

Custom widgets are defined with the \widget pragma, which works analogously to the \procedure pragma.

The names of user defined widgets must start with the prefix $$.

Built-in widgets can be overridden by using a single $. Note that it is not possible to create new widgets named with a single $, only to override existing built-in widgets. This is to prevent users creating widgets whose names clash with future core widgets.

Invoking a custom widget transcludes the contents of the variable containing the definition. The widget definition can transclude the content of the calling widget by using the construction <$slot $name="ts-raw"/>.

For example:

\widget $$hello(name)
<div class="greeting">
    <h1>Hello <$text text=<<name>>/></h1>
    <$slot $name="ts-raw"/>
</div>
\end

The custom widget is invoked in the usual way:

<$$hello name="Jeremy">
    This is the greeting
</$$hello>

That example would render as:

<div class="greeting">
    <h1>Hello Jeremy</h1>
    This is the greeting
</div>

Functions

Functions are analogous to procedures except that they encapsulate filter expressions rather than wikitext. They are defined in the same way:

\function add-tax(rate:"10")
[multiply<rate>]
\end

Functions can be invoked via the new function operator:

Total: <$text text={{{ [<total>function[add-tax],[12.5]] }}}/>

Functions can also be invoked as transcluded variable attributes. For example:

<div class=<<myfunction "foobar">>>
...
</div>

Custom Filter Operator Functions

If a user defined function name starts with a period then it can also be invoked directly as a custom filter operator:

\function .add-tax(rate:"1.10") [multiply<rate>]

...

Total: <$text text={{{ [<total>.add-tax[1.125]] }}}/>

Transclude Widget Updates

Many of these improvements depend for their implementation on a significantly upgraded <$transclude> widget. To accommodate the new functionality without breaking backwards compatibility, the widget now operates in two modes:

• Modern mode enables all the new functionality. It is engaged when at least one attribute name starts with a $ character such as $tiddler, $field, $index etc.
• Legacy mode is fully backwards compatible, using attribute names such as tiddler, field, index. It is engaged when none of the attributes start with a $ character

Like macros, procedures are implemented as a special type of variable. The syntax for invoking procedures using the <$transclude> widget is as follows:

<$transclude $variable="hello" when="morning"/>

Note how the attributes $variable, $tiddler, $field, $index etc. that start with a single dollar sign are used to define what to transclude, while the plain attributes are used as parameters to be passed to the procedure.

The transclude widget can be used to pass named parameters to the transclusion. The transcluded content can reference the parameters via the <$parameters> widget.

<$transclude $tiddler="MyTiddler" param1="HelloThere"/>

Parameterised Transclusion

The transclusion shortcut syntax has been extended to allow parameters to be passed to all types of transclusion, not just transclusions of macros/procedures/custom widgets.

The parameters are passed according to their position in the corresponding parameters declaration, not by name.

For example, with the text below in a tiddler titled "FooBar":

\parameters (a,b,c,d)
(definition of the transclusion goes here...)

The shortcut syntax for invoking the parameterised transclusion uses a single vertical bar to separate them from the title of the tiddler being transcluded:

{{FooBar|first|second|third|fourth}}
{{FooBar||template|first|second|third}}

Parameters can be omitted to skip them. For example:

{{FooBar|first||third|fourth}}

Note that omitting the first parameter {{FooBar||second|third}} creates an ambiguity because the resulting double vertical bar causes it to be interpreted as {{title||template|param1}}.

Slotted Parameters

"Slots" provide a way to pass complex structured data to a transclusion instead of the simple string values supported by string parameters.

Slots are named areas that are defined within transcluded text using the <$slot> widget. They are replaced with custom content provided by a corresponding <$fill> widget inside the transclusion invocation. For example:

\procedure hello
<div class="frame">
    <$slot $name="greeting"/>
</div>
\end

The syntax for invoking the procedure using the <$transclude> widget uses the <$fill> widget to pass the content for the slot:

<$transclude $variable="hello">
    <$fill $name="greeting">
        <h1>A heading</h1>
        <p>A paragraph</p>
    </$fill>
</$transclude >

The result would be equivalent to:

<div class="frame">
    <h1>A heading</h1>
    <p>A paragraph</p>
</div>

The contents of the <$slot> widget provide the default contents that are used in case a value is not provided for a named slot. For example:

\procedure hello
<div class="frame">
    <$slot $name="greeting">
        Default text for the named slot "greeting"
    </$slot>
</div>
\end

The entire contents of the transclude widget invocation is available as the special slot fill value ts-raw.

Specifying Content to Display When Target is Missing

Up until now, the content of the transclude widget has been used as the fallback content to display when the target of the transclusion is missing.

To allow room for the new <$fill> widget, the new behaviour is to use the special slot value ts-missing if present, and if there are no <$fill> widgets present, then falling back to the content of the transclude widget.

Open Questions

These need to be settled before merging:

• Do custom widgets really need their own type of definition – they could just be implemented as procedures
• Review naming of "procedures" and "functions"
• Consider adding support for custom filter run prefixes. While it could be added later there's a chance that the design might influence the design of custom filter operators
• Is $$ the best prefix for custom widgets? We have a (very) loose guideline that $ stands for "system", and it is used as a kind of warning sign for system-y stuff that generally doesn't need to bother casual users. But we don't have an equivalent prefix for a user defined namespace. Perhaps ., as we're using with custom operators?
• Is <$fill> a good name? The rationale is that it pairs logically with <$slot> (which in turn reflects the W3C <slot> element). The trouble is that end users won't see the <$slot> widget until they start writing their own relatively complex procedures/widgets, and until that point <$fill> won't make much sense
• Consider making transcluding tiddler text fields understand fields like _parameters, _is_procedure etc

Future Possibilities

These can be addressed after merging:

• Now that the entire parse tree of the contents of the transclude widget is passed to the transclusion as a JSON blob, it might be useful to be able to render a parse tree. In particular, a new flag for the transclude widget that causes it to interpret the target as a JSON parse tree
• Consider adding custom commands, analogous to procedures but for action strings. We may be able to retcon the existing commands into wikitext implementations too
• Refactor core icons to parameterise the width and height (the calendar icon can also parameterise the date that is shown)

Progress

This PR is fully functional, and very, very nearly complete.

Incomplete:

• Disable widget overrides in safe mode
• Function operator should return the input list if the function is missing
• Review performance impact of these changes
• Documentation (see below)
• Refactor isFunctionDefinition/isProcedureDefinition/isWidgetDefinition flags into a single field?
• Review whether other operators that have a fake widget.getVariable() handler (eg $:/core/modules/filters/filter.js) should also implement widget.getVariableInfo()
• Check that transclude widget refresh optimisations still work
• Review and rename test cases
• Extend set widget to set the hidden parse tree properties like isprocedure, and to be able to specify parameters
• Finish (or defer) visible transclusion demo

Completed

• Recursion detection for functions
• ubertransclude widget
• parameters widget
• slot widget
• values widget
• genesis widget
• widget -> ubertransclusion mapping
• use ubertransclude widget for wikitext transclude syntax
• improved framework for wiki-based rendering tests
• extend wikitext transclusion syntax with positional parameters
• wikitext parser rule for new-style \function definitions
• wikitext parser rule for \parameters construction
• importvariables should skip the parameters widget
• parse trees for transcluded variables should be cached
• custom filter operators
• genesis widget shouldn't evaluate attributes of child widget, leave it to the child widget
• improve efficient of recursion detection (currently the $transclusion variable gets excessively long)
• Support for $:/tags/Global alongside $:/tags/Macro

Documentation progress

• Procedure Definitions
• Function Definitions
• Widget Definitions
• Macro Definitions retcon
• Transclusion in WikiText update to add parameters
• SlotWidget
• ParametersWidget
• MacroCallWidget updates – to note that the transclude widget is preferred now
• ImportVariablesWidget updates to skip parameters widgets
• FillWidget
• ErrorWidget
• SetWidget updates to add conditional attribute
• Unknown Operator
• Function Operator
• TranscludeWidget updates
• Format:Json Operator
• JSON Operators Docs
• Visible Transclusion How To
• LetWidget update noting that dollars are now allowed
• GenesisWidget

Notes for documentation

• What we've previously referred to as "JavaScript Macros" are actually now better thought of as "JavaScript functions", because they work more like functions than macros
• Note the difference between the subfilter and function operators (the former interprets the operand as the filter while the latter interprets the operand as the name of the variable containing the filter)
• Note that first parameter of user defined filter operator functions cannot be missing, because the empty double square brackets are required by the operator syntax (in other words, the operator syntax doesn't permit operators to be called with zero operands, just one or more)
• Note that macros, procedures, widgets and functions are all variables, and all share the same namespace. Definitions with the same name will overwrite one another
• Fill widget $name attribute must be specified as a literal string, not a transcluded attribute

Other tickets that can be closed when this is merged

• Macros, all parameters, parameter #3750

[joshuafontany]

Really interesting stuff, and I can see the appeal of simplifying the wikitext and shortcut syntaxes.

[tw-FRed]

@Jermolene, just for clarity and if I understand the doc correctly, the render of the Widget Syntax Invocation example quoted below misses a line and should read:

<div class="greeting">
		<h1>Hello Jeremy</h1>
    This is the greeting
</div>

Fred

Invoking Macros via Widget Syntax

Macros whose names conform to a special format can also be invoked via the standard widget syntax. When invoking a widget called foo the core looks for a variable called <$foo> and if it finds it, transcludes the contents of the variable instead of invoking the widget in the usual way. It also passes the content of the widget as the named slot value body.

For example:

\macro <$hello>(name)
<div class="greeting">
		<h1>Hello <$text text=<<name>>/></h1>
    <$slot $name="body"/>
</div>
\end

The custom widget is invoked in the usual way:

<$hello name="Jeremy">
    This is the greeting
</$hello>

Would render as:

<div class="greeting">
    This is the greeting
</div>

[AnthonyMuscio]

@Jermolene I applaud the objectives of this piece of work and will do what I can to help, text and support this initiative. A couple of points about what I see as critical to keeping this understandable to new and average users.

• As it stands the explanation and names used are possibly too complex to average users to easily adopt. This is understandable at this stage of development. I would suggest allowing time and feedback from the community to help craft appropriate terms. eg other than slot.
• Perhaps the use of the $ variables in the transclude widget could invoke the new behaviour rather than introduce uberwidget (or by another name). Perhaps a parameter such as $content=varormacroname is what results in the use of the other form?
  • <$transclude $tiddler= /> vs <$transclude $content= />
  • In other discussions of the standard transclude, we felt an alias for the "tiddler parameter" of a "template parameter" would allow code to look more readable. <$transclude $template=tiddlername/> because currentTiddler is maintained.

As you suggest I think the Add support for string literal attributes with textual substitution
#6663 is an essential addition. I mention fields there.

[Jermolene]

@Jermolene, just for clarity and if I understand the doc correctly, the render of the Widget Syntax Invocation example quoted below misses a line and should read:

Thanks @tw-FRed, fixed.

• As it stands the explanation and names used are possibly too complex to average users to easily adopt. This is understandable at this stage of development. I would suggest allowing time and feedback from the community to help craft appropriate terms.

Yes @AnthonyMuscio, I think we do need to review all the associated terminology. I've had to invent some new terms and some of the old terms might no longer be such a good fit. For example, I'm not sure that "macros" makes sense now; we might consider "function".

eg other than slot.

The term "slot" is taken from the web custom elements specification:

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/slot

Generally, the implementation of custom widgets is very similar to the custom elements spec, so it seems to make sense to share terminology.

• Perhaps the use of the $ variables in the transclude widget could invoke the new behaviour rather than introduce uberwidget (or by another name). Perhaps a parameter such as $content=varormacroname is what results in the use of the other form?

I think you're suggesting that the transclude widget use the presence of a $ attribute to trigger the new behaviour. That would mean that <$transclude $tiddler="title"/> would trigger the new behaviour (and allow parameters), while <$transclude tiddler="title"/> would still work, but not support passing parameters.

I don't think that will work; the $tiddler attribute is optional, so how would one know whether <$transclude tiddler="title"/> was intending to transclude a tiddler called "title", or to transclude the current tiddler passing a parameter called "tiddler".

• <$transclude $tiddler= /> vs <$transclude $content= />
• In other discussions of the standard transclude, we felt an alias for the "tiddler parameter" of a "template parameter" would allow code to look more readable. <$transclude $template=tiddlername/> because currentTiddler is maintained.

That won't work. A key change here is that we have a single widget that can transclude variables or tiddlers. So we need to have the explicit attribute names so that we know which is which. It would be ambiguous whether <$transclude $content="foobar"/> was transcluding a tiddler or a variable.

[xcazin]

Hi @Jermolene, this feature looks very powerful indeed! FWIW, a name that came to mind was <$paraclude>.

In the spirit of grand unification, I was wondering whether we couldn't do without the $variable, $tiddler, $field and $index attributes, in favour of a single $content attribute that would expect a wikitext value instead of a content source name. For instance:

• <$paraclude $content=<<hello when:"morning">> /> would transclude the content of the macro hello with a morning value for the param when;
• <$paraclude $content=<<hello>> when="morning" /> would transclude the content of the macro/variable hello, passing the morning value to any when <$parameter> widget present in the hello content.
• <$paraclude $content={{hello}} when="morning"/> would transclude the content of the text field of the hello tiddler, passing the morning value to any <$parameter> widget present in the hello tiddler
• <$paraclude $content={{{ [[hello]get[when]] }}} /> would transclude the content of the field when in the hello tiddler
• <$paraclude field="morning"/> would use current tiddler's text field as the default transcluded content, and pass it the morning value through any field attribute present in the <$parameter> widget.
• <$paraclude tiddler="hello"/> would use current tiddler's text field as the default transcluded content, and pass it the hello value through any tiddler attribute present in the <$parameter> widget.

[pmario]

I think you're suggesting that the transclude widget use the presence of a $ attribute to trigger the new behaviour.

I don't think, the idea is too far off. ...

I don't think that will work; the $tiddler attribute is optional,

That's right <$transclude /> will do the same thing as it does now.
<$transclude tiddler=x /> <$transclude $tiddler=x /> are the the same thing. IMO it doesn't matter if it triggers new or old behaviour, since there are no parameters.

... so how would one know whether <$transclude tiddler="title"/> was intending to transclude a tiddler called "title", or to transclude the current tiddler passing a parameter called "tiddler".

Since there is no $ it's the old behaviour. ... done.

There could be a possibility for an empty $ parameter, that triggers the new behaviour eg: <$transclude tiddler=x $/> or <$transclude $="" tiddler=x /> for a more verbose writing. .. I would prefer the first version

Just some thoughts.

[Jermolene]

I've pushed some significant changes, and made substantial edits to the original post at the top of this PR.

The changes are:

• New wikitext shortcut syntax for passing parameters to transclusions (eg {{tiddler|param|another}}) – the syntax is borrowed from Wikipedia
• Merging the ubertransclude widget into the transclude widget (using the technique suggested by @AnthonyMuscio and @pmario above)

In the spirit of grand unification, I was wondering whether we couldn't do without the $variable, $tiddler, $field and $index attributes, in favour of a single $content attribute that would expect a wikitext value instead of a content source name. For instance:

Thanks @xcazin I have been considering that possibility. The advantage of the present arrangement is that because the widget is working with tiddler titles (or variable names) it can easily cache the results of parsing. With a generic $content attribute the widget would only see a plain string, without knowing whether it came from a variable or a tiddler. We might be able to reconstruct that information by looking at the parse tree, but it might get difficult for complex cases.

[pmario]

I'll check out the code and have a closer look soon. Looks very interesting.

[AnthonyMuscio]

I am in the bush so cant review with a fine tooth comb. So friendly feed back.

Do you actualy intend to drop the use of parameter:"value" and use "=" in both function definition and calls?

If so this is added complexity going forward so could we do one of the following?

• also allow = to be an alias for : in macro definitions and calls
• or instead use : in the new function definitions and calls not =

Can we use $macrocall for functions and if we do what happens?

Because I think it relates to the handling in this functions;

Personaly I have long felt it would help if we made available in wikitext to be able to convert between keyword="value" pairs and variables with values and the reverse, generate a set of keyword value pairs and handling the delimiters ' " """.

An example may be programatically building a set of keyword value pairs and passing them to actioncreatetiddler or taking a set of variables and save them as settings in a tiddler field or text.

Another example would be uri's do not permit spaces so a link field could contain a link plus more info eg
https://tiddlywiki.com pretty='tiddlywiki' target="tw"
Then the link field value can be used to get the url split[ ]first[] then use the rest[] to get the keyword value pairs and convert them to variables to use in the link.

Please note I am not trying to abuse by going off topic. Functions as proposed will be at least translating keyword/value pairs to variables if not in the reverse. This seems to me importiant to concider now, if not ultimately spawning another issue.

[Jermolene]

I am now merging this PR, a few days later than scheduled in #7279. Part of the rationale is that this PR is now very long and the comments are hard to navigate.

We can still make refinements to these changes.

[kookma]

@Jermolene
I think some of tiddlers have wrong caption! Example

Procedure Parameter Handling has caption: Macro Definitions

Procedure Definitions has caption: Macro Definitions

[Jermolene]

@Jermolene I think some of tiddlers have wrong caption! Example

Thanks @kookma much appreciated. I'll attend to those as soon as I can.

[kookma]

Direct call to function only returns the first output.

Look at the below examples. A function is defined and is used in several ways.

\function .fn(tid) [tag<tid>]

<<.fn HelloThere>>

<$count filter="[function[.fn],[HelloThere]]" />

<$list filter="[.fn[HelloThere]]" >

</$list>

In the above examples all works except <<.fn HelloThere>> it returns only the first output. Is this by design or a bug?

[btheado]

it returns only the first output. Is this by design or a bug?

As I understand it, it is as intended. See the table in this comment #6666 (comment) for full details on the behavior (though that table is not up to date from this change: #7009).

[kookma]

See the table in this comment #6666 (comment) for full details.

Much appreciated! Is this table added to documentation? It is needed to understand how different new variables work!

[pmario]

@kookma I think it's time to create new issues with questions like the last post. So they can be handled and closed independently.

[Jermolene]

As I understand it, it is as intended. See the table in this comment #6666 (comment) for full details on the behavior (though that table is not up to date from this change: #7009).

Thank you @btheado It would be great if you could make a PR for an updated version of the table? I'd be open to suggestions, but perhaps as a new tiddler "Variable Usage"?

[Jermolene]

This thread is now locked – please create a new ticket with any further questions, comments or bug reports.