templater docs

doc/0x.KaraTemplater.md

@@ -0,0 +1,272 @@
+Hi. I'm The0x539's templater. I don't really have a name per se, so The0x539 wrote my docs in the first person as a very strange workaround. It doesn't come up much, so hopefully it isn't too distracting.
+
+If you already have experience with a templater, many aspects of my operation may be quite familiar. However, there are a few key differences in basic usage that may cause confusion when switching over.
+Notably, though I share many features with other templaters, I do not support templates written for them, or vice versa, aside from simple incidental cases.
+This isn't actually anything new: the two other major templaters already aren't compatible with one another, but it seems worthwhile to make the incompatibility clear and explicit. When writing a template, one must choose which templater to target. This was already the case before I came along.
+
+# Templates
+Any reasonable template will consist of two things, which these docs will call *components* and *input*.
+In-depth descriptions of each will follow, but briefly, for those somewhat familiar with templating, a *component* line is one marked `code`/`template`(/`mixin`, more on that later) and an `input` line is one marked `kara`.
+With some exceptions (primarily `code once`), the idea behind executing a template is to, for each input line, "run" each component against that line (or against each char/etc within it).
+
+## Input
+Input is probably the simpler of the two varieties of component. Simply put, for a typical template, an input line consists of one line of lyrics for a song.
+The lyrics may include k-timing tags, for use with `syl`-class components. Details on k-timing are beyond the scope of this document.
+Beyond k-timing, I currently have no first-class features for working with karaskel's inline-fx tags (*e.g.*, `\-foo`, note the hyphen), but they are nonetheless present in the `syl` objects for use in code.
+
+Importantly, a line of input *must* have its effect field set to `kara` (or `karaoke`).
+**Unlike other templaters, I do not treat a line as input unless it is already explicitly marked as such.**
+This is not only a deliberate design decision, but also (arguably) necessary due to my more advanced conditional execution logic.
+
+Input lines' actor fields are very useful for conditional execution purposes.
+On a basic level, different actor values can be used to denote different parts of a song, applying a different set of components to each.
+Getting more advanced, a template could be written to have a common `template` component apply to all the input, but select different `mixin`s based on parts of the song, perhaps to differ in color but share all other attributes.
+In its most powerful form, an input line's actor field can be used as a collection of modifiers to mix and match different components on a per-line basis.
+More on this later.
+
+## Components
+Components are the *fun* part of making a template. Components come in three varieties: `code`, `template`, and `mixin`.
+My `code` and `template` components are, at a basic level, very similar to those of other templaters, and `mixin` components are like a more powerful form of karaOK's `template lsyl` family of components.
+
+A component is defined by its effect field, which acts as a space-separated list of tokens.
+The first token must be `code`, `template`, or `mixin`, depending on which kind of component it is.
+The second token is the component's *class*, which is one of the following:
+
+|**Class**|**Description**|
+|-|-|
+|`once`|Runs once at the beginning of template evaluation. Only valid for `code` components. (`template once` TBD)|
+|`line`|Runs once for each line of input.|
+|`syl`|Runs once for each k-timed `syl` within each line of input. Lines with no k-timing information will be treated as one big syl.|
+|`word`|Like `syl`, but separated by whitespace instead of k-timing tags.|
+|`char`|Runs once for each character (currently uses `unicode.chars`) within each line of input.|
+
+Further tokens within a component's effect field, if present, are *modifiers*. More on those later.
+
+### `code` Components
+A `code` component is simple: running it executes its text as a block of Lua code.
+Their most common use is `code once` components that import utility libraries, define helper functions, or set up global constants for use elsewhere in the template.
+However, the other classes do occasionally prove useful, executing the code on, *e.g.*, a per-line basis.
+
+### `template` Components
+As their name might imply, `template` components are central to writing a template, as with my predecessors.
+It is their sole responsibility to generate new lines of *output*. If there are no `template` components, no lines will be generated.
+Generally speaking, running a `template` component entails evaluating any inline variables or inline expressions in its text, then generating a line of output consisting of the evaluated template text and the piece of text that the component's running against.
+For example, consider the following simple template and the output it produces based on the given input:
+
+```
+template line: {\placeholder}
+template syl: {\foo}
+kara: {\k86}Word{\k41}less{\k24}ly {\k87}watch{\k33}ing {\k63}he {\k66}waits{\k25} {\k44}by {\k22}the {\k57}win{\k69}dow
+kara: {\k36}And {\k38}won{\k59}ders{\k43} {\k25}at {\k16}the {\k40}emp{\k21}ty {\k49}place {\k26}in{\k85}side
+fx: {\placeholder}Wordlessly watching he waits by the window
+fx: {\foo}Word
+fx: {\foo}less
+fx: {\foo}ly
+fx: {\foo}watch
+fx: {\foo}ing
+fx: {\foo}he
+fx: {\foo}waits
+fx: {\foo}
+fx: {\foo}by
+fx: {\foo}the
+fx: {\foo}win
+fx: {\foo}dow
+fx: {\placeholder}And wonders at the empty place inside
+fx: {\foo}And
+fx: {\foo}won
+fx: {\foo}ders
+fx: {\foo}
+fx: {\foo}at
+fx: {\foo}the
+fx: {\foo}emp
+fx: {\foo}ty
+fx: {\foo}place
+fx: {\foo}in
+fx: {\foo}side
+```
+
+Already this is useful for simple song styles, since you can now apply a set of tags to a set of lines, perhaps even with multiple layers, while not having to manually keep everything consistent.
+Add some inline expressions to generate more complicated tags to each line and things start to get interesting.
+
+### `mixin` Components
+Mixins, as a first-class component variety, are my defining feature. They modify the output produced by `template` components, generally adding tags. If `template lsyl` from karaOK is unfamiliar to you, consider the following basic demonstration:
+```
+template line: {\Tpl}
+mixin line: {\line}
+mixin syl: {\syl}
+mixin char: {\char}
+kara: {\k86}Word{\k41}less{\k24}ly {\k87}watch{\k33}ing {\k63}he {\k66}waits{\k25} {\k44}by {\k22}the {\k57}win{\k69}dow
+kara: {\k36}And {\k38}won{\k59}ders{\k43} {\k25}at {\k16}the {\k40}emp{\k21}ty {\k49}place {\k26}in{\k85}side
+fx: {\Tpl\line\syl\syl\char}W{\char}o{\char}r{\char}d{\syl\syl\char}l{\char}e{\char}s{\char}s{\syl\syl\char}l{\char}y{\char} {\syl\syl\char}w{\char}a{\char}t{\char}c{\char}h{\syl\syl\char}i{\char}n{\char}g{\char} {\syl\syl\char}h{\char}e{\char} {\syl\syl\char}w{\char}a{\char}i{\char}t{\char}s{\syl\syl\char} {\syl\syl\char}b{\char}y{\char} {\syl\syl\char}t{\char}h{\char}e{\char} {\syl\syl\char}w{\char}i{\char}n{\syl\syl\char}d{\char}o{\char}w
+fx: {\Tpl\line\syl\syl\char}A{\char}n{\char}d{\char} {\syl\syl\char}w{\char}o{\char}n{\syl\syl\char}d{\char}e{\char}r{\char}s{\syl\syl\char} {\syl\syl\char}a{\char}t{\char} {\syl\syl\char}t{\char}h{\char}e{\char} {\syl\syl\char}e{\char}m{\char}p{\syl\syl\char}t{\char}y{\char} {\syl\syl\char}p{\char}l{\char}a{\char}c{\char}e{\char} {\syl\syl\char}i{\char}n{\syl\syl\char}s{\char}i{\char}d{\char}e
+```
+
+Any number of `mixin` components can "apply to" any number of `template` components using my powerful conditional execution abilities, offering much more expressive templating than the "name association" system my predecessors offer.
+
+Mixins can be very useful for relying more heavily on `template line` than is strictly necessary, resulting in cleaner output that's also easier to make sense of.
+For instance, combining `template line` with `mixin syl` allows for simple (read: doesn't involve movement or horizontal scaling) in-line highlight effects, and `mixin char` can be useful for by-character gradients.
+
+`mixin line` is an interesting construct. It goes once at the start of each generated line, like the text of the template it's modifying, but as a mixin, it works as a modifier. Even if the template's class is `syl`, `word`, or `char`, it's appropriate to use `mixin line` to modify the entire line *of output*.
+Combined with conditional execution, this behavior becomes quite useful:
+- If a song style is largely uniform, but uses different colors for different groups of lines (perhaps color-coded by character), then the common tags could be put in a `template` that applies to the entire song, and there would be a `mixin` for each color, with each one set to execute when appropriate (likely using the `actor` modifier). This could also have been achieved using a sort of lookup table, but it makes for a decent example, and this approach actually scales better when space-separated actors get involved.
+- If a style has some complicated stuff going on, but there is, for instance, a variety of templates meant to generate a consistent "glow" effect, then a mixin can be used to apply the same tags to all "glow" output without adding them to each template.
+
+### Modifiers
+In addition to its variety and class, a component's effect field may contain any amount of *modifiers*, which alter its behavior.
+Most modifiers allow you to specify the conditions under which a component will execute based on the input or whatever else.
+
+If there are multiple different restrictions on a component, then all restrictions must be satisfied for the component to execute.
+For example, a mixin with the modifiers `actor foo actor bar layer 3 layer 5` will execute only for lines where ((actor contains `foo` OR actor contains `bar`) AND (layer is 3 or layer is 5)).
+
+#### `style`
+By default, a component is "interested" in only the style that it is itself set to. This means that it will only execute for input with a matching style.
+Adding `style foo` to a template will add `foo` to its set of interested styles, so it will also execute for input with the `foo` style.
+Styles with names containing spaces do not currently work with this modifier.
+
+#### `anystyle`
+Similar to the `all` modifier in other templaters, this makes the component interested in any style.
+
+#### `actor`
+Similar to the `style` modifier, each component has a set of actor values it is "interested" in.
+By default, this set is empty and a component won't look at the input's actor field.
+Adding one or more `actor` modifiers will restrict the component to only execute for input with matching actor values.
+
+**Advanced usage:** The actor field of a line of input is treated not as a single string, but as a space-separated list of actors, so input can be given multiple "actor values" that the component tries to match.
+If there is any intersection between the component's set of interested actors and the input's set of actor values, the component will execute.
+
+#### `t_actor`
+Only valid for `mixin`s.
+Very similar to `actor`, but checks the base template the mixin is running on, not the input.
+For instance, if one has a `template` that generates particle effects, one could put `tri` in the its actor field and then use `t_actor tri` to make mixins that only apply to the particles.
+
+#### `layer`
+Only valid for `mixin`s.
+
+Again, similar to `style` or `actor`.
+By default, a mixin is "interested" in all layers.
+This modifier restricts a mixin to only execute for certain layer numbers.
+
+When checking the layer number, I look at *the output's current layer number*.
+Output lines start with a layer number copied from the `template` that generated them, but the number may be changed by functions such as `relayer`.
+If the layer is changed, I will "see" the updated layer number, not the original.
+
+#### `if` and `unless`
+Easily the most powerful conditional execution modifier.
+Given an `if foo` modifier, I will look in the Lua environment for a value named `foo`.
+When checking whether to execute, I will expect this value to be either a boolean or a function.
+If it is a function, I will call it (with no arguments) and look at the result.
+Either way, I then use this value as a predicate to decide whether the component should execute.
+
+For `if`, the component will only execute if the value is `true`.
+For `unless`, the component will only execute if the value is `false`.
+
+At the moment, only one instance of the `if`/`unless` modifier is allowed per component.
+
+`cond` is an alias for `if`.
+
+#### `noblank`
+Primarily useful for `syl` and `char` components.
+Similar to the `noblank` modifier in other templaters, but also works for mixins.
+This is useful for, say, preventing a `mixin char` component from pointlessly executing for spaces.
+
+Components with the `noblank` modifier will not execute for input whose text is empty or consists entirely of whitespace.
+
+#### `keepspace`
+Only valid for `template`s. Primarily useful for `syl` and `word`.
+By default, just before inserting a line of output into the file, I will trim away any trailing whitespace.
+The `keepspace` modifier will disable this behavior for an individual template, should this be desirable for any reason.
+
+#### `nomerge`
+Only valid for `template`s.
+By default, just before removing trailing whitespace, I will merge consecutive tag blocks from output lines.
+This is done using the primitive technique of naively removing `}{` from the output, so if this breaks anything fancy, the `nomerge` modifier can disable this behavior.
+
+#### `loop`
+Only valid for `code` and `template`s.
+
+Other templaters support giving a component a `loop n` modifier, where `n` is some integer.
+The component will then execute *n* times, and the current/maximum loop values can be accessed using `j` and `maxj`, available as both Lua variables and inline variables.
+Loops follow the convention set by Lua, starting at 1 and including the maximum.
+
+My implementation of this feature is similar, but I require specifying a name in the modifier, e.g., `loop foo 3`.
+The loop count is then accessed using `$loop_foo` or, in Lua, `loopctx.state.foo`, and the maximum with `$maxloop_foo` or `loopctx.max.foo`.
+This allows for multiple independent named loops on a single component.
+Having multiple loops can be very powerful for anything that requires more than one "axis" of repetition, like combining some frame-by-frame effect with a vertical gradient.
+
+If there are multiple loops, they will be nested in *reverse* order of appearance.
+As a demonstration of this, consider the following example:
+```
+template line loop foo 2 loop bar 3: {\foo$loop_foo\bar$loop_bar}
+kara: {\k86}Word{\k41}less{\k24}ly {\k87}watch{\k33}ing {\k63}he {\k66}waits{\k25} {\k44}by {\k22}the {\k57}win{\k69}dow
+kara: {\k36}And {\k38}won{\k59}ders{\k43} {\k25}at {\k16}the {\k40}emp{\k21}ty {\k49}place {\k26}in{\k85}side
+fx: {\foo1\bar1}Wordlessly watching he waits by the window
+fx: {\foo2\bar1}Wordlessly watching he waits by the window
+fx: {\foo1\bar2}Wordlessly watching he waits by the window
+fx: {\foo2\bar2}Wordlessly watching he waits by the window
+fx: {\foo1\bar3}Wordlessly watching he waits by the window
+fx: {\foo2\bar3}Wordlessly watching he waits by the window
+fx: {\foo1\bar1}And wonders at the empty place inside
+fx: {\foo2\bar1}And wonders at the empty place inside
+fx: {\foo1\bar2}And wonders at the empty place inside
+fx: {\foo2\bar2}And wonders at the empty place inside
+fx: {\foo1\bar3}And wonders at the empty place inside
+fx: {\foo2\bar3}And wonders at the empty place inside
+```
+See the `incr` method in the `loopctx` class for how this works.
+Basically, usage of `loop` is treated as one big loop, where the iteration step entails treating the loopctx as a little-endian list of digits:
+* Increment the first loop variable.
+* If this exceeds that loop's maximum value, step forward by resetting it to 1 and incrementing the *next* loop variable.
+* If the next loop variable also overflowed, step forward again in the same fashion. Repeat.
+* If the final loop variable has overflowed, the "big loop" is done.
+
+`repeat` is an alias for `loop`.
+
+#### `prefix`
+Only valid for `mixin`s.
+Mixin output is usually placed just before the text it's associated with.
+The `prefix` modifier instead places it at the beginning of the line, alongside the base `template` text and any `mixin line` components.
+Consider the following example, where one `mixin word` component has the `prefix` modifier and one does not: ```
+template line: {\X}
+mixin word prefix: {\Y(!word.text!)}
+mixin word: {\Z}
+kara: {\k86}Word{\k41}less{\k24}ly {\k87}watch{\k33}ing {\k63}he {\k66}waits{\k25} {\k44}by {\k22}the {\k57}win{\k69}dow
+kara: {\k36}And {\k38}won{\k59}ders{\k43} {\k25}at {\k16}the {\k40}emp{\k21}ty {\k49}place {\k26}in{\k85}side
+fx: {\X\Y(Wordlessly )\Y(watching )\Y(he )\Y(waits )\Y(by )\Y(the )\Y(window)\Z}Wordlessly {\Z}watching {\Z}he {\Z}waits {\Z}by {\Z}the {\Z}window
+fx: {\X\Y(And )\Y(wonders )\Y(at )\Y(the )\Y(empty )\Y(place )\Y(inside)\Z}And {\Z}wonders {\Z}at {\Z}the {\Z}empty {\Z}place {\Z}inside
+```
+
+This may seem like a fairly strange modifier to offer, but I have it for a fairly specific purpose: using `\t(\clip)` in a `mixin syl prefix` component to produce behavior similar to `\kf`, but with all the flexibility of clips and transforms.
+
+# Template Execution
+TODO
+
+## Execution Order/Semantics
+```
+for each line of input:
+	(try to) run all `code line` components
+	(try to) run all `template line` components
+
+	for each word in the line:
+		(try to) run all `code word` components
+		(try to) run all `template word` components
+
+	do the above for syls and chars
+```
+(mixins happen as part of "running a template")
+see the `apply_templates` function for more details
+
+TODO
+
+## Inline Expressions
+In a template's text, Lua expressions enclosed in a pair of `!` will be evaluated, with their result placed in the output text, like `!word.text!` as used in an earlier example.
+Unlike other templaters, I will treat `nil` as an empty string here, which is convenient when writing functions like `retime` and `relayer` that are useful for their side effects.
+
+TODO
+
+## Inline Variables
+In a template's text, a dollar sign followed by certain names will be expanded to certain values, like `$loop_foo` as used in an earlier example.
+I don't support all the same loop variables as other templaters, and the ones I do support will often have different names.
+More loop variables will probably be added in the future, since they're so convenient.
+For now, see the `eval_inline_var` function for what I currently offer.
+
+TODO