Skip to content

Commit

Permalink
Fix compiler failing with nested with expression
Browse files Browse the repository at this point in the history
The previous implementation of `WithParser` used regex, which struggles
with parsing nested structures correctly. This commit improves
`WithParser` to track and parse all nested `with` expressions.

Other improvements:

- Throw meaningful errors when syntax is wrong. Replacing the prior
  behavior of silently ignoring such issues.
- Remove `I` prefix from related interfaces to align with newer code
  conventions.
- Add more unit tests for `with` expression.
- Improve documentation for templating.
- `ExpressionRegexBuilder`:
  - Use words `capture` and `match` correctly.
  - Fix minor issues revealed by new and improved tests:
     - Change regex for matching anything except surrounding
       whitespaces. The new regex ensures that it works even without
       having any preceeding text.
     - Change regex for capturing pipelines. The old regex was only
       matching (non-greedy) first character of the pipeline in tests,
       new regex matches the full pipeline.
- `ExpressionRegexBuilder.spec.ts`:
  - Ensure consistent way to define `describe` and `it` blocks.
  - Replace `expectRegex` tests, regex expectations test internal
    behavior of the class, not the external.
  - Simplified tests by eliminating the need for UUID suffixes/prefixes.
  • Loading branch information
undergroundwires committed Oct 25, 2023
1 parent dfd4451 commit 80821fc
Show file tree
Hide file tree
Showing 7 changed files with 958 additions and 403 deletions.
215 changes: 146 additions & 69 deletions docs/templating.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,79 +2,142 @@

## Benefits of templating

- Generating scripts by sharing code to increase best-practice usage and maintainability.
- Creating self-contained scripts without cross-dependencies.
- Use of pipes for writing cleaner code and letting pipes do dirty work.
- **Code sharing:** Share code across scripts for consistent practices and easier maintenance.
- **Script independence:** Generate self-contained scripts, eliminating the need for external code.
- **Cleaner code:** Use pipes for complex operations, resulting in more readable and streamlined code.

## Expressions

- Expressions start and end with mustaches (double brackets, `{{` and `}}`).
- E.g. `Hello {{ $name }} !`
- Syntax is close to [Go Templates ❤️](https://pkg.go.dev/text/template) but not the same.
- Functions enables usage of expressions.
- In script definition parts of a function, see [`Function`](./collection-files.md#Function).
- When doing a call as argument values, see [`FunctionCall`](./collection-files.md#Function).
- Expressions inside expressions (nested templates) are supported.
- An expression can output another expression that will also be compiled.
- E.g. following would compile first [with expression](#with), and then [parameter substitution](#parameter-substitution) in its output.

```go
{{ with $condition }}
echo {{ $text }}
{{ end }}
```
**Syntax:**

### Parameter substitution
Expressions are enclosed within `{{` and `}}`.
Example: `Hello {{ $name }}!`.
They are a core component of templating, enhancing scripts with dynamic capabilities and functionality.

**Syntax similarity:**

The syntax shares similarities with [Go Templates ❤️](https://pkg.go.dev/text/template), but with some differences:

**Function definitions:**

A simple function example:
You can use expressions in function definition.
Refer to [Function](./collection-files.md#function) for more details.

Example usage:

```yaml
function: EchoArgument
name: GreetFunction
parameters:
- name: 'argument'
code: Hello {{ $argument }} !
- name: name
code: Hello {{ $name }}!
```

It would print "Hello world" if it's called in a [script](./collection-files.md#script) as following:
If you assign `name` the value `world`, invoking `GreetFunction` would result in `Hello world!`.

**Function arguments:**

You can also use expressions in arguments in nested function calls.
Refer to [`Function | collection-files.md`](./collection-files.md#functioncall) for more details.

Example with nested function calls:

```yaml
script: Echo script
-
name: PrintMessageFunction
parameters:
- name: message
code: echo "{{ $message }}"
-
name: GreetUserFunction
parameters:
- name: userName
call:
function: EchoArgument
name: PrintMessageFunction
parameters:
argument: World
argument: 'Hello, {{ $userName }}!'
```

Here, if `userName` is `Alice`, invoking `GreetUserFunction` would execute `echo "Hello, Alice!"`.

**Nested templates:**

You can nest expressions inside expressions (also called "nested templates").
This means that an expression can output another expression where compiler will compile both.

For example, following would compile first [with expression](#with), and then [parameter substitution](#parameter-substitution) in its output:

```go
{{ with $condition }}
echo {{ $text }}
{{ end }}
```

A function can call other functions such as:
### Parameter substitution

Parameter substitution dynamically replaces variable references with their corresponding values in the script.

**Example function:**

```yaml
-
function: CallerFunction
parameters:
- name: 'value'
call:
function: EchoArgument
parameters:
argument: {{ $value }}
-
function: EchoArgument
parameters:
- name: 'argument'
code: Hello {{ $argument }} !
name: DisplayTextFunction
parameters:
- name: 'text'
code: echo {{ $text }}
```

Invoking `DisplayTextFunction` with `text` set to `"Hello, world!"` would result in `echo "Hello, World!"`.

### with

Skips its "block" if the variable is absent or empty. Its "block" is between `with` start (`{{ with .. }}`) and end (`{{ end }`}) expressions.
E.g. `{{ with $parameterName }} Hi, I'm a block! {{ end }}` would only output `Hi, I'm a block!` if `parameterName` has any value..
The `with` expression enables conditional rendering and provides a context variable for simpler code.

**Optional block rendering:**

If the provided variable is falsy (`false`, `null`, or empty), the compiler skips the enclosed block of code.
A "block" lies between the with start (`{{ with .. }}`) and end (`{{ end }}`) expressions, defining its boundaries.

It binds its context (value of the provided parameter value) as arbitrary `.` value. It allows you to use the argument value of the given parameter when it is provided and not empty such as:
Example:

```go
{{ with $optionalVariable }}
Hello
{{ end }}
```

This would display `Hello` if `$optionalVariable` is truthy.

**Parameter declaration:**

You should set `optional: true` for the argument if you use it like `{{ with $argument }} .. {{ end }}`.

Declare parameters used for `with` condition as optional such as:

```yaml
name: ConditionalOutputFunction
parameters:
- name: 'data'
optional: true
code: |-
{{ with $data }}
Data is: {{ . }}
{{ end }}
```

**Context variable:**

`with` statement binds its context (value of the provided parameter value) as arbitrary `.` value.
`{{ . }}` syntax gives you access to the context variable.
This is optional to use, and not required to use `with` expressions.

For example:

```go
{{ with $parameterName }}Parameter value is {{ . }} here {{ end }}
```

It supports multiline text inside the block. You can have something like:
**Multiline text:**

It supports multiline text inside the block. You can write something like:

```go
{{ with $argument }}
Expand All @@ -83,40 +146,54 @@ It supports multiline text inside the block. You can have something like:
{{ end }}
```

You can also use other expressions inside its block, such as [parameter substitution](#parameter-substitution):
**Inner expressions:**

You can also embed other expressions inside its block, such as [parameter substitution](#parameter-substitution):

```go
{{ with $condition }}
This is a different parameter: {{ $text }}
{{ end }}
```

💡 Declare parameters used for `with` condition as optional. Set `optional: true` for the argument if you use it like `{{ with $argument }} .. {{ end }}`.

Example:
This also includes nesting `with` statements:

```yaml
function: FunctionThatOutputsConditionally
parameters:
- name: 'argument'
optional: true
code: |-
{{ with $argument }}
Value is: {{ . }}
```go
{{ with $condition1 }}
Value of $condition1: {{ . }}
{{ with $condition2 }}
Value of $condition2: {{ . }}
{{ end }}
{{ end }}
```

### Pipes

- Pipes are functions available for handling text.
- Allows stacking actions one after another also known as "chaining".
- Like [Unix pipelines](https://en.wikipedia.org/wiki/Pipeline_(Unix)), the concept is simple: each pipeline's output becomes the input of the following pipe.
- You cannot create pipes. [A dedicated compiler](./application.md#parsing-and-compiling) provides pre-defined pipes to consume in collection files.
- You can combine pipes with other expressions such as [parameter substitution](#parameter-substitution) and [with](#with) syntax.
- ❗ Pipe names must be camelCase without any space or special characters.
- **Existing pipes**
- `inlinePowerShell`: Converts a multi-lined PowerShell script to a single line.
- `escapeDoubleQuotes`: Escapes `"` characters, allows you to use them inside double quotes (`"`).
- **Example usages**
- `{{ with $code }} echo "{{ . | inlinePowerShell }}" {{ end }}`
- `{{ with $code }} echo "{{ . | inlinePowerShell | escapeDoubleQuotes }}" {{ end }}`
Pipes are functions designed for text manipulation.
They allow for a sequential application of operations resembling [Unix pipelines](https://en.wikipedia.org/wiki/Pipeline_(Unix)), also known as "chaining".
Each pipeline's output becomes the input of the following pipe.

**Pre-defined**:

Pipes are pre-defined by the system.
You cannot create pipes in [collection files](./collection-files.md).
[A dedicated compiler](./application.md#parsing-and-compiling) provides pre-defined pipes to consume in collection files.

**Compatibility:**

You can combine pipes with other expressions such as [parameter substitution](#parameter-substitution) and [with](#with) syntax.

For example:

```go
{{ with $script }} echo "{{ . | inlinePowerShell | escapeDoubleQuotes }}" {{ end }}
```

**Naming:**

❗ Pipe names must be camelCase without any space or special characters.

**Available pipes:**

- `inlinePowerShell`: Converts a multi-lined PowerShell script to a single line.
- `escapeDoubleQuotes`: Escapes `"` characters for batch command execution, allows you to use them inside double quotes (`"`).
Original file line number Diff line number Diff line change
Expand Up @@ -14,45 +14,44 @@ export class ExpressionRegexBuilder {
.addRawRegex('\\s+');
}

public matchPipeline() {
public captureOptionalPipeline() {
return this
.expectZeroOrMoreWhitespaces()
.addRawRegex('(\\|\\s*.+?)?');
.addRawRegex('((?:\\|\\s*\\b[a-zA-Z]+\\b\\s*)*)');
}

public matchUntilFirstWhitespace() {
public captureUntilWhitespaceOrPipe() {
return this
.addRawRegex('([^|\\s]+)');
}

public matchMultilineAnythingExceptSurroundingWhitespaces() {
public captureMultilineAnythingExceptSurroundingWhitespaces() {
return this
.expectZeroOrMoreWhitespaces()
.addRawRegex('([\\S\\s]+?)')
.expectZeroOrMoreWhitespaces();
.expectOptionalWhitespaces()
.addRawRegex('([\\s\\S]*\\S)')
.expectOptionalWhitespaces();
}

public expectExpressionStart() {
return this
.expectCharacters('{{')
.expectZeroOrMoreWhitespaces();
.expectOptionalWhitespaces();
}

public expectExpressionEnd() {
return this
.expectZeroOrMoreWhitespaces()
.expectOptionalWhitespaces()
.expectCharacters('}}');
}

public buildRegExp(): RegExp {
return new RegExp(this.parts.join(''), 'g');
}

private expectZeroOrMoreWhitespaces() {
public expectOptionalWhitespaces() {
return this
.addRawRegex('\\s*');
}

public buildRegExp(): RegExp {
return new RegExp(this.parts.join(''), 'g');
}

private addRawRegex(regex: string) {
this.parts.push(regex);
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ export class ParameterSubstitutionParser extends RegexParser {
protected readonly regex = new ExpressionRegexBuilder()
.expectExpressionStart()
.expectCharacters('$')
.matchUntilFirstWhitespace() // First match: Parameter name
.matchPipeline() // Second match: Pipeline
.captureUntilWhitespaceOrPipe() // First capture: Parameter name
.expectOptionalWhitespaces()
.captureOptionalPipeline() // Second capture: Pipeline
.expectExpressionEnd()
.buildRegExp();

Expand Down
Loading

0 comments on commit 80821fc

Please sign in to comment.