Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/concepts/macros/macro_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ We describe SQLMesh's predefined variables below; user-defined macro variables a
## Predefined Variables
SQLMesh comes with predefined variables that can be used in your queries. They are automatically set by the SQLMesh runtime.

These variables are related to time and comprise a combination of prefixes (start, end, execution) and postfixes (date, ds, ts, epoch, millis).
These variables are mostly related to time and comprise a combination of prefixes (start, end, execution) and postfixes (date, ds, ts, epoch, millis).

SQLMesh uses the python [datetime module](https://docs.python.org/3/library/datetime.html) for handling dates and times. It uses the standard [Unix epoch](https://en.wikipedia.org/wiki/Unix_time) start of 1970-01-01. *All predefined variables with a time component use the [UTC time zone](https://en.wikipedia.org/wiki/Coordinated_Universal_Time).*

Expand Down Expand Up @@ -86,3 +86,10 @@ All predefined macro variables:
* @start_millis
* @end_millis
* @execution_millis

Other macro variables:

* @runtime_stage - A string value that denotes the current stage of the SQLMesh runtime. It can take one of the following values:
* 'loading' - The project is currently being loaded into SQLMesh's runtime context.
* 'creating' - The model tables are being created.
* 'evaluating' - The models' logic is being evaluated.
41 changes: 40 additions & 1 deletion docs/concepts/macros/sqlmesh_macros.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,10 +260,22 @@ It includes three elements:
2. A value to return if the condition is `TRUE`
3. A value to return if the condition is `FALSE` [optional]

These elements are specified as `@IF([logical condition], [value if TRUE], [value if FALSE])`.
These elements are specified as:

```sql linenums="1"
@IF([logical condition], [value if TRUE], [value if FALSE])
```

The value to return if the condition is `FALSE` is optional - if it is not provided and the condition is `FALSE`, then the macro has no effect on the resulting query.

It's also possible to use this macro in order to conditionally execute pre/post-statements:

```sql linenums="1"
@IF([logical condition], [statement to execute if TRUE]);
```

The `@IF` pre/post-statement itself must end with a semi-colon, but the inner statement argument must not.

The logical condition should be written *in SQL* and is evaluated with [SQLGlot's](https://github.com/tobymao/sqlglot) SQL engine. It supports the following operators:

- Equality: `=` for equals, `!=` or `<>` for not equals
Expand Down Expand Up @@ -318,6 +330,33 @@ SELECT
FROM table
```

Another example is conditionally executing a pre/post-statement depending on the current [runtime stage](./macro_variables.md#predefined-variables). For instance, the following `@IF` post-statement will only be executed at model evaluation time:

```sql linenums="1"
MODEL (
name sqlmesh_example.full_model,
kind FULL,
cron '@daily',
grain item_id,
audits [assert_positive_order_ids],
);

SELECT
item_id,
count(distinct id) AS num_orders,
FROM
sqlmesh_example.incremental_model
GROUP BY item_id
ORDER BY item_id;

@IF(
@runtime_stage = 'evaluating',
ALTER TABLE sqlmesh_example.full_model ALTER item_id TYPE VARCHAR
);
```

NOTE: we can also, say, alter the type of a column if the `@runtime_stage` is `'creating'`, but that will only have meaningful effects if the corresponding model is of an incremental kind, since `FULL` models are rebuilt on each evaluation and hence any changes made at their creation stage will be overwritten.

#### @EVAL

`@EVAL` evaluates its arguments with SQLGlot's SQL executor.
Expand Down
1 change: 1 addition & 0 deletions sqlmesh/core/renderer.py
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,7 @@ def render(
snapshots=snapshots,
table_mapping=table_mapping,
is_dev=is_dev,
runtime_stage=runtime_stage,
**kwargs,
)
except ParsetimeAdapterCallError:
Expand Down
2 changes: 1 addition & 1 deletion sqlmesh/core/snapshot/evaluator.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,14 +152,14 @@ def apply(query_or_df: QueryOrDF, index: int = 0) -> None:
end=end,
execution_time=execution_time,
has_intervals=bool(snapshot.intervals),
runtime_stage=RuntimeStage.EVALUATING,
**kwargs,
)

render_statements_kwargs = dict(
engine_adapter=self.adapter,
snapshots=snapshots,
is_dev=is_dev,
runtime_stage=RuntimeStage.EVALUATING,
**common_render_kwargs,
)

Expand Down
12 changes: 11 additions & 1 deletion tests/core/test_snapshot_evaluator.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ def increment_stage_counter(evaluator) -> None:
@increment_stage_counter();
@if(@runtime_stage = 'evaluating', ALTER TABLE test_schema.foo MODIFY COLUMN c SET MASKING POLICY p);

SELECT 1 AS a;
SELECT 1 AS a, @runtime_stage AS b;
"""
),
macros=macro.get_registry(),
Expand All @@ -196,6 +196,16 @@ def increment_stage_counter(evaluator) -> None:
[parse_one("ALTER TABLE test_schema.foo MODIFY COLUMN c SET MASKING POLICY p")]
)

assert snapshot.model.render_query().sql() == '''SELECT 1 AS "a", 'loading' AS "b"'''
assert (
snapshot.model.render_query(runtime_stage=RuntimeStage.CREATING).sql()
== '''SELECT 1 AS "a", 'creating' AS "b"'''
)
assert (
snapshot.model.render_query(runtime_stage=RuntimeStage.EVALUATING).sql()
== '''SELECT 1 AS "a", 'evaluating' AS "b"'''
)


def test_evaluate_paused_forward_only_upstream(mocker: MockerFixture, make_snapshot):
model = SqlModel(name="test_schema.test_model", query=parse_one("SELECT a, ds"))
Expand Down