Wrappable Relation for custom rendering

[github-christophe-oudar]

Context

Relation is the powerful class that's used for multiple core functions as ref, source or this jinja functions.
Those functions can be overridden for a lot of custom behavior such as CI, sandboxing, staging or custom production environments.
One of the use cases that matters to me is CI and dev environments: in those environments, I want to LIMIT the input data for the model and I want to do so "automatically" so that developers don't have to think about it (and forget it).

Problem

Creating relations is possible through api.Relation.create function however as visible in the code:

@classmethod
    def create(
        cls: Type[Self],
        database: Optional[str] = None,
        schema: Optional[str] = None,
        identifier: Optional[str] = None,
        type: Optional[RelationType] = None,
        **kwargs,
    ) -> Self:
        kwargs.update(
            {
                "path": {
                    "database": database,
                    "schema": schema,
                    "identifier": identifier,
                },
                "type": type,
            }
        )
        return cls.from_dict(kwargs)

It doesn't provide a way to change the render method:

def render(self) -> str:
        # if there is nothing set, this will return the empty string.
        return ".".join(part for _, part in self._render_iterator() if part is not None)

Solution

To introduce a way to have that wrapping feature work, I'd suggest to have a new Optional value to a jinja function that takes the relation and expect a string output in the api.Relation.create function. That way we could then update the render method to execute the custom render if present or fallback on the usual one else.

Usage

Then following ref snippet would work (assuming render on rel is the original render):

{% macro ref(model_name, use_limit_ci = true, use_limit_dev = true, time_partitioning_table = false) %}

  {% set rel = builtins.ref(model_name) %}

  {%- if target.name == 'ci' and use_limit_ci -%}
    {% do return(api.Relation.create(rel.database, rel.schema, rel.identifier, limit_ci_render)) %}
  {%- elif target.name == 'dev' and use_limit_dev -%}
       {% do return(api.Relation.create(rel.database, rel.schema, rel.identifier, limit_dev_render)) %}
  {%- else -%}
    {% do return(rel) %}
  {%- endif -%}

{% endmacro %}

{% macro limit_ci_render(rel) %}
  {%- set selected_fields  = "*, _PARTITIONTIME, _PARTITIONDATE" if time_partitioning_table == True else "*" -%}
   {%- set render -%}
     (select {{ selected_fields }} from {{ rel.render() }} LIMIT 0)
   {%- endset -%}
   {% do return(render) %}
{% endmacro %}

{% macro limit_dev_render(rel) %}
  {%- set selected_fields  = "*, _PARTITIONTIME, _PARTITIONDATE" if time_partitioning_table == True else "*" -%}
   {%- set render -%}
     (select {{ selected_fields }} from {{ rel.render() }} LIMIT 100)
   {%- endset -%}
   {% do return(render) %}
{% endmacro %}

So that mydatabase.myschema.mytable is replaced in every query by

(SELECT * FROM `mydatabase`.`myschema`.`mytable` LIMIT 0)

Alternative considered

I think it exists some alternative such as inferring a new "intermediate" ephemeral model that would have the same behavior but I'm not sure it would fit the dbt architecture as I don't think we want to create model dynamically as discussed in #5101 but maybe it could be revisited?