dbt packages with expected dbt context generates an error

[github-christophe-oudar]

dbt projects imported using sqlmesh that contains packages such as elementary-data/elementary can't be used within sqlmesh because the package will expose models that are using dbt context.

sqlmesh error

Version: 0.71.0
Type: <class 'sqlmesh.utils.errors.ConfigError'>
Origin: API -> settings -> get_loaded_context
Could not render or parse jinja at '/Users/christophe.oudar/dev/projects/dbt-queries/dbt_packages/elementary/models/edr/run_results/job_run_results.sql'. 'dbt' is undefined

stacktrace

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/renderer.py", line 124, in _render
    rendered_expression = jinja_env.from_string(self._expression.name).render()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 2, in top-level template code
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/utils/jinja.py", line 512, in __call__
    return macro_callable(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 2, in template
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/utils/jinja.py", line 512, in __call__
    return macro_callable(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 2, in template
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/utils/jinja.py", line 512, in __call__
    return macro_callable(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 2, in template
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/utils/jinja.py", line 512, in __call__
    return macro_callable(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 2, in template
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/jinja2/environment.py", line 485, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'dbt' is undefined

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/web/server/settings.py", line 70, in get_loaded_context
    return await loop.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/web/server/settings.py", line 45, in _get_loaded_context
    context.load()
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/context.py", line 440, in load
    project = self._loader.load(self, update_schemas)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/dbt/loader.py", line 70, in load
    return super().load(context, update_schemas)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/loader.py", line 123, in load
    models = self._load_models(macros, jinja_macros)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/dbt/loader.py", line 112, in _load_models
    sqlmesh_model = cache.get_or_load_model(
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/dbt/loader.py", line 229, in get_or_load_model
    model = self._model_cache.get_or_load(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/model/cache.py", line 55, in get_or_load
    model=loaded_model, rendered_query=loaded_model.render_query(optimize=False)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/model/definition.py", line 950, in render_query
    query = self._query_renderer.render(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/renderer.py", line 407, in render
    expressions = super()._render(
                  ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/dbt-1.7/lib/python3.11/site-packages/sqlmesh/core/renderer.py", line 135, in _render
    raise ConfigError(

Failing model code

https://github.com/elementary-data/dbt-data-reliability/blob/master/models/edr/run_results/job_run_results.sql

solution leads

• the model is referencing elementary.edr_cast_as_timestamp("run_started_at") and run_started_at is related to dbt context. It's indeed already referenced as not supported in the sqlmesh documentation.

Then solutions could be the following:

• Have a placeholder value to avoid blocking those jinjas
• Add a configuration to disable the package from sqlmesh
• Fully support related dbt internals