load_macros_from_path config causes SQLFluff to exit with an error

[modernplumber]

Expected Behaviour

The program should load macros from the defined path and continue execution.

Observed Behaviour

With the load_macros_from_path config present in .sqlfluff file no command of SQLFluff runs. It raises the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/bin/sqlfluff", line 8, in <module>
    sys.exit(cli())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/cli/commands.py", line 344, in lint
    result = lnt.lint_paths(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/linter.py", line 820, in lint_paths
    self.lint_path(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/linter.py", line 794, in lint_path
    for linted_file in runner.run(fnames, fix):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/runner.py", line 92, in run
    for fname, partial in self.iter_partials(fnames, fix=fix):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/runner.py", line 45, in iter_partials
    for fname, rendered in self.iter_rendered(fnames):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/runner.py", line 38, in iter_rendered
    yield fname, self.linter.render_file(fname, self.config)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/linter.py", line 557, in render_file
    return self.render_string(raw_file, fname, config)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/linter/linter.py", line 534, in render_string
    templated_file, templater_violations = self.templater.process(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/templaters/jinja.py", line 227, in process
    self._extract_macros_from_path(macros_path, env=env, ctx=live_context)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/templaters/jinja.py", line 76, in _extract_macros_from_path
    cls._extract_macros_from_path(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/templaters/jinja.py", line 68, in _extract_macros_from_path
    cls._extract_macros_from_template(template, env=env, ctx=ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sqlfluff/core/templaters/jinja.py", line 46, in _extract_macros_from_template
    for k in macro_template.module.__dict__:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 1189, in module
    return self._get_default_module()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 1172, in _get_default_module
    self._module = rv = self.make_module()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 1155, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/environment.py", line 1238, in __init__
    body_stream = list(template.root_render_func(context))
  File "<template>", line 12, in root
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/sandbox.py", line 407, in getattr
    value = getattr(obj, attribute)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/runtime.py", line 753, in __getattr__
    return self._fail_with_undefined_error()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2/runtime.py", line 747, in _fail_with_undefined_error
    raise self._undefined_exception(self._undefined_message)
jinja2.exceptions.UndefinedError: 'project' is undefined

Steps to Reproduce

Add following configs to a repo that is a dbt project, and run sqlfluff lint

Version

sqlfluff, version 0.6.0
Python 3.9.5

Configuration

[sqlfluff]
templater = jinja
dialect = bigquery
exclude_rules = L016,L026,L028,L031,L034,L035,L044

# Reasons for excluding different rules
# L016: We don't yet have a preference for line length, we can test this later if we see a problem
# L026: Doesn't work with nested fields in the query
# L028: Doesn't work with nested fields in the query
# L031: Doesn't make sense when you have long CTE names, it's easier to read when you alias them
# L034: It's easier to make the surrogate key to be separate column and after that list other columns with a wildcard
# L035: Even though "ELSE NULL" would be redundant, it helps with readability and tells the reader that you are aware of the nulls.
# L044: It is enough that we limit the columns at some point in the DAG, but no need for all tables to require for specified amount of columns.

ignore_templated_areas = True

[sqlfluff:templater:jinja]
apply_dbt_builtins = True
load_macros_from_path=macros

[sqlfluff:templater:jinja:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable) %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}

[sqlfluff:rules]
tab_space_size = 4
# Line lenght currently not enforced
max_line_length = 100
indent_unit = space
comma_style = trailing
allow_scalar = True
single_table_references = consistent
unquoted_identifiers_policy = aliases

# Some rules have their own specific config.
[sqlfluff:rules:L003]
lint_templated_tokens = True

[sqlfluff:rules:L010]  # Keywords
capitalisation_policy = upper

[sqlfluff:rules:L014]  # Unquoted identifiers
extended_capitalisation_policy = lower

[sqlfluff:rules:L029]  # Keyword identifiers
unquoted_identifiers_policy = column_aliases

[sqlfluff:rules:L030]  # Function names
capitalisation_policy = upper

[sqlfluff:rules:L038]
select_clause_trailing_comma = forbid

[sqlfluff:rules:L040]  # Null & Boolean Literals
capitalisation_policy = upper

[sqlfluff:rules:L042]
# Enforce the use of CTEs over subqueries for readability
forbid_subquery_in = both

[sqlfluff:rules:L047]  # Consistent syntax to count all rows
prefer_count_1 = False

[barrywhart]

I'm not able to reproduce this error. I tested in SQLFluff 0.6.0, 0.6.5 and in the latest main (commit e02da762367b839f0dc5bb17c0c4373e84d862b9). Can you please test again and see if you're still having this issue, @kosti-hokkanen-supermetrics?

[barrywhart]

No response from OP. Closing.