Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2889] Macros executed via run-operation cannot ref private models #8248

Open
2 tasks done
Tracked by #10125 ...
nicholasyager opened this issue Jul 28, 2023 · 5 comments
Open
2 tasks done
Tracked by #10125 ...
Labels
enhancement New feature or request help_wanted Trickier changes, with a clear starting point, good for previous/experienced contributors model_groups_access Issues related to groups multi_project

Comments

@nicholasyager
Copy link
Contributor

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Suppose you have a model (order.sql) setup in a group (example) with access set to private. If you execute a macro that refs this private model, dbt will abort with an AttributeError: 'Macro' object has no attribute 'group' exception.

19:22:02  Traceback (most recent call last):
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/adapters/snowflake/connections.py", line 255, in exception_handler
    yield
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/adapters/base/impl.py", line 1043, in execute_macro
    result = macro_function(**kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 35, in macro
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/context/providers.py", line 259, in __call__
    return self.resolve(name, package, version)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/context/providers.py", line 508, in resolve
    if not self.model.group or self.model.group != target_model.group:
AttributeError: 'Macro' object has no attribute 'group'

This issue seems to originate in the resolve method in RuntimeRefResolver. Specifically, this chunk of logic:

        elif (
            target_model.resource_type == NodeType.Model
            and target_model.access == AccessType.Private
            # don't raise this reference error for ad hoc 'preview' queries
            and self.model.resource_type != NodeType.SqlOperation
            and self.model.resource_type != NodeType.RPCCall  # TODO: rm
        ):
            if not self.model.group or self.model.group != target_model.group:
                raise DbtReferenceError(
                    unique_id=self.model.unique_id,
                    ref_unique_id=target_model.unique_id,
                    group=cast_to_str(target_model.group),
                )

When running a macro via run-operation, target_model is not a ParsedNode, but rather a Macro, which do not have a concept of a group. Hence, the AttributeError.

Expected Behavior

Given that macros are not part of the group and access abstraction, I would expect macros executed within a run-operation context to be able to ref any model in a dbt project without verifying group membership.

Steps To Reproduce

  1. Create a model, and configure it to be in a group.
  2. Create a macro that ref's the grouped model.
  3. Attempt to execute the macro via run-operation.

Relevant log output

See above in current behavior

Environment

- OS: MacOS 13.5
- Python: 3.9.6
- dbt: 1.5.4

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

@nicholasyager nicholasyager added bug Something isn't working triage labels Jul 28, 2023
@github-actions github-actions bot changed the title [Bug] Macros executed via run-operation cannot ref private models [CT-2889] [Bug] Macros executed via run-operation cannot ref private models Jul 28, 2023
@jtcohen6 jtcohen6 added the model_groups_access Issues related to groups label Aug 1, 2023
@jtcohen6
Copy link
Contributor

jtcohen6 commented Aug 2, 2023

@nicholasyager Thanks for opening! :)

It does feel a bit weird that macros can reference private models, if called from within other models in the same group, but not if the same macro is called via run-operation.

I'm inclined to say that we should support this, same as how we added support for compilation/preview of --inline queries that reference private models:

I'll queue this up for the next milestone of multi-project work. If the fix is quite simple (and I think it is, right where you're pointing to!) we could backport for inclusion in a v1.6.x patch.

@graciegoheen
Copy link
Contributor

Notes from refinement:

  • macros need to all be loaded first - the groups won't have been loaded with the macros are loaded for parsing
  • this is during execution not parse time

@graciegoheen graciegoheen removed this from the v1.6.x milestone Feb 12, 2024
@b-per
Copy link
Contributor

b-per commented May 10, 2024

Hi team. Any update on when this might be updated/fixed?

The label says backport 1.7 but 1.8 is out now.

@jtcohen6
Copy link
Contributor

jtcohen6 commented May 10, 2024

@b-per I've just created a new epic for multi-project work over the next few months, and I'm including this on there. I see this lower priority than other bugs / QoL improvements that have been coming up more often, but definitely open to hearing that I should be prioritizing it more highly.

At this point, I'd be inclined to "roll forward" with this fix, and include it in the subsequent minor version.

I don't think this would be a very big lift, so I'd say it's also fair game for community contribution (by you or someone else) :)

@b-per
Copy link
Contributor

b-per commented May 10, 2024

Yes, my comment about seeing the backport 1.7 label was more whether it was still an issue considered for the future.
I agree that it doesn't look critical enough to backport it to previous versions.

Should we add the community label then?

And I'll check the new epic! 👀

@jtcohen6 jtcohen6 added the help_wanted Trickier changes, with a clear starting point, good for previous/experienced contributors label May 10, 2024
@graciegoheen graciegoheen added enhancement New feature or request and removed bug Something isn't working Medium Severity bug with minor impact that does not have resolution timeframe requirement labels May 14, 2024
@graciegoheen graciegoheen changed the title [CT-2889] [Bug] Macros executed via run-operation cannot ref private models [CT-2889] Macros executed via run-operation cannot ref private models May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help_wanted Trickier changes, with a clear starting point, good for previous/experienced contributors model_groups_access Issues related to groups multi_project
Projects
None yet
Development

No branches or pull requests

5 participants