Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Wilt Chamberlain] Support ref and source in Operations #1507

Closed
drewbanin opened this issue Jun 5, 2019 · 2 comments · Fixed by #1526
Closed

[Wilt Chamberlain] Support ref and source in Operations #1507

drewbanin opened this issue Jun 5, 2019 · 2 comments · Fixed by #1526

Comments

@drewbanin
Copy link
Contributor

Issue

Issue description

Operations do not currently support ref or source method calls.

An exception is raised when ref is used:

2019-06-05 16:47:27,534 (MainThread): Encountered an error while running operation: Compilation Error in macro say_hi (macros/my-op.sql)
  argument of type 'NoneType' is not iterable
2019-06-05 16:47:27,534 (MainThread): 
Traceback (most recent call last):
  File "/Users/drew/fishtown/dbt/core/dbt/clients/jinja.py", line 104, in call
    return macro(*args, **kwargs)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/runtime.py", line 575, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 110, in _invoke
    return original_invoke(self, arguments, autoescape)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/runtime.py", line 579, in _invoke
    rv = self._func(*arguments)
  File "dbt-a67bac59d05187e871611b73", line 41, in macro
    to_string(environment.call(context, (undefined(name='ref') if l_2_ref is missing else l_2_ref), 'idk')),
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/sandbox.py", line 438, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/runtime.py", line 262, in call
    return __obj(*args, **kwargs)
  File "/Users/drew/fishtown/dbt/core/dbt/context/runtime.py", line 44, in do_ref
    if target_model_id not in model.get('depends_on', {}).get('nodes'):
TypeError: argument of type 'NoneType' is not iterable

An exception is also raised when source() is used:

2019-06-05 16:48:52,545 (MainThread): Encountered an uncaught exception while running operation: 'ParsedMacro' object has no attribute 'sources'
2019-06-05 16:48:52,545 (MainThread): 
Traceback (most recent call last):
  File "/Users/drew/fishtown/dbt/core/dbt/task/run_operation.py", line 44, in run
    result = self._run_unsafe()
  File "/Users/drew/fishtown/dbt/core/dbt/task/run_operation.py", line 37, in _run_unsafe
    manifest=manifest
  File "/Users/drew/fishtown/dbt/core/dbt/adapters/base/impl.py", line 894, in execute_macro
    result = macro_function(**kwargs)
  File "/Users/drew/fishtown/dbt/core/dbt/clients/jinja.py", line 104, in call
    return macro(*args, **kwargs)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/runtime.py", line 575, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 110, in _invoke
    return original_invoke(self, arguments, autoescape)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/runtime.py", line 579, in _invoke
    rv = self._func(*arguments)
  File "dbt-d96473c053fb049fa5d01868", line 41, in macro
    to_string(environment.call(context, (undefined(name='source') if l_2_source is missing else l_2_source), 'idk', 'id')),
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/sandbox.py", line 438, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/drew/fishtown/dbt/env/lib/python3.7/site-packages/jinja2/runtime.py", line 262, in call
    return __obj(*args, **kwargs)
  File "/Users/drew/fishtown/dbt/core/dbt/context/runtime.py", line 81, in do_source
    model.sources.append([source_name, table_name])
  File "/Users/drew/fishtown/dbt/core/dbt/api/object.py", line 125, in __getattr__
    ).format(type(self).__name__, name))
AttributeError: 'ParsedMacro' object has no attribute 'sources'

Results

dbt raised an exception, whereas I expected dbt to interpolate model/source relation representations.

System information

The output of dbt --version:

dev/wilt-chamberlain (ddb1785698fc920fb681a240376ae8cf3c27c78)

The operating system you're running on: osx

The python version you're using (probably the output of python --version) 3.7

Steps to reproduce

Create a model and a source:
models/my_source.yml

version: 2
sources:
  - name: my_source
    tables:
      - name: my_table

models/my_model.sql

select 1 as id

Create and run an operation

{% macro my_model_op() %}
{{ ref('my_model') }}
{% endmacro %}

{% macro my_source_op() %}
{{ source('my_source', 'my_table') }}
{% endmacro %}
$ dbt run-operation --macro my_model_op
$ dbt run-operation --macro my_source_op
@drewbanin drewbanin added this to the Wilt Chamberlain milestone Jun 5, 2019
@drewbanin
Copy link
Contributor Author

I am interested in supporting string interpolation of ref and source function calls -- it is less important that these functions actually build edges -- these Operations will not show up in the docs, and we do not need to invoke Operations that depend on a ref/source, for instance.

@drewbanin
Copy link
Contributor Author

@beckjake can you have a think about how feasible this is?

beckjake added a commit that referenced this issue Jun 13, 2019
…ations

Add ref and source support to run-operation macros (#1507)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant