[CT-159] [Bug] Partial parsing fails to find new doc block after git pull

[jaypeedevlin]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

After pulling changes from a remote git branch that include new docblocks, I get an error on compile/run:

Running with dbt=0.20.2
Partial parsing enabled: 0 files deleted, 15 files added, 13 files changed.
Encountered an error:
Compilation Error in model source_facebook_ads__videos (models/sources/facebook_ads/source_facebook_ads__videos.sql)
  Documentation for 'model.project_name.source_facebook_ads__videos' depends on doc 'facebook_ads__campaign_id'  which was not found

If I run the same command with --no-partial parse, the error disappears and subsequent runs are also fine.

Expected Behavior

Being able to run without disabling partial parse

Steps To Reproduce

I don't know if these are exact repro steps, but both time I've encountered this it's been:

1. A colleague makes a change on an existing branch that includes doc block changes
2. I pull that branch locally from remote
3. When I try and dbt compile I hit the error.

Relevant log output

2021-10-27 16:02:54.206833 (MainThread): Traceback (most recent call last):
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/task/runnable.py", line 406, in run
    self._runtime_initialize()
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/task/runnable.py", line 122, in _runtime_initialize
    super()._runtime_initialize()
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/task/runnable.py", line 79, in _runtime_initialize
    self.load_manifest()
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/task/runnable.py", line 66, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/parser/manifest.py", line 182, in get_full_manifest
    manifest = loader.load()
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/parser/manifest.py", line 353, in load
    self.process_docs(self.root_project)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/parser/manifest.py", line 732, in process_docs
    _process_docs_for_node(ctx, node)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/parser/manifest.py", line 916, in _process_docs_for_node
    column.description = get_rendered(column.description, context)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/clients/jinja.py", line 598, in get_rendered
    return render_template(template, ctx, node)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/clients/jinja.py", line 549, in render_template
    return template.render(ctx)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/jinja2/sandbox.py", line 462, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/context/docs.py", line 73, in doc
    doc_target_not_found(self.node, doc_name, doc_package_name)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/exceptions.py", line 524, in doc_target_not_found
    raise_compiler_error(msg, model)
  File "/Users/josh.devlin/.dbt/versions/0.20.2/lib/python3.8/site-packages/dbt/exceptions.py", line 444, in raise_compiler_error
    raise CompilationException(msg, node)
dbt.exceptions.CompilationException: Compilation Error in model source_facebook_ads__conversions (models/sources/facebook_ads/source_facebook_ads__conversions.sql)
  Documentation for 'model.project-name.source_facebook_ads__conversions' depends on doc 'facebook_ads__campaign_id'  which was not found

Environment

- OS: Mac OS 11.6
- Python: 3.8.10
- dbt: 0.20.2

What database are you using dbt with?

snowflake

Additional Context

No response

[jtcohen6]

Thanks for the report @jaypeedevlin! I'll see if I can pull together a simple reproduction case.

It seems like we you all the way through parsing successfully, but dbt missed adding a docs block to the manifest, such that you trip doc_target_not_found later on. Unfortunately, the way we implemented exception-catching for partial parsing in v0.20.2 won't be able to help there.

Just wondering, are you able to tell if the model + doc block in question in totally new files, or edited files? (I see the branch included 15 files added, 13 files changed)

[saraleon1]

+1 dbt Cloud user

Context of how this came up:

I had to make a GIT operations outside of dbt, so I ended up pulling the revert of the revert
This resulted in the following compilation message:

"Compilation Error in model ()
Documentation for 'model.' depends on doc '' which was not found"

The doc file was nicely in place but I suspect that the incremental parsing was causing it to not be seen.
I have just added a space and saved the file to fix the issue

[NicolasPA]

I also encountered this error with DBT 1.0.4 after a simple branch change.

I was working on my feature branch where I added a file docs_blocks.md with docs block my_doc. dbt compile/run/docs were working fine with the documentation showing up as expected.

Then I came back to my main branch, which doesn't have the file with docs blocks, to execute some runs that also worked as expected.

But when I came back to my feature branch, the dbt compile failed with:

Compilation Error in model my_model (models\my_model.sql)
  Documentation for 'model.my_model' depends on doc 'my_doc'  which was not found

So it looks like DBT didn't notice that the branch change reintroduced the file docs_blocks.md, it didn't bother to add it in the compilation and thus the compilation of my_schema.yml which describes my_model failed.

My workaround was to add a small change in docs_blocks.md, then dbt compile worked as expected.

Environment:

• dbt 1.0.4
• Python 3.8.10
• mintty 3.5.2 (Git Bash for Windows)
• Windows 10 Pro

[alvare16]

@jtcohen6 hope this helps, although it seems like my steps to reproduce this are very similar to that of the original issue.

1. On a local git branch add a doc x for model y and define it as the models description within your models yml
2. Push branch to remote, create a pr and merge to master
3. Pull to local master branch and create a new branch to work on
4. Try running a model with dbt run --models
5. You should see the following error 'Documentation for model y depends on doc x which was not found'

[jaypeedevlin]

I just wanted to add in that I still see this issue come up every few weeks when I'm working across multiple branches.

[Yatinglin233]

I also want to add in that I'm still seeing this issue come up, hope DBT could help to fix this.

[jtcohen6]

Huzzah! Following @alvare16's instructions, I was able to reliably reproduce this error:

-- models/my_model.sql
select 1 as id

# my_model.yml
version: 2
models:
  - name: my_model
    columns:
      - name: id
        description: "{{ doc('whatever') }}"

<!--- my_model.md --->
{% docs whatever %}
  cool stuff
{% enddocs %}

Steps:

1. dbt parse (successful)
2. Remove the docs block + description
3. dbt parse (successful)
4. Add back the docs block + description
5. dbt parse - fails with

04:02:21  Encountered an error:
Compilation Error in model my_model (models/my_model.sql)
  Documentation for 'model.testy.my_model' depends on doc 'whatever'  which was not found

I've only been able to do a little spelunking on this. So far my hunch is that the documentation file ID seems to stick around in the partial parsing (previous save) manifest, even after the file is removed. This causes the file to be re-parsed in the manifest, with no contents, but its file ID is still tracked, so it isn't slated for addition when it's actually added back.