docs generation with error in schema.yml gives unhelpful error message

[elgabito]

Describe the bug

When running dbt docs generate I would get the following output:

Encountered an error while generating catalog: Runtime Error
  name 'dbt' is not defined
dbt encountered 1 failure while writing the catalog

This is very difficult to troubleshoot! After hours of digging I realized I simply had an error in my schema.yml

Steps To Reproduce

Take any working model and edit the schema.yml - change the database to something that doesn't exist.
Execute dbt docs generate

Expected behavior

An error should clearly happen because it's incorrect, but a more helpful error message would be great!

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: Oracle)

The output of dbt --version:

installed version: 0.19.1
   latest version: 0.19.1

Up to date!

Plugins:
  - oracle: 0.19.1

The operating system you're using:
Windows 10

The output of python --version:
Python 3.8.0

Additional context

Add any other context about the problem here.

[bvancil]

Does this happen for any model or just when specifying sources?

[jtcohen6]

@elgabito Thanks for opening the issue. I think I have a sense of what's going on here, though I do have a few clarifying questions:

• Are you defining models or sources in the schema.yml file?
• Is dbt (from the error message) the name of a database that doesn't exist?

If you're using this dbt-oracle plugin, I see this note in oracle__get_catalog macro:

   {#
      If the user has multiple databases set and the first one is wrong, this will fail.
      But we won't fail in the case where there are multiple quoting-difference-only dbs, which is better.
    #}

So it's a known limitation that an incorrect database causes an error when generating the catalog. I also agree with your assessment that this should raise an error. As far as raising a more helpful error, there I'm less sure how to proceed.

As in so many things in dbt, we can only do so much before actually running a query against the database. Oftentimes your best bet is targeted debugging:

• check out logs/dbt.log
• find the exact SQL that dbt ran right before encountering the error
• try to run it yourself to reproduce the error
• inspect the SQL
• comment out lines and re-run to isolate which line is causing the error

There is another approach we could take: treat catalog generation more like relational cache construction, which has handling for missing objects. For instance, Snowflake's list_relations_without_caching tries to run show terse objects in all databases of interest; if a database is missing, it handles the resulting error and keeps on moving:

https://github.com/fishtown-analytics/dbt/blob/c712c96a0b0c03389cb8769ea4a1c437076d1d06/plugins/snowflake/dbt/adapters/snowflake/impl.py#L135-L141

In this case, the goal isn't to move past the error uninterrupted, but rather to handle the vague error and raise a more descriptive one instead, since name ... is not defined is specifically unhelpful. Not all adapters have this problem with catalog generation, or error messages this vague. So one possible move could be to reimplement _get_one_catalog in the Oracle adapter to catch exceptions like name ... is not defined and instead raise something like, Database ... not found.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.