Fix docs generation for cross-db sources in REDSHIFT RA3 node

[kostek-pl]

resolves #3236(partially)

This PR is related to #3236. It fixes documentation generation for cross-db objects used as SOURCES in models. However it not resolves full cross-db support as it's not possible at this moment due to AWS cross-db restrictions.

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: kostkaw.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[jtcohen6]

Thanks for the contribution @kostek-pl! I'm in favor of the overall concept. There's a little bit of reorganization needed in order to get the tests passing.

[jtcohen6]

Postgres credentials don't have an ra3_node property, so this is raising an error and breaking tests. I think the right move here is to reimplement _get_catalog_schemas within redshift/impl.py, including this RA3-specific logic.

[jtcohen6]

Just a naming nit: this change isn't so much about duplicates per se; it's about allowing multiple databases at all. In the past, the only reason we'd expect to see multiple databases on Postgres/Redshift was if a metadata query accidentally returned duplicates.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: kostkaw.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[kostek-pl]

Thank you very much @jtcohen6 for the review and tips.
I've updated code according to your suggestions, hope that it's done in the way you expected.

[jtcohen6]

Thanks @kostek-pl! The change here is looking good to me. Since we don't have an RA3 cluster available for automated tests, could you confirm that it resolves the issue for you locally? Namely, that you're able to generate docs that include sources from another database.

A number of tests were failing because they were missing secrets, on account of running from the kostek-pl CircleCI org instead of the fishtown-analytics CI org. I re-ran in our org, everything is passing except the linting step, which is just a handful of pep8 errors:

core/dbt/adapters/base/relation.py:436:53: E252 missing whitespace around parameter equals
core/dbt/adapters/base/relation.py:436:54: E252 missing whitespace around parameter equals
plugins/redshift/dbt/adapters/redshift/impl.py:70:1: W293 blank line contains whitespace
plugins/redshift/dbt/adapters/redshift/impl.py:73:16: E121 continuation line under-indented for hanging indent
plugins/redshift/dbt/adapters/redshift/impl.py:89:14: W291 trailing whitespace

• Could you remove this repo from your CircleCI org, so that it will run in our org instead?
• It looks like some of your commits have been co-authored by kostkaw <wojciech.kostka@contractors.roche.com>, which which hasn't signed the CLA. Could you amend those commits to be singularly authored by the email associated with your @kostek-pl account?

[kostek-pl]

@jtcohen6 thank you a lot for the remarks.
I've amended previous commits and got rid of my corporate email, removed CircleCi from my project and adjusted the code to be consistent with flake8.

Yes, I can confirm that This PR resolves the problem locally with docs generating.
To be more precise I've created 2 models, the first one reads(SOURCE) from the table that is created
in another database(sources) and the second one reads(REF) the first one which is on current database(dev).

with RA3_node not set or set to false I get an error:

Error sending message, disabling tracking
Compilation Error
Cross-db references allowed only in redshift RA3.* node. Got {'sources', 'dev'}

when RA3_node is set to true then docs are generated

example used:

profile:

[...]
target: dev
  outputs:
    dev:
      type: redshift
      ra3_node: true
[...]

schema:

version: 2

sources:
  - name: sources
    database: sources
    schema: public
    tables:
      - name: cross_db_test_table

models:
    - name: cross_db_step1

    - name: cross_db_step2

cross_db_step1:

select * from {{ source('sources', 'cross_db_test_table') }}

cross_db_step2:

select * from {{ ref('cross_db_step1') }}

result:

[jtcohen6]

This looks great @kostek-pl!

Could you add a changelog entry, and add yourself to the list of contributors? We've just released 0.20.0rc1, and this PR represents net-new functionality, so it should really go under a new section dbt next (i.e. v0.21). If we end up putting out another release candidate for v0.20, however, I'll consider sneaking it in :)

[jtcohen6]

Thanks for this @kostek-pl!

[jtcohen6]

If we end up putting out another release candidate for v0.20, however, I'll consider sneaking it in :)

Update: We will be putting out v0.20.0rc2 to fix some bugs and installation issues with v0.20.0rc1. That said, we're going to keep the scope of v0.21 narrow, and be disciplined about putting out a first prerelease within 6-8 weeks of v0.20.0-final. So I'm going to stay disciplined here, too, and keep the change in v0.21.