Fix: add warning on duplicated yaml keys

[jeremyyeo]

resolves #5114

Description

When we have a duplicate top level key in our schema yml files, we should see the following warning:

$ dbt run
08:01:05  Running with dbt=1.2.0-a1
08:01:05  [WARNING]: Compilation Error
  Duplicate 'models' top level key found in yaml file models/schema.yml.
08:01:05  [WARNING]: Compilation Error
  Duplicate 'models' top level key found in yaml file models/sub/schema.yml.
08:01:05  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 167 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
08:01:05  
08:01:06  Concurrency: 1 threads (target='dev')
08:01:06  
08:01:06  1 of 2 START table model public.my_model_a ..................................... [RUN]
08:01:06  1 of 2 OK created table model public.my_model_a ................................ [SELECT 1 in 0.10s]

---

1. In core/dbt/exceptions.py, DuplicateYamlKeyException is a no-op of CompilationException - is this okay?

2. For the tests, because we warn instead of error, I'm using run_dbt_and_capture() from dbt.tests.utils and asserting that stdout has the right string - not too sure if this is the right way to do this. I also tried something along the lines of:

from dbt.exceptions import DuplicateYamlKeyException

...

def test_duplicate_key_in_yaml(project):
    with pytest.raises(DuplicateYamlKeyException):
        run_dbt(["run"])

Which didn't quite work because we print a warning instead of raising an error. (Ftr, I just cribbed what was in dbt/tests/functional/basic/tests_basic.py for this one).

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
• I have run changie new to create a changelog entry

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[jeremyyeo]

Integration test from days of yonder fails led me to a yaml test file (https://github.com/dbt-labs/dbt-core/blob/main/test/integration/006_simple_dependency_tests/local_dependency/models/schema.yml) with actual duplicated top level keys - it looks like the "mapping node should be a set" ¹ so curious if that particular test worked properly before 🤔

---

Looks like as a byproduct of this, previously when the project had multiple duplicated keys:

...
models:
  - name: my_model_a
    columns:
      - name: id
        tests:
          - unique
  
 models:
  - name: my_model_b
    columns:
      - name: id
        tests:
          - unique

A dbt test would silently test according to the models key that is the last in the schema yml file (i.e. just my_model_b will be tested), but now, since the DuplicateYamlKeyException is raised in the loader, we will not be doing any tests at all.

Footnotes

1. https://stackoverflow.com/questions/47668308/duplicate-key-in-yaml-configuaration-file ↩

[emmyoop]

I had a few comments about structure that I wanted to get back to you on. I'll take a look at the failing legacy tests and see what's happening there next.

[emmyoop]

You should use warn_or_raise here instead of warn_or_error. That way the exception you throw above bubbles up instead of getting overlaid by the CompilationException by warn_or_error. To get the message passed through as DuplicateYamlKeyException you can modify the message.

e.msg = f"{e} {path.searched_path}/{path.relative_path}."
dbt.exceptions.warn_or_raise(e, log_fmt=warning_tag("{}"))

[jeremyyeo]

With the new changes I see:

$ dbt run                                 
10:59:42  Running with dbt=1.2.0-a1
10:59:42  [WARNING]: Compilation Error
  Compilation Error
    Duplicate 'models' key found in yaml file models/schema.yml.
10:59:43  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 167 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
10:59:43  
10:59:43  Concurrency: 1 threads (target='dev')
...

Wondering if that was what you expected here?

[jeremyyeo]

@emmyoop surprisingly fixing the duplicate top level keys in this yaml file in the legacy tests causes it to pass so now really curious what this was doing previously... makes me wonder if we had left 2 sources keys in this file by mistake the last time around but test was passing due to pyaml default of only reading the last source key of this file.

[emmyoop]

I'm not seeing anything in the tests that indicate we did it on purpose so you may be right. These tests are going to be rewritten and we'll work through some of the weirdness in them as we go.

[gshank]

Looks good.