[CT-590] [Bug] Macro change not picked up by state:modified if macro is invoked after an unchanged macro (from a package) in the model code

[jeremyyeo]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

This is related to a previous bug #4678 but this time the unchanged macro is coming from a package instead.

Expected Behavior

State comparison should correctly identify all models as being modified.

Steps To Reproduce

1. Install packages with dbt deps:

# packages.yml
packages:
  - package: dbt-labs/dbt_utils
    version: 0.8.0

2. Setup macros:

{% macro my_macro_a() -%}
foo
{%- endmacro %}

{% macro my_macro_b() -%}
foo
{%- endmacro %}

We will be modifying my_macro_a in this reproduction.

3. Setup models:

-- self_then_self_modified.sql
select '{{ my_macro_b() }}' as val, '{{ my_macro_a() }}' as valval

-- self_modified_then_self.sql
select '{{ my_macro_a() }}' as val, '{{ my_macro_b() }}' as valval

-- self_modified_then_package.sql
select '{{ my_macro_a() }}' as val, {{ dbt_utils.current_timestamp() }} as time_now

-- package_then_self_modified.sql
select {{ dbt_utils.current_timestamp() }} as time_now, '{{ my_macro_a() }}' as val

4. Do a run to generate the state:

$ dbt run
02:43:39  Running with dbt=1.0.6
02:43:40  Found 4 models, 0 tests, 0 snapshots, 0 analyses, 354 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
02:43:40  
02:43:40  Concurrency: 1 threads (target='dev')
02:43:40  
02:43:40  1 of 4 START table model public.package_then_self_modified...................... [RUN]
02:43:41  1 of 4 OK created table model public.package_then_self_modified................. [SELECT 1 in 0.10s]
02:43:41  2 of 4 START table model public.self_modified_then_package...................... [RUN]
02:43:41  2 of 4 OK created table model public.self_modified_then_package................. [SELECT 1 in 0.07s]
02:43:41  3 of 4 START table model public.self_modified_then_self......................... [RUN]
02:43:41  3 of 4 OK created table model public.self_modified_then_self.................... [SELECT 1 in 0.06s]
02:43:41  4 of 4 START table model public.self_then_self_modified......................... [RUN]
02:43:41  4 of 4 OK created table model public.self_then_self_modified.................... [SELECT 1 in 0.06s]
02:43:41  
02:43:41  Finished running 4 table models in 0.51s.
02:43:41  
02:43:41  Completed successfully
02:43:41  
02:43:41  Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4

5. Save our state (manifest.json) to another folder so we can point to it later:

mv target target_old

6. Modify our my_macro_a:

{% macro my_macro_a() -%}
bar
{%- endmacro %}

7. Check which models are considered to be modified with dbt ls --select state:modified --state target_old:

$ dbt ls --select state:modified --state target_old
postgres.self_modified_then_package
postgres.self_modified_then_self
postgres.self_then_self_modified

Looks like #4820 resolved the macro ordering issue but only if the namespace was the root project - not if the unchanged macros were coming from a different namespace (an install dbt package - e.g. dbt_utils).

Relevant log output

No response

Environment

- OS: macOS
- Python: 3.8.7
- dbt: 1.0.6

What database are you using dbt with?

postgres

Additional Context

Some info from manifest.json

  "model.postgres.self_modified_then_package": {
      "raw_sql": "select '{{ my_macro_a() }}' as val, {{ dbt_utils.current_timestamp() }} as time_now",
      "depends_on": {
        "macros": [
          "macro.postgres.my_macro_a",
          "macro.dbt_utils.current_timestamp",
          "macro.dbt.drop_relation_if_exists",
          "macro.dbt.run_hooks",
          "macro.dbt.statement",
          "macro.dbt.create_indexes",
          "macro.dbt.persist_docs"
        ],

    "model.postgres.self_modified_then_self": {
      "raw_sql": "select '{{ my_macro_a() }}' as val, '{{ my_macro_b() }}' as valval",
      "depends_on": {
        "macros": [
          "macro.postgres.my_macro_a",
          "macro.postgres.my_macro_b",
          "macro.dbt.drop_relation_if_exists",
          "macro.dbt.run_hooks",
          "macro.dbt.statement",
          "macro.dbt.create_indexes",
          "macro.dbt.persist_docs"
        ],

    "model.postgres.self_then_self_modified": {
      "raw_sql": "select '{{ my_macro_b() }}' as val, '{{ my_macro_a() }}' as valval",
      "depends_on": {
        "macros": [
          "macro.postgres.my_macro_b",
          "macro.postgres.my_macro_a",
          "macro.dbt.drop_relation_if_exists",
          "macro.dbt.run_hooks",
          "macro.dbt.statement",
          "macro.dbt.create_indexes",
          "macro.dbt.persist_docs"
        ],

   "model.postgres.package_then_self_modified": {
      "raw_sql": "select {{ dbt_utils.current_timestamp() }} as time_now, '{{ my_macro_a() }}' as val",
      "depends_on": {
        "macros": [
          "macro.dbt_utils.current_timestamp",  # Issue occurs when this comes before the modified macro
          "macro.postgres.my_macro_a",
          "macro.dbt.drop_relation_if_exists",
          "macro.dbt.run_hooks",
          "macro.dbt.statement",
          "macro.dbt.create_indexes",
          "macro.dbt.persist_docs"
        ],

[stu-k]

@jeremyyeo Apologies for the late reply, I have been looking into this!