[CT-1915] [Epic] Multi-project collaboration - Milestone 1

[MichelleArk]

Develop new constructs that enable dbt developers to create, contract, and communicate data models like software APIs. This work should enable more scalable mono-repos, while also laying the foundation for Milestone 2.

Discussions:

• Multi-project collaboration #6725
• Model contracts #6726
• Model groups & access #6730
• Model versions #6736

Tasks

Beta Give feedback

We're having trouble rendering your tasklist, it's most likely has one of the following formatting errors:

• indented tasks (i.e. nested tasklist; support for this is coming!)
• an empty task (i.e. - [ ] on a line by itself)
• any empty new lines (before/after the tasklist or in between tasks)
• duplicate issue/pr links
• a draft task exceeds 512 characters

Please check the tasklists documentation for more information. 🔗 Feedback Discussion

Thank you for participating in the Private Beta ❤️

No tasks being tracked yet.

Model contracts - Adapters

Beta Give feedback

1. Introducing constraints and contract:
2. dbt Constraints / model contracts dbt-snowflake#341
   cla:yes tech_debt +2
   
3. dbt Constraints / model contracts dbt-redshift#229
   cla:yes +1
   
4. dbt Constraints / model contracts dbt-bigquery#426
   Team:Adapters cla:yes +2
   
5. dbt Constraints / model contracts dbt-spark#574
   cla:yes +1
   
6. Renaming constraints_enabled to contract:
7. [CT-2151] Rename "constraints_enabled" to "contract" dbt-snowflake#479
   model_contracts +1
   
8. [CT-2152] Rename "constraints_enabled" to "contract" dbt-bigquery#548
   model_contracts +1
   
9. [CT-2153] Rename "constraints_enabled" to "contract" dbt-spark#645
   model_contracts +1
   
10. [CT-2154] Rename "constraints_enabled" to "contract" dbt-redshift#330
    model_contracts +1
    
11. Implement get_column_schema_from_sql macro:
12. [CT-2101] Implement get_column_schema_from_query macro dbt-snowflake#464
    model_contracts +1
    
13. [CT-2102] Implement get_column_schema_from_query macro dbt-bigquery#529
    model_contracts +1
    
14. [CT-2104] Implement get_column_schema_from_query macro dbt-spark#639
    model_contracts +1
    
15. [CT-2103] Implement get_column_schema_from_query macro dbt-redshift#319
    model_contracts +1
    
16. Enforcing model contracts in view materializations:
17. [CT-2239] [Spark] Enforce model contract in spark__create_view_as dbt-spark#654
    model_contracts +1
    
18. [CT-2168] [BigQuery] Enforce model contract in bigquery__create_view_as dbt-bigquery#555
    model_contracts +1
    
19. [CT-2238] [Snowflake] Enforce model contract in snowflake__create_view_as_with_temp_flag dbt-snowflake#490
    model_contracts +1
    
20. [CT-2237] [Redshift] Enforce model contract in redshift__create_view_as dbt-redshift#340
    model_contracts +1
    
21. Use unified constraints data structure:
22. [CT-2212] [Snowflake] Use new ColumnLevelConstraint structure in snowflake__get_columns_spec_ddl dbt-snowflake#491
    model_contracts +1
    
23. [CT-2213] [BigQuery] Use new ColumnLevelConstraint structure in bigquery__get_columns_spec_ddl dbt-bigquery#567
    model_contracts +1
    
24. [CT-2214] [Redshift] Use new ColumnLevelConstraint structure in redshift__get_columns_spec_ddl dbt-redshift#341
    model_contracts +1
    
25. [CT-2215] [Spark] Use new ColumnLevelConstraint structure in spark__alter_table_add_constraints and spark__alter_column_set_constraints dbt-spark#655
    model_contracts +1
    
26. constraint_support methods:
27. [CT-2216] [Snowflake] Remove constraints_check warning in get_columns_spec_ddl dbt-snowflake#492
    model_contracts +1
    
28. [CT-2217] [Redshift] Remove constraints_check warning in get_columns_spec_ddl dbt-redshift#342
    model_contracts +1
    
29. [CT-2218] [BigQuery] Implement constraint_support on BigQueryAdapter dbt-bigquery#568
    model_contracts +1
    
30. [CT-2219] [Spark] Implement constraint_support on SparkAdapter dbt-spark#656
    model_contracts +1
    
31. model-level constraints:
32. [CT-2221] [BigQuery] Add model-level constraints to get_columns_spec_ddl dbt-bigquery#569
    model_contracts +1
    
33. [CT-2220] [Snowflake] Add model-level constraints to get_columns_spec_ddl dbt-snowflake#493
    model_contracts +1
    
34. [CT-2222] [Redshift] Add model-level constraints to get_columns_spec_ddl dbt-redshift#343
    model_contracts +1
    
35. [CT-2223] [Spark] Add model-level constraints to get_columns_spec_ddl dbt-spark#657
    model_contracts +1
    
36. Order-agnostic constraint ddl:
37. [ADAP-349] [Snowflake] Generate order-agnostic constraints ddl when contract: true dbt-snowflake#498
    model_contracts +1
    
38. [ADAP-351] [BigQuery] Generate order-agnostic constraints ddl when contract: true dbt-bigquery#579
    model_contracts +1
    
39. [ADAP-352] [Redshift] Generate order-agnostic constraints ddl when contract: true dbt-redshift#356
    model_contracts +1
    
40. [ADAP-364] [Spark] Generate order-agnostic constraints ddl when contract: true dbt-spark#671
    model_contracts +1
    
41. Convert contract from bool to dictionary
42. [ADAP-417] Convert contract into a dict dbt-bigquery#633
    +0
    
43. [ADAP-418] Convert contract into a dict dbt-snowflake#539
    +0
    
44. [ADAP-419] Convert contract into a dict dbt-redshift#382
    +0
    
45. [ADAP-420] Convert contract into dict dbt-spark#693
    +0
    

Model groups & access

Beta Give feedback

1. [CT-1999] Unify ExposureOwner dataclass with group Owner #6833
   Team:Language model_groups_access multi_project user docs +4
   
2. [CT-1989] Parse group resource #6822
   Team:Language model_groups_access multi_project user docs +4
   
3. [CT-1991] Parse group config on 'groupable' nodes #6823
   Team:Language model_groups_access multi_project user docs +4
   
4. [CT-1990] Parse access property on model node #6824
   Team:Language model_groups_access multi_project user docs +4
   
5. [CT-1993] Raise an exception from ref if referencing a private model across groups #6826
   Team:Language model_groups_access multi_project user docs +4
   
6. [CT-1992] Support group: selection method #6825
   Team:Language model_groups_access multi_project node selection user docs +5
   

Model versions

Beta Give feedback

1. [CT-2352] Versioned Models - parsing and referencing #7263
   model_versions multi_project +2
   
2. [CT-1815] [Feature] Remove required 'version' from dbt_project.yml #6603
   enhancement help_wanted +2
3. [CT-2189] Remove required version: 2 and config-version: 2 #7054
   Team:Language enhancement good_first_issue model_versions paper_cut +5
   
4. [CT-2035] [Spike] Parsing: model ‘version’ attribute #6866
   Team:Language model_versions multi_project spike +4
   
5. [CT-2036] Parsing: model ‘latest’ attribute #6867
   Team:Language model_versions multi_project +3
   
6. [CT-2041] Accept optional 'version' keyword argument to ref #6872
   Team:Language model_versions multi_project +3
   
7. [CT-2040] Include model version in alias #6871
   Team:Language model_versions multi_project +3
   
8. [CT-2038] Detect breaking changes to column names and data types in state:modified check #6869
   Team:Language model_contracts model_versions multi_project user docs +5
   
9. [CT-2319] Selection for versioned models #7199
   Refinement Team:Language model_versions multi_project node selection +5
   
10. [CT-2401] [Bug] Adding a new column to a model with an enforced contract should not be a breaking change #7332
    Team:Language bug +2
    

dbt-docs Improvements

Beta Give feedback

1. [CT-2200] Model view: when available, display model owner in dbt-docs dbt-docs#377
   Team:Language dbt-docs multi-project +3
   
2. [CT-2204] Model view: when available, indicate a model's access status (private/public) dbt-docs#381
   Team:Language dbt-docs multi-project +3
   
3. [CT-2201] DAG visualization: visually distinguish models based on 'access' dbt-docs#378
   Team:Language dbt-docs multi-project +3
   
4. [CT-2202] Add a view of all public models with the same group dbt-docs#379
   Team:Language dbt-docs multi-project +3
   
5. [CT-2203] DAG visualization: selection by group dbt-docs#380
   multi-project +1
   
6. [CT-2328] [Spike] Display versioned models in dbt-docs dbt-docs#406
   Team:Language multi-project +2
   
7. [CT-2326] Model view: display column-level constraints dbt-docs#405
   Team:Language multi-project +2

docs.getdbt.com

Beta Give feedback

1. Narrative: Model contracts docs.getdbt.com#2839
   Initialize content dbt-core v1.5 improvement size: large +5
2. Narrative: Model groups & access docs.getdbt.com#2840
   content dbt-core v1.5 improvement size: large +4
3. Narrative: Model versions docs.getdbt.com#2841
   Initialize content dbt-core v1.5 improvement size: large +5
4. Update multi-project docs for v1.5.0-b3 (March 2) docs.getdbt.com#2938
   content dbt-core v1.5 improvement +3
   
5. Update multi-project docs for v1.5.0-b4 (March 16) docs.getdbt.com#2945
   content dbt-core v1.5 improvement +3
   
6. Update multi-project docs for v1.5.0-b5 (March 30) docs.getdbt.com#3023
   content dbt-core v1.5 improvement +3
   
7. Update multi-project docs for v1.5.0-rc1 (April 13) docs.getdbt.com#3099
   content dbt-core v1.5 improvement +3
   

Tracking

Beta Give feedback

1. [CT-2305] Add tracking for groups, contract, access, and version usage #7170
   Team:Language multi_project +2
   
2. [CT-2306] Update tracking.RESOURCE_COUNTS to include groups and metrics #7171
   Team:Language multi_project +2
   

Moved to milestone 2

Beta Give feedback

1. [CT-2126] [Spike] Enforce model contracts for python models #6984
   model_contracts multi_project spike stale +4
   
2. [CT-2037] Parsing: model ‘deprecation_date’ attribute #6868
   Refinement Team:Language model_versions multi_project +4
3. [CT-2039] Raise deprecation warning when compiling/running resource that depends on model with deprecation date #6870
   Refinement Team:Language model_versions multi_project +4
4. [CT-2195] [Feature] Validation: disallow access:private on ungrouped model #7063
   Team:Language enhancement model_groups_access multi_project +4
5. [CT-2327] [Spike] Produce publication artifact and resolve cross-project refs #7227
   Team:Language multi_project spike +3
   
6. [CT-2325] Validate public models are not materialized as ephemeral #7226
   Team:Language model_groups_access multi_project +3
   
7. [CT-2197] [Feature] Detect breaking changes to constraints in state:modified check #7065
   Team:Language enhancement model_contracts multi_project user docs +5