NTD: 2024 NTD XLSX warehouse tables

[charlie-costanzo]

Description

This PR adds new 2024 NTD XLSX endpoint tables to the warehouse via new staging SQL files, updated source/and staging YML files, and the appropriate intermediate table unions.

Resolves #4486

Type of change

• New feature

How has this been tested?

local dbt
poetry run dbt run -s +dim_agency_information +fct_contractual_relationships

Post-merge follow-ups

• Actions required (specified below)
  Developers see up to date NTD Pipeline documentation #4483
  We need to make sure that we document the unioning processes for the intermediate tables captured In this PR in the work for the issue above

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/composer/us

Plan: 0 to add, 1 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_composer_environment.calitp-staging-composer will be updated in-place
!~  resource "google_composer_environment" "calitp-staging-composer" {
        id               = "projects/cal-itp-data-infra-staging/locations/us-west2/environments/calitp-staging-composer"
        name             = "calitp-staging-composer"
#        (5 unchanged attributes hidden)

!~      config {
#            (8 unchanged attributes hidden)

!~          software_config {
!~              image_version            = "composer-2.10.2-airflow-2.9.3" -> "composer-2.13.1-airflow-2.9.3"
#                (6 unchanged attributes hidden)

#                (1 unchanged block hidden)
            }

#            (8 unchanged blocks hidden)
        }

#        (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #958

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 2 to add, 4 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/ntd_annual_reporting/int_ntd__unioned_agency_information.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "gNkAIw==" -> (known after apply)
!~      detect_md5hash      = "H5p78HVLc0ihl6KxRVvtVw==" -> "different hash"
!~      generation          = 1749663114675954 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_agency_information.sql"
!~      md5hash             = "H5p78HVLc0ihl6KxRVvtVw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_agency_information.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/ntd_annual_reporting/int_ntd__unioned_contractual_relationships.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "leS+9Q==" -> (known after apply)
!~      detect_md5hash      = "/ztwxJjXRpPR/tQID6Bf5A==" -> "different hash"
!~      generation          = 1749663117980508 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_contractual_relationships.sql"
!~      md5hash             = "/ztwxJjXRpPR/tQID6Bf5A==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_contractual_relationships.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/ntd_annual_reporting/_src.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "VU39NA==" -> (known after apply)
!~      detect_md5hash      = "ipjE1mifzCdzNFwXs7YZ6Q==" -> "different hash"
!~      generation          = 1762295803375743 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/staging/ntd_annual_reporting/_src.yml"
!~      md5hash             = "ipjE1mifzCdzNFwXs7YZ6Q==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/_src.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/ntd_annual_reporting/_stg_ntd_annual_reporting.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "GykjiA==" -> (known after apply)
!~      detect_md5hash      = "wmZ4b8VMSVWZoXI24Ojpxw==" -> "different hash"
!~      generation          = 1762295803149278 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/staging/ntd_annual_reporting/_stg_ntd_annual_reporting.yml"
!~      md5hash             = "wmZ4b8VMSVWZoXI24Ojpxw==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/_stg_ntd_annual_reporting.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__2024_agency_information.sql"] will be created
+   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
+       bucket         = "calitp-staging-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_agency_information.sql"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_agency_information.sql"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__2024_contractual_relationships.sql"] will be created
+   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
+       bucket         = "calitp-staging-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_contractual_relationships.sql"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_contractual_relationships.sql"
+       storage_class  = (known after apply)
    }

Plan: 2 to add, 4 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #958

[github-actions]

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 2 to add, 4 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/ntd_annual_reporting/int_ntd__unioned_agency_information.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "gNkAIw==" -> (known after apply)
!~      detect_md5hash      = "H5p78HVLc0ihl6KxRVvtVw==" -> "different hash"
!~      generation          = 1751416668440263 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_agency_information.sql"
!~      md5hash             = "H5p78HVLc0ihl6KxRVvtVw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_agency_information.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/ntd_annual_reporting/int_ntd__unioned_contractual_relationships.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "leS+9Q==" -> (known after apply)
!~      detect_md5hash      = "/ztwxJjXRpPR/tQID6Bf5A==" -> "different hash"
!~      generation          = 1751416665077842 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_contractual_relationships.sql"
!~      md5hash             = "/ztwxJjXRpPR/tQID6Bf5A==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/ntd_annual_reporting/int_ntd__unioned_contractual_relationships.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/staging/ntd_annual_reporting/_src.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "VU39NA==" -> (known after apply)
!~      detect_md5hash      = "ipjE1mifzCdzNFwXs7YZ6Q==" -> "different hash"
!~      generation          = 1762295799253611 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/staging/ntd_annual_reporting/_src.yml"
!~      md5hash             = "ipjE1mifzCdzNFwXs7YZ6Q==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/_src.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/staging/ntd_annual_reporting/_stg_ntd_annual_reporting.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "GykjiA==" -> (known after apply)
!~      detect_md5hash      = "wmZ4b8VMSVWZoXI24Ojpxw==" -> "different hash"
!~      generation          = 1762295799249425 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/staging/ntd_annual_reporting/_stg_ntd_annual_reporting.yml"
!~      md5hash             = "wmZ4b8VMSVWZoXI24Ojpxw==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/_stg_ntd_annual_reporting.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__2024_agency_information.sql"] will be created
+   resource "google_storage_bucket_object" "calitp-composer-dags" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_agency_information.sql"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_agency_information.sql"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__2024_contractual_relationships.sql"] will be created
+   resource "google_storage_bucket_object" "calitp-composer-dags" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_contractual_relationships.sql"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../warehouse/models/staging/ntd_annual_reporting/stg_ntd__2024_contractual_relationships.sql"
+       storage_class  = (known after apply)
    }

Plan: 2 to add, 4 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #958

[mjumbewu]

LGTM