add configurable stu-program-agg columns

[rlittle08]

Description and Motivation

By default in EDU, programs have is_active and is_annual as indicators that are aggregated by k_student. Those are written in build models like here. This feature branch adds the ability to add arbitrarily many more of those types of student aggregate program indicators, with custom rules.

For example, if you want to add a new indicator for whether a student was in a 504 Plan Program during the school year, you can configure these variable in your dbt_project.yml, assuming your implementation has program_name values equal to '504 Plan':

  # custom indicators in special ed model at the same agg level as is_special_ed_active & annual
  'edu:special_ed:custom_program_agg_indicators':
    is_504_plan_annual:
      agg_sql: >
        -- if the student had a 504 program enrollment any time during the year  
 
        max(
          program_name = '504 Plan'
        )

And the compiled code in the build model will look like this:

-- custom special ed program agg indicators
-- if the student had a 504 program enrollment any time during the year
max(
  program_name = '504 Plan'
)
 as is_504_plan_annual,

And any student with a program enrollment with name '504 Plan' will now have is_504_plan_annual = TRUE in dim_student. Plus, those values will be automatically added to dim_subgroup and fct_student_subgroup.

Changes to existing models:

• bld_ef3__student_program__special_education, this code optionally adds more agg columns by grabbing sql code from the project config:
  

  edu_wh/models/build/edfi_3/students/bld_ef3__student_program__special_education.sql

  Lines 36 to 41 in 543a8b6

  	-- custom special ed program agg indicators
  	{% if custom_program_agg_indicators is not none and custom_program_agg_indicators | length -%}
  	{%- for indicator in custom_program_agg_indicators -%}
  	{{ custom_program_agg_indicators[indicator]['agg_sql'] }} as {{ indicator }},
  	{%- endfor -%}
  	{%- endif %}

  • (Same code is repeated for bld_ef3__student_program__homeless, bld_ef3__student_program__language_instruction, and bld_ef3__student_program__title_i )
• dim_student, this code adds new custom indicators as columns:

  edu_wh/models/core_warehouse/dim_student.sql

  Lines 93 to 96 in 543a8b6

  	{% if custom_program_agg_indicators is not none and custom_program_agg_indicators | length -%}
  	{% for custom_indicator in custom_program_agg_indicators %}
  	coalesce(stu_special_ed.{{custom_indicator}}, false) as {{custom_indicator}},
  	{% endfor %}

  • (Same code is repeated for bld_ef3__student_program__homeless, bld_ef3__student_program__language_instruction, and bld_ef3__student_program__title_i )

New models added:

None

Tests and QC Done:

Tested on Boston with configuration of 504 plan annual & active indicators.

Also tested on SCDE with configuration of homeless unaccompanied youth indicators.

Tested in both Boston and SCDE with an empty & missing config

If adding configuration, did you test with empty, broken, and nonexistent configuration?

Yes, I tested with no configuration added, and the new code compiles to empty.

PR Merge Priority:

(Promised this feature by end of Sept for BPS)

• Low
• Medium
• High

Questions:

This code assumes your configuration will include SQL to aggregate over k_student. What if you want to add a k_student_xyear aggregation? Should that be included in this branch?

[jalvord1]

we talked through this idea but noting here that I think it's a common enough use-case to account for any additional complexity, especially considering the added logic to not require the new var

[jayckaiser]

What is the reason for nesting the SQL query under the agg_sql key? Are there other keys that we will want to add in the future for flexibility?

[rlittle08]

it is possible in future we want more attributes, e.g. include_in_dim_student

[jayckaiser]

I've left two comments regarding general structure of the changes. Overall, this addition looks good and adds much-needed flexibility when it comes to custom programs.