Create notebook looking at subplant id across years

[rouille]

Purpose

Compare mapping of (plant_id_eia, generator_id) to subplant_id across years. Closes CAR-3518

What the code is doing

Load the sub-plant crosswalk files for years 2019 through 2022. For each year combination, the intersection of (plant_id_eia, generator_id) is retrieved and occurrences where the subplant_id values differ are saved as a pandas data frame.

Testing

N/A

Where to look

A new notebook named validate_subplant_crosswalk encloses this short analysis

Usage Example/Visuals

See notebook outputs here, e.g.,

Number of difference in subplant_id for same (plant_id_eia, generator_id) combination
2019-2020: 448
2019-2021: 895
2019-2022: 1193
2020-2021: 488
2020-2022: 831
2021-2022: 435

Review estimate

10min

Future work

N/A

Checklist

• Update the documentation to reflect changes made in this PR
• Format all updated python files using black
• Clear outputs from all notebooks modified
• Add docstrings and type hints to any new functions created

[grgmiller]

Thanks, Ben - this is really helpful. Looks like our subplants are not staying static across the years, which will be an issue for using these for GRETA. This will likely be one of the issues we'll want to prioritize fixing once we start on that. It looks like maybe we currently assign subplant IDs based on sorting the generator IDs at each plant, as we had been previously been assuming that new generators would have a higher number than existing generators. However, at least looking at the first few cases, it looks like a new generator is often added out of order (eg 5.1, 8B) which then pushes the remaining subplant numbers. There might also be an issue with subplant grouping (ie I'd assume that generator 8A and 8B would be part of the same subplant). Let's find some time to discuss this later this week in the context of other potential data quality issues.

[rouille]

I have cleared the outputs of the notebook to comply with the guideline