Update PUDL to use Python 3.12

[zaneselvans]

Overview

Switch PUDL over to using Python 3.12, which has been out since October 2nd, 2023. All of our current dependencies now support Python 3.12.

However, I note that theconda-forge package python-duckdb has a Python 3.12 build issue. I think the issue has been resolved for Linux and MacOS so I've poked the maintainer. While duckdb isn't a current PUDL dependency, it will become one as soon as @katie-lamb's Splink PR #3302 is merged, so we should probably hold off on merging this PR until the python-duckdb package is updated on conda-forge.

Closes #3327

Testing

To-do list

Beta Give feedback

1. Ensure the full ETL runs locally
2. Update the release notes, reference the PR and related issues.
3. Ensure docs build, unit & integration tests, and test coverage pass locally with make pytest-coverage (otherwise the merge queue may reject your PR)
4. Ensure data validation passes locally
5. Review the PR yourself and call out any questions or issues you have
6. Run a full build on the branch

[zaneselvans]

0.59 is the first numba version that works with Python 3.12

[zaneselvans]

I think 1.6.5 (0.22.5) is the first Dagster version that works with Python 3.12

[zaneselvans]

464 is the first version of the google-cloud-sdk that works with Python 3.12 (thanks to my prodding...)

[zaneselvans]

@katie-lamb & @zschira I did a full ETL using Python 3.12 locally and it almost completed. The only error that came up was in out_pudl__yearly_assn_eia_ferc1_plant_parts.ferc_to_eia.get_model_predictions during some splinking. Error below. Any idea what might be up?

IndexError: list index out of range
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/dagster/_utils/__init__.py", line 463, in iterate_with_context
    next_output = next(iterator)
                  ^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/dagster/_core/execution/plan/compute_generator.py", line 131, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
             ^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/dagster/_core/execution/plan/compute_generator.py", line 125, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
                                                                    ^^^^^^^^^^^^^^^^^^
  File "/Users/zane/code/catalyst/pudl/src/pudl/analysis/record_linkage/eia_ferc1_record_linkage.py", line 236, in get_model_predictions
    preds_df = linker.predict(threshold_match_probability=threshold_prob)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/linker.py", line 1766, in predict
    sqls = predict_from_comparison_vectors_sqls(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/predict.py", line 21, in predict_from_comparison_vectors_sqls
    select_cols = settings_obj._columns_to_select_for_bayes_factor_parts
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/settings.py", line 262, in _columns_to_select_for_bayes_factor_parts
    cols.extend(cc._columns_to_select_for_bayes_factor_parts)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/comparison.py", line 254, in _columns_to_select_for_bayes_factor_parts
    sqls = [cl._tf_adjustment_sql for cl in self.comparison_levels]
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/comparison_level.py", line 584, in _tf_adjustment_sql
    u_prob_exact_match = self._u_probability_corresponding_to_exact_match
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/comparison_level.py", line 535, in _u_probability_corresponding_to_exact_match
    if not level._is_exact_match:
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/comparison_level.py", line 503, in _is_exact_match
    if not _is_exact_match(expr):
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.12/site-packages/splink/comparison_level.py", line 45, in _is_exact_match
    if identifiers[0] == identifiers[1]:

[katie-lamb]

@zaneselvans See moj-analytical-services/splink#2072 , the issue seems to be from incompatibility of splink with the latest version of the SQLGlot dependency. See moj-analytical-services/splink#2076 (comment) , which suggests pinning sqlglot==22.5.0. Seems like the issue will be fixed imminently by moj-analytical-services/splink#2079 , so I think we could just wait too.

[zaneselvans]

Ah great, thanks for looking into it! I'll pin the version and see if all is well.

[zaneselvans]

Woo! This fixed it. Will do a full build on the branch and merge into main tonight if it succeeds.