Skip to content

Commit

Permalink
Revert "[Airflow] Unify Dataflow task group code (Recidiviz/recidiviz…
Browse files Browse the repository at this point in the history
…-data#29453)" (Recidiviz/recidiviz-data#29533)

## Description of the change

This reverts commit dedeac1cdc1bc51a693ae3836c3542f0174aafb9.

This change caused a few problems:
1) NC normalization pipelines are crashing (NC wasn't running
normalization before)
2) Ingest DAG is failing to launch pipelines because the way I'm pulling
`raw_data_upper_bound_dates_json` is buggy.

## Type of change

> All pull requests must have at least one of the following labels
applied (otherwise the PR will fail):

| Label | Description |
|-----------------------------
|-----------------------------------------------------------------------------------------------------------
|
| Type: Bug | non-breaking change that fixes an issue |
| Type: Feature | non-breaking change that adds functionality |
| Type: Breaking Change | fix or feature that would cause existing
functionality to not work as expected |
| Type: Non-breaking refactor | change addresses some tech debt item or
prepares for a later change, but does not change functionality |
| Type: Configuration Change | adjusts configuration to achieve some end
related to functionality, development, performance, or security |
| Type: Dependency Upgrade | upgrades a project dependency - these
changes are not included in release notes |

## Related issues

Related to Recidiviz/recidiviz-data#27378.

## Checklists

### Development

**This box MUST be checked by the submitter prior to merging**:
- [x] **Double- and triple-checked that there is no Personally
Identifiable Information (PII) being mistakenly added in this pull
request**

These boxes should be checked by the submitter prior to merging:
- [x] Tests have been written to cover the code changed/added as part of
this pull request

### Code review

These boxes should be checked by reviewers prior to merging:

- [x] This pull request has a descriptive title and information useful
to a reviewer
- [x] Potential security implications or infrastructural changes have
been considered, if relevant

GitOrigin-RevId: bfb164555c01b63ecd9249c20abd1a4d46f3cb87
  • Loading branch information
ageiduschek authored and Helper Bot committed May 16, 2024
1 parent 4eec745 commit 3ffa2a0
Show file tree
Hide file tree
Showing 21 changed files with 287 additions and 473 deletions.
6 changes: 3 additions & 3 deletions recidiviz/admin_panel/ingest_dataflow_operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
from recidiviz.ingest.direct.types.direct_ingest_instance import DirectIngestInstance
from recidiviz.pipelines.ingest.dataset_config import state_dataset_for_state_code
from recidiviz.pipelines.ingest.pipeline_utils import (
DEFAULT_PIPELINE_REGIONS_BY_STATE_CODE,
DEFAULT_INGEST_PIPELINE_REGIONS_BY_STATE_CODE,
)
from recidiviz.utils import metadata

Expand Down Expand Up @@ -89,10 +89,10 @@ def get_latest_job_for_state_instance(
return None

# TODO(#209930): remove this check once dataflow is launched for all states
if state_code not in DEFAULT_PIPELINE_REGIONS_BY_STATE_CODE:
if state_code not in DEFAULT_INGEST_PIPELINE_REGIONS_BY_STATE_CODE:
return None

location = DEFAULT_PIPELINE_REGIONS_BY_STATE_CODE[state_code]
location = DEFAULT_INGEST_PIPELINE_REGIONS_BY_STATE_CODE[state_code]
client = dataflow_v1beta3.JobsV1Beta3Client()

if job_id:
Expand Down
16 changes: 0 additions & 16 deletions recidiviz/airflow/dags/calculation/dataflow/__init__.py

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

0 comments on commit 3ffa2a0

Please sign in to comment.