Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BigQuery] Print out longest path through view graph (Recidiviz/recid…
…iviz-data#29579) ## Description of the change Updates the stats logging at the end of each view update / sandbox load to print out the longest path through the graph so we can begin to identify bottlenecks! Tested with by running this command: ``` python -m recidiviz.tools.load_views_to_sandbox \ --sandbox_dataset_prefix ageiduschek manual \ --view_ids_to_load shared_metric_views.event_based_commitments_from_supervision_for_matrix \ --update_ancestors False \ --update_descendants True ``` ... which produced this output: ``` INFO:root:### BQ DAG PROCESSING STATS ### Total processing time: 7.64 sec Nodes processed: 33 Average queue wait time: 0.0 seconds Max queue wait time: 0.0 seconds Top [25] most expensive nodes in DAG: 1) 1.46 sec: ageiduschek_dashboard_views.revocations_matrix_distribution_by_violation 2) 1.44 sec: ageiduschek_shared_metric_views.supervision_termination_matrix_by_person 3) 1.38 sec: ageiduschek_shared_metric_views.event_based_commitments_from_supervision_for_matrix 4) 1.38 sec: ageiduschek_shared_metric_views.supervision_matrix_by_person 5) 1.35 sec: ageiduschek_dashboard_views.revocations_matrix_distribution_by_officer 6) 1.31 sec: ageiduschek_shared_metric_views.admission_types_per_state_for_matrix 7) 1.31 sec: ageiduschek_dashboard_views.revocations_matrix_distribution_by_race 8) 1.30 sec: ageiduschek_dashboard_views.revocations_matrix_filtered_caseload 9) 1.29 sec: ageiduschek_dashboard_views.revocations_matrix_distribution_by_gender 10) 1.29 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_gender_comparison_supervision_errors 11) 1.25 sec: ageiduschek_dashboard_views.revocations_matrix_distribution_by_risk_level 12) 1.23 sec: ageiduschek_validation_views.revocation_matrix_comparison_revocations_by_officer_errors 13) 1.22 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_race_comparison_revocation_errors 14) 1.22 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_race_comparison_recommendation_errors 15) 1.22 sec: ageiduschek_validation_views.revocation_matrix_comparison_revocation_cell_vs_caseload 16) 1.21 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_gender_comparison 17) 1.21 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_race_comparison 18) 1.21 sec: ageiduschek_validation_views.revocation_matrix_comparison_supervision_population 19) 1.20 sec: ageiduschek_validation_views.revocation_matrix_comparison_by_month_errors 20) 1.19 sec: ageiduschek_validation_views.revocation_matrix_comparison_by_month 21) 1.19 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_race_comparison_supervision_errors 22) 1.19 sec: ageiduschek_dashboard_views.revocations_matrix_supervision_location_ids_to_names 23) 1.18 sec: ageiduschek_validation_views.revocation_matrix_distribution_by_gender_comparison_recommendation_errors 24) 1.18 sec: ageiduschek_validation_views.revocation_matrix_caseload_admission_history_errors 25) 1.18 sec: ageiduschek_validation_views.revocation_matrix_caseload_admission_history Most expensive path [7.63 seconds total]: * ageiduschek_shared_metric_views.event_based_commitments_from_supervision_for_matrix (1.38 sec) * ageiduschek_shared_metric_views.revocations_matrix_by_person (1.15 sec) * ageiduschek_shared_metric_views.admission_types_per_state_for_matrix (1.31 sec) * ageiduschek_dashboard_views.revocations_matrix_distribution_by_gender (1.29 sec) * ageiduschek_validation_views.revocation_matrix_distribution_by_gender_comparison (1.21 sec) * ageiduschek_validation_views.revocation_matrix_distribution_by_gender_comparison_supervision_errors (1.29 sec) ``` ## Type of change > All pull requests must have at least one of the following labels applied (otherwise the PR will fail): | Label | Description | |----------------------------- |----------------------------------------------------------------------------------------------------------- | | Type: Bug | non-breaking change that fixes an issue | | Type: Feature | non-breaking change that adds functionality | | Type: Breaking Change | fix or feature that would cause existing functionality to not work as expected | | Type: Non-breaking refactor | change addresses some tech debt item or prepares for a later change, but does not change functionality | | Type: Configuration Change | adjusts configuration to achieve some end related to functionality, development, performance, or security | | Type: Dependency Upgrade | upgrades a project dependency - these changes are not included in release notes | ## Related issues Related to Recidiviz/recidiviz-data#29291 ## Checklists ### Development **This box MUST be checked by the submitter prior to merging**: - [x] **Double- and triple-checked that there is no Personally Identifiable Information (PII) being mistakenly added in this pull request** These boxes should be checked by the submitter prior to merging: - [x] Tests have been written to cover the code changed/added as part of this pull request ### Code review These boxes should be checked by reviewers prior to merging: - [x] This pull request has a descriptive title and information useful to a reviewer - [x] Potential security implications or infrastructural changes have been considered, if relevant GitOrigin-RevId: 3ae10312ce236e16bc5ccc93321fdd9cd8b08850
- Loading branch information